If you want to extract mails out of your .nsf database, this might be the tool you’re looking for. Bonus point: it’s written in Python !

Installing nlconverter and its dependencies

Here is how I installed nlconverter on a Windows 2000 (SP4) machine:

1. First I downloaded and installed the official Python builds for Windows (2.6.6 precisely):
   
   
   
   
   
2. Then Python for Windows extensions (build 214 for Python 2.6 in my case):
   
   
   
   
3. Finally I had to download the latest icalendar archive, then extract the \iCalendar-1.2\src\icalendar folder to C:\Python26\Lib\site-packages\:
   
4. Next step is to download nlconverter itself and extract it:
   

nlconverter GUI

First thing you have to do is to create an export of your mails as a .nsf database. Follow the previous link to get the instructions.

Now let’s convert this nsf to a mbox. nlconverter’s FAQ tells you to run the gui.exe program to perform the conversion.

Unfortunately it didn’t work for me:

So I tried the alternative approach by using the command line.

nlconverter command line

Again, most of the things I’m writing here are based on nlconverter’s FAQ:

1. First, we have to download the notes2mbox.py script from nlconverter’s mercurial repository, as this file is not distributed in the winnlc-alpha-1.zip archive I unzipped previously. Let’s put notes2mbox.py in C:\winnlc-alpha-1\:
   
2. Now we’ll modify the notes2mbox.py script to set the password (via the notesPasswd variable) and location (notesNsfPath variable) of the .nsf file. Here are the modifications I applied:

   --- notes2mbox.py.orig	2010-09-02 13:49:58.000000000 +0200
   +++ notes2mbox.py	2010-09-02 13:51:24.000000000 +0200
   @@ -14,8 +14,8 @@
    import NlconverterLib
   
    #Constantes
   -notesPasswd = "foobar"
   -notesNsfPath = "C:\\archive.nsf"
   +notesPasswd = "XXXXXXXXXXXXX"
   +notesNsfPath = "C:\\winnlc-alpha-1\\kevin-notes-big-backup-part-1.nsf"
   
    #Connection à Notes
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
   

3. Before running the script, we have to register a Notes DLL used by nlconverter:

   regsvr32 "C:\Program Files\Notes\nlsxbe.dll"
   

   
   And make the Python interpreter available system-wide:

   C:\winnlc-alpha-1>SET Path=%Path%;C:\Python26
   

4. Now we can run the notes2mbox.py script:

   C:\winnlc-alpha-1>C:\Python26\python.exe notes2mbox.py
   

If you’re lucky, you’ll get a nice mbox at the end of the process.

But I was not and the notes2mbox.py ended up with the following error:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

As you can see, I tried hard to make nlconverter working, without any success. But this should not stop you to try. In fact I suspect the Lotus Notes installed on my machine to be crippled or corrupted (can’t really tell). So you may be more lucky than me. In any case, feel free to report any success or failure in the comment section below !

You can give the script a list of email headers to ignore while it compares mails between each others. This is particularly helpful to find duplicate mails having the exact same content but different headers/metadatas.

I created this script to clean up a Maildir folder I messed up after moving repeatedly tons of mails from a Lotus Notes database. As you can see below, the same mail imported twice contain a variable header based on the date and time the import was performed:

This variable header make mails looks different from the point of view of the script. That’s explain why I implemented the HEADERS_TO_IGNORE parameter with the default set to X-MIMETrack.

The script is available on my GitHub repository. It was tested on MacOS X 10.6 with python 2.6.2 but should work on other systems and versions as the code is really simple (and stupid).

I used it to create some effects for my 4 el-cheapo Mac-Mah LED wider panels. This script helped me generate column and row presets of my 4×8 pixels LED matrix for some basic colors:

Here are some photos of my preliminary tests at home:

And finally photos of the panels on stage (taken by Toma Heroow during Cool Cavemen’s concert in last november):

As usual, use and hack this script at you own risks, and feel free to send me bug reports and contributions !

This is a simple hack which get commit metadata out of a Subversion repository and generate an iCalendar file containing all commits of a given author. I used it back then to visualize in a calendar my commit activity. Nowadays this script is quite useless as services like Ohloh and GitHub provides great timeline and activity streams. But this script can still be useful for private repositories.

And in the same spirit of this script, I uncovered maildir2ical.py, a script that look in a maildir folder for mails sent by a particular author, then generate an iCal file based on mail dates.

The old gallery was based on autogallery, a e107 plugin. We assume here that both e107 and Zenphoto are well configured and installed at the root of you web hosting space (/www in this case).

The first step is to copy the autogallery album structure, with all its content, to Zenphoto:

cd /www
cp -ax ./e107_plugins/autogallery/Gallery/* ./zenphoto/albums/

Then we delete all previews, thumbnails and XML metadatas, to keep in Zenphoto original assets only:

find ./zenphoto/albums/ -iname "*.xml" | xargs rm -f
find ./zenphoto/albums/ -iname "pv_*" | xargs rm -f
find ./zenphoto/albums/ -iname "th_*" | xargs rm -f

By now, you should be able to play with your medias using Zenphoto’s admin interface.

But if you’re unlucky as I was, you will find a strange bug which break down drag’n'drop album sorting. The fix I found was to remove, in photo filenames, the numerical prefix (and the following dot) set by autogallery to define the sort order. This operation should be performed, before the copy from autogallery to Zenphoto (= the first command in this post). By the way, if you know a one-liner to do this, please, please… share !

To migrate comments, I have no automatic solution. I choose to do this manually, editing the database by hand. In my case it was the quickest way as I only had a dozen of comments to migrate.

And last but not least, if you care about measuring the popularity of your photos, you should consider migrating the view counter associated with each of your media. Don’t worry, this time I wrote a script to take care of it automagically. It will generate a bunch of SQL statements you’ll have to execute on your Zenphoto MySQL database. Here is my “e107 autogallery to Zenphoto hit counter migration script” (nice name isn’t it ? ) that do the job:

#!/usr/bin/python

##############################################################################
#
# Copyright (C) 2008 Kevin Deldycke <kevin@deldycke.com>
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
##############################################################################

"""
  Last update: 2008 aug 21
"""

########### User config ###########

AUTOGALLERY_ALBUM_PATH = "/www/e107_plugins/autogallery/Gallery"
ZENPHOTO_ALBUM_PATH    = "/www/zenphoto/albums"
ZENPHOTO_TABLE_PREFIX  = "zenphoto_"

######## End of user config #######

import os, hashlib
import xml.etree.ElementTree as ET

# Calculate hash of a given file
def getHash(path):
  # Calculate the hash from file raw data
  if not os.path.isfile(path):
    return None
  try:
    file_object = open(path, 'r')
    data = file_object.read()
  except:
    return None
  if not len(data):
    return None
  return hashlib.sha224(data).hexdigest()

# Associate each autogallery photo having a hitcounter greater than 0 with its MD5 hash
def populateHashTable(arg, dirname, names):
  global hash_table
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Get hit count for %s" % file_path
    # Check that the file as a positive hit counter associated with
    xml_file_path = "%s.xml" % file_path
    if not os.path.isfile(xml_file_path):
      continue
    try:
      tree = ET.parse(xml_file_path)
    except:
      continue
    node = tree.find("viewhits")
    if node is None:
      continue
    try:
      hits = int(node.text)
    except:
      continue
    if not hits > 0:
      continue
    # Update hash table with data we care about
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    hash_table[file_hash] = hits + hash_table.get(file_hash, 0)

# Generate hitcount SQL request for each matching file
def generateSQL(arg, dirname, names):
  global sql
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Search hitcounter matching file %s" % file_path
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    if file_hash in hash_table:
      sql += "UPDATE `%simages` SET `hitcounter`=`hitcounter`+%d WHERE `filename`=%r;\n" % (ZENPHOTO_TABLE_PREFIX, hash_table[file_hash], name)

# Core of the script
hash_table = {}
sql        = ""
# Normalize path
source_path = os.path.abspath(AUTOGALLERY_ALBUM_PATH)
dest_path   = os.path.abspath(ZENPHOTO_ALBUM_PATH)

os.path.walk(source_path, populateHashTable, None)
# print repr(hash_table)
os.path.walk(dest_path, generateSQL, None)
print sql

I think code and comments are self-explainatory. And do not forget to update constants at the top of the script to match your installation paths and database’s tables prefix.

And finally, for your information, I tested all of this on following versions:

• e107 0.7.11
• autogallery 2.61
• Zenphoto 1.2
• Python 2.5.2
• Linux server

Changelog:

• Check version of Python (at least v2.4 is required)
• Rename --debug option to --verbose
• Add a --dry-run option for testing
• Remove use of deprecated pexpect methods
• Add and update some error messages

Instinctively, I assumed that my folder was of the Maildir format, and Kmail local mails too. So I tried to copy my ~/Maildir folder from the mail server to my local machine (~/.kde/share/apps/kmail/mail/). And that was the result in Kmail:

It looks good but it’s not: there is no sub-folders !

After some googling, I found what was wrong: my ~/Maildir folder is not a Maildir, but a Maildir++ folder. This kind of folder is handle by popular IMAP MTA like qmail, Dovecot and courier-imap (which was used on the mail server where my ~/Maildir come from). There is some advantages of using the “++” flavor of Maildir over the classic one, like quotas and sub-folders. Unfortunately Kmail is not able to read the Maildir++ folder structure.

To fix this, I’ve created a tiny python script to migrate a Maildir++ folder to Kmail.

How-to use it ? Simply:

1. Download it to your disk,
2. Edit it and change the MAILDIR_SOURCE and KMAILDIR_DEST variables to match your local configuration,
3. Give it execution privileges,
4. Run it !

I advise you to try it first in a safe environment (like under a temporary user account). And don’t forget to backup everything before playing with it: because this script work for me doesn’t mean that it will work for you !

The main update is about the lock file, which I implemented in the last version to keep the script to run twice (or more) in parallel. This is a nice feature to avoid overlapping processes that fight each other to use the same ressources. But in some extreme cases (reboot or power failure during backup, …), the lock file will remain and so will prevent the script to start (until you notice the problem and remove the lock file manually). This new version take care of this problem and is now able to remove the lock automatically if a timeout is reached. It also kill all remaining child processes.

Here is the detailed changelog:

• Auto-kill the script if the backup process take to much time. Timeout can be defined via a constant.
• Clean kill: track all child processes to kill them safely before removing the lock file.
• Require newer versions of python (>= v2.4), rsync (>= v2.6.7) and rdiff-backup (>= v1.1.0).
• Use --preserve-numerical-ids option when adding rdiff-backup increment.
• Keep 15 increments by default instead of 20. This value can be easily changed thanks to a defined constant.
• Remove deleted file first during mirroring and delete outdated increments before adding a new one to gain space. This strategy is safer for target disk with low remaining free space.
• Tell rsync to print human-readable values.

I run this script to backup all the local machines within my LAN. I start the backup process everyday thanks to a cron entry similar to this one:

0 20 * * * root /root/system-backup.py >> /mnt/backup-disk/backup.log

If you need more information about the rsync part the script, please have a look to my previous Remote Backup with rsync article, which detail how-to setup key authentification with ssh.

Change log:

• Each item of the user’s backup_list must specify the type property (FTP, FTPs, SSH, MySQLdump or MySQLdump+ssh).
• The property previously known as site is now host.
• File system structure changed: /ftp-mirror folders renamed to /mirror.
• Add SSH backups.
• The script is able to detect if a SSH connexion can be initiated without a password. This was designed for people who don’t like the idea of storing clear password in the script. Thanks to this feature, you can benefit public key authentication from OpenSSH.
• Use of rsync whenever it’s possible for bandwidth efficiency.
• FTP and FTPs (aka FTP over SSL) are now handled separately: this suppress the default fall-back to FTP if FTPs is not supported by the remote server. This is safer as it doesn’t let lftp make the decision for you to send your clear password on the net.
• All ports are optionnal, no need to specify it you use default ports.
• Add MySQL backups thanks to mysqldump.
• Two mode of MySQL backups: through SSH or direct connection to server.
• A particular database to backup can be specified. Else, all databases are backed up.
• Much more detailed logs that include external command’s output.
• Auto-detect the existence of required external tools and commands at launch.
• Use pexpect lib to simulate user password input.
• Run all external commands in english for consistency.
• Check that the script is running in a posix environnement.
• Fix bug related to directory creation.

If you were using a previous version of my backup script and want to use this updated version, take care of changes, especially the ones describes in the first 3 items of the change log above.