Ultimate guide of Lotus Notes mail migration

The title may sounds pretentious but extracting mails out of Lotus Notes is soooo hard and complicated, that achieving such a task feels like winning an epic battle against the forces of evil.

Anyway. The goal of this post is to help you migrate all your Lotus Notes mails to a more convenient and standard format like maildir or mailbox.

There are several ways of extracting all your mail trapped in Lotus Notes’ proprietary databases.

Method #1: using integrated IMAP service

This is probably the simplest method. It consists in using the Lotus Notes desktop client (were your emails currently resides) as an IMAP client.

Essentially, what you have to do is just to create a secondary account linked to an IMAP server, like Gmail, etc. This works well and is explained in details in this tutorial.

But sometimes your Notes client is behind firewalls and proxys. So you can’t reach the Internet.

And some other times, Lotus Notes clients are crippled and don’t let you create an IMAP connexion. Unfortunately this happened to me:

So I had to found another approach.

Method #2: using nlconverter

This method is explored in details in my previous article.

But again, it seems that the Lotus Notes on my machine was crippled and/or corrupted. The nlconverter GUI gave me this error:

And the command line gave me this:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

After these two failed attempts, I was quite depressed and not far from surrender to the evil power of proprietary software. Then I managed to setup a new (but complicated) strategy.

Method #3: using Lotus Notes client for Mac OS X

This is the only method that worked for me, and basically, is the same as the first one, but on Mac OS X. Additionally, it involves a local mail server. This procedure was tested several times on Leopard (OSX 10.5).

  1. First things first, download the trial version of Lotus Notes client for Mac OS X (unfortunately you have to register) and install it. I used Lotus Notes 7.0.3 for Mac OS X Leopard (10.5):
  2. You’ll be welcomed by a wizard:
  3. On the next screen, enter a dummy name and uncheck the “I want to connect to a Domino server” box:
  4. Then proceed to the next step in which you’ll uncheck “Setup instant messaging“:
  5. In the last screen, uncheck all boxes. We don’t want to setup any other service yet:
  6. Initial setup is now complete:

    You can now launch Lotus Notes:

  7. On first run, there will be the following screen, where you should click on the “No thanks, just give me the defaults“:
  8. You’ll end up on what will be your default Lotus Notes main page from now on:
  9. The next step is to go back to the machine (Windows for me) from which you’re running the Notes client containing all the mails you plan to migrate. From there, export your mail database:

    All details of this operation can be found in this dedicated article.
  10. Then go back to your Mac OS X machine and import your freshly exported .nsf database. This is as simple as opening the file via the File > Database > Open... menu and dialog:

  11. When Notes attempts to open the file, you’ll be prompted by several dialogs regarding the security attached to the database. If you get the “Create Cross Certificate” screen, then just answer “Yes” as below:

    And every time you get an “Execution Security Alert” message, always check the “Start trusting the signer to execute this action” option before clicking “OK“:
  12. The client will then rebuild the index before giving you a plain view of your inbox:

  13. Next step is to setup a local IMAP server:

    As you can see I used Dovecot, and all is explained here.
  14. Now it’s time to create a new account in Lotus Notes to access this local IMAP server. Click on the Address Book in your toolbar and add a new Account:

  15. Here is where you configure Notes to let it be aware of our local server existence. Only the first tab must be changed to your local parameters. You can left the last two tabs untouched:
  16. Open within Notes your local IMAP mailbox. It is found in the workspace, which you can access via the Databases icon on your toolbar:

  17. You’ll be welcomed by a useless help screen:

    Just close it to get your local IMAP mail view:
  18. While trying to opening the local IMAP mailbox, you may encounter this TCPIP port error:

    In this case, please have a look at my other article explaining how to open TCPIP port in Lotus Notes.
  19. For this step, just copy or cut, then paste, mails from your local .nsf database to your local IMAP account:




  20. While playing with copy’n'paste, you may encounter this error:

    A workaround can be found in this article.
  21. Finally, if like me you’ve played a lot with mails during the transfer step above, you may ends up with loads of duplicate mails. In this case have a look at the deduplication script I wrote. It will help you clean-up your Maildir folder.
  22. That’s it ! You now have a standard Maildir of your Lotus Notes mails, located in your user home directory (~/Maildir):

My ultimate action was to convert the Dovecot maildir to Kmail maildir, as I wanted to use Kmail to finally upload everything in Gmail. But you can use anything that suit your needs, like thunderbird or any mail conversion tools.

Conclusion

  • Lotus Notes sucks. Everybody knows that, but I feel liberated saying that ! ;)
  • The smartest thing to do is to avoid Notes like the plague in the first place. Sadly when working for the man, it’s not always possible… :(
  • The only method I found to work for me (the third solution in this article) is far from perfect from my point of view. What I dream about is a 100% automated solution, like a command line utility we can name nsf2maildir. And as I don’t plan to own Apple hardware and software in a near future, such a command should be 100% free software and running on Linux. I really think there is a “market” for a free software component able to read and understand .nsf files. Any motivated volunteer ? ;)

How-to fix Lotus Notes’ disabled TCPIP port error

Today I encountered this error message:

Error logging into server localhost: You must enable the Notes TCPIP port.

This appeared in the fat Lotus Notes client v7.0.3 running on Mac OS X Leopard (10.5).

To fix this issue, first of all, quit Notes. Then locate the Notes’ preference file attached to your current user. Mine was found in my home directory at /Users/kevin/Library/Preferences/Notes Preferences. At the end of this file, add these two directives:

TCPIP=TCP,0,15,0
Ports=TCPIP

Then relaunch Lotus Notes and switch from Island (Disconnected) location to another that will allow your client to listen to the network. In my case, Home (Network Dialup) did the trick:

Problem solved ! :)

Maildir deduplication script in Python

Some months ago I wrote a tiny Python script which scan all folders and sub-folders of a Maildir, then remove duplicate mails.

You can give the script a list of email headers to ignore while it compares mails between each others. This is particularly helpful to find duplicate mails having the exact same content but different headers/metadatas.

I created this script to clean up a Maildir folder I messed up after moving repeatedly tons of mails from a Lotus Notes database. As you can see below, the same mail imported twice contain a variable header based on the date and time the import was performed:

This variable header make mails looks different from the point of view of the script. That’s explain why I implemented the HEADERS_TO_IGNORE parameter with the default set to X-MIMETrack.

The script is available on my GitHub repository. It was tested on MacOS X 10.6 with python 2.6.2 but should work on other systems and versions as the code is really simple (and stupid).

Setup a lightweight IMAP server on MacOS X Leopard with Dovecot

Last week I needed a local IMAP server on MacOS X Leopard (10.5) for temporary testing. After struggling with courier-imap for hours, I’ve finally settled on Dovecot. You’ll see below how easy it is to install and configure it.

We’re lucky, Dovecot is available in Mac Ports, so we can install it easily:

port install dovecot

It’s time to configure it. We start with the default configuration template:

cp /opt/local/etc/dovecot/dovecot-example.conf /opt/local/etc/dovecot/dovecot.conf

Then we can edit the dovecot.conf configuration file as we wish. FYI, here are my modifications:

--- /opt/local/etc/dovecot/dovecot-example.conf	2010-04-23 14:29:52.000000000 +0200
+++ /opt/local/etc/dovecot/dovecot.conf	2010-04-23 14:51:06.000000000 +0200
@@ -21,7 +21,7 @@

 # Protocols we want to be serving: imap imaps pop3 pop3s
 # If you only want to use dovecot-auth, you can set this to "none".
-#protocols = imap imaps
+protocols = imap

 # A space separated list of IP or host addresses where to listen in for
 # connections. "*" listens in all IPv4 interfaces. "[::]" listens in all IPv6
@@ -45,7 +45,7 @@
 # SSL/TLS is used (LOGINDISABLED capability). Note that if the remote IP
 # matches the local IP (ie. you're connecting from the same computer), the
 # connection is considered secure and plaintext authentication is allowed.
-#disable_plaintext_auth = yes
+disable_plaintext_auth = no

 # Should all IMAP and POP3 processes be killed when Dovecot master process
 # shuts down. Setting this to "no" means that Dovecot can be upgraded without
@@ -221,7 +221,7 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-#mail_location =
+mail_location = maildir:~/Maildir

 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.

Before starting Dovecot, we have to create a dummy SSL certificate:

mkdir -p /opt/local/etc/ssl/{certs,private}
openssl req -new -x509 -days 3650 -nodes -out /opt/local/etc/ssl/certs/dovecot.pem -keyout /opt/local/etc/ssl/private/dovecot.pem

And finally, we can launch the Dovecot server itself as root:

dovecot

That’s all !

You can now access your local IMAP server with any client. Here is an example with Thunderbird:

And if you have problems, the first reflex is to read dovecot’s logs:

tail -F /var/log/mail.log

Fuse and sshfs on MacOSX Leopard

I’m used to access distant machine’s file systems via ssh. My favorite environment, KDE, makes things easy thanks to the support of sftp:// URLs via a kio_slave. MacOSX is not as friendly and don’t have any built-in mechanism of that kind.

To get similar features in Leopard, we have to rely on MacFuse and sshfs. I’ll explain here how I’ve installed these components on MacOSX 10.5.

MacFUSE_Banner

First, download the latest MacFuse dmg and install it. FYI, the version I’ve got was MacFuse 2.0.3,2.

Then, download the sshfs executable for Leopard, either the gzipped version or the binary from the SVN as explained in the MacFuse wiki.

From a terminal, rename the binary:

sudo mv ./sshfs-static-leopard ./sshfs

Then allow the binary to be executed and place it in the system:

sudo chmod +x sshfs
sudo install sshfs /usr/local/bin

From now you can test sshfs mounting with the following command:

sshfs user@myserver.net:/folder/ /Network/distant-folder -p 22

I personally had a problem here: sshfs complained about a missing library. I fixed this by downloading the required file from the MacFusion project and copying it beside the sshfs binary:

sudo wget http://www.macfusionapp.org/trac/export/86/trunk/SSHFS/sshnodelay.so
sudo mv ./sshnodelay.so /usr/local/bin/
sudo chmod +x /usr/local/bin/sshnodelay.so

If this fail you can also check:

  • that the current user you’re logged with has access to the distant server with the ssh user@myserver.net command;
  • or that the local mount point exists (you can create it with mkdir -p /Network/distant-folder);
  • and finally, you can add the -o debug option to the sshfs command above to get additional clues.

Now we will automate the mounting of sshfs at every start.

At this point I recommend you to register the root user of your MacOSX system to the distant server:

sudo cat ~/.ssh/id_rsa.pub | sudo ssh -p 22 user@myserver.net "cat >> ~/.ssh/authorized_keys"

If doesn’t exists, we have to create the /etc/fstab to edit it:

sudo touch /etc/fstab
sudo vi /etc/fstab

And add the following directives:

dummy:user@myserver.net:/folder/ /Network/distant-folder sshfs allow_other,auto_cache,reconnect,port=22,follow_symlinks,volname="Distant folder" 0 0

As you can see I’ve added lots of options to accommodate my uses. You can get more informations about sshfs options through traditional help pages:

sshfs --help

MacOSX’s automount daemon will look for a script called mount_sshfs at start. Actually it doesn’t exists on your system, but sshfs command line is compatible with what automount expect. So creating a symbolic link will do the trick:

sudo ln -s /usr/local/bin/sshfs /sbin/mount_sshfs

Finally, we can tell automount to acknowledge all our modifications:

sudo automount -vc