Ultimate guide of Lotus Notes mail migration

The title may sounds pretentious but extracting mails out of Lotus Notes is soooo hard and complicated, that achieving such a task feels like winning an epic battle against the forces of evil.

Anyway. The goal of this post is to help you migrate all your Lotus Notes mails to a more convenient and standard format like maildir or mailbox.

There are several ways of extracting all your mail trapped in Lotus Notes’ proprietary databases.

Method #1: using integrated IMAP service

This is probably the simplest method. It consists in using the Lotus Notes desktop client (were your emails currently resides) as an IMAP client.

Essentially, what you have to do is just to create a secondary account linked to an IMAP server, like Gmail, etc. This works well and is explained in details in this tutorial.

But sometimes your Notes client is behind firewalls and proxys. So you can’t reach the Internet.

And some other times, Lotus Notes clients are crippled and don’t let you create an IMAP connexion. Unfortunately this happened to me:

So I had to found another approach.

Method #2: using nlconverter

This method is explored in details in my previous article.

But again, it seems that the Lotus Notes on my machine was crippled and/or corrupted. The nlconverter GUI gave me this error:

And the command line gave me this:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

After these two failed attempts, I was quite depressed and not far from surrender to the evil power of proprietary software. Then I managed to setup a new (but complicated) strategy.

Method #3: using Lotus Notes client for Mac OS X

This is the only method that worked for me, and basically, is the same as the first one, but on Mac OS X. Additionally, it involves a local mail server. This procedure was tested several times on Leopard (OSX 10.5).

  1. First things first, download the trial version of Lotus Notes client for Mac OS X (unfortunately you have to register) and install it. I used Lotus Notes 7.0.3 for Mac OS X Leopard (10.5):
  2. You’ll be welcomed by a wizard:
  3. On the next screen, enter a dummy name and uncheck the “I want to connect to a Domino server” box:
  4. Then proceed to the next step in which you’ll uncheck “Setup instant messaging“:
  5. In the last screen, uncheck all boxes. We don’t want to setup any other service yet:
  6. Initial setup is now complete:

    You can now launch Lotus Notes:

  7. On first run, there will be the following screen, where you should click on the “No thanks, just give me the defaults“:
  8. You’ll end up on what will be your default Lotus Notes main page from now on:
  9. The next step is to go back to the machine (Windows for me) from which you’re running the Notes client containing all the mails you plan to migrate. From there, export your mail database:

    All details of this operation can be found in this dedicated article.
  10. Then go back to your Mac OS X machine and import your freshly exported .nsf database. This is as simple as opening the file via the File > Database > Open... menu and dialog:

  11. When Notes attempts to open the file, you’ll be prompted by several dialogs regarding the security attached to the database. If you get the “Create Cross Certificate” screen, then just answer “Yes” as below:

    And every time you get an “Execution Security Alert” message, always check the “Start trusting the signer to execute this action” option before clicking “OK“:
  12. The client will then rebuild the index before giving you a plain view of your inbox:

  13. Next step is to setup a local IMAP server:

    As you can see I used Dovecot, and all is explained here.
  14. Now it’s time to create a new account in Lotus Notes to access this local IMAP server. Click on the Address Book in your toolbar and add a new Account:

  15. Here is where you configure Notes to let it be aware of our local server existence. Only the first tab must be changed to your local parameters. You can left the last two tabs untouched:
  16. Open within Notes your local IMAP mailbox. It is found in the workspace, which you can access via the Databases icon on your toolbar:

  17. You’ll be welcomed by a useless help screen:

    Just close it to get your local IMAP mail view:
  18. While trying to opening the local IMAP mailbox, you may encounter this TCPIP port error:

    In this case, please have a look at my other article explaining how to open TCPIP port in Lotus Notes.
  19. For this step, just copy or cut, then paste, mails from your local .nsf database to your local IMAP account:




  20. While playing with copy’n'paste, you may encounter this error:

    A workaround can be found in this article.
  21. Finally, if like me you’ve played a lot with mails during the transfer step above, you may ends up with loads of duplicate mails. In this case have a look at the deduplication script I wrote. It will help you clean-up your Maildir folder.
  22. That’s it ! You now have a standard Maildir of your Lotus Notes mails, located in your user home directory (~/Maildir):

My ultimate action was to convert the Dovecot maildir to Kmail maildir, as I wanted to use Kmail to finally upload everything in Gmail. But you can use anything that suit your needs, like thunderbird or any mail conversion tools.

Conclusion

  • Lotus Notes sucks. Everybody knows that, but I feel liberated saying that ! ;)
  • The smartest thing to do is to avoid Notes like the plague in the first place. Sadly when working for the man, it’s not always possible… :(
  • The only method I found to work for me (the third solution in this article) is far from perfect from my point of view. What I dream about is a 100% automated solution, like a command line utility we can name nsf2maildir. And as I don’t plan to own Apple hardware and software in a near future, such a command should be 100% free software and running on Linux. I really think there is a “market” for a free software component able to read and understand .nsf files. Any motivated volunteer ? ;)

How-to fix Lotus Notes’ disabled TCPIP port error

Today I encountered this error message:

Error logging into server localhost: You must enable the Notes TCPIP port.

This appeared in the fat Lotus Notes client v7.0.3 running on Mac OS X Leopard (10.5).

To fix this issue, first of all, quit Notes. Then locate the Notes’ preference file attached to your current user. Mine was found in my home directory at /Users/kevin/Library/Preferences/Notes Preferences. At the end of this file, add these two directives:

TCPIP=TCP,0,15,0
Ports=TCPIP

Then relaunch Lotus Notes and switch from Island (Disconnected) location to another that will allow your client to listen to the network. In my case, Home (Network Dialup) did the trick:

Problem solved ! :)

Setup a lightweight IMAP server on MacOS X Leopard with Dovecot

Last week I needed a local IMAP server on MacOS X Leopard (10.5) for temporary testing. After struggling with courier-imap for hours, I’ve finally settled on Dovecot. You’ll see below how easy it is to install and configure it.

We’re lucky, Dovecot is available in Mac Ports, so we can install it easily:

port install dovecot

It’s time to configure it. We start with the default configuration template:

cp /opt/local/etc/dovecot/dovecot-example.conf /opt/local/etc/dovecot/dovecot.conf

Then we can edit the dovecot.conf configuration file as we wish. FYI, here are my modifications:

--- /opt/local/etc/dovecot/dovecot-example.conf	2010-04-23 14:29:52.000000000 +0200
+++ /opt/local/etc/dovecot/dovecot.conf	2010-04-23 14:51:06.000000000 +0200
@@ -21,7 +21,7 @@

 # Protocols we want to be serving: imap imaps pop3 pop3s
 # If you only want to use dovecot-auth, you can set this to "none".
-#protocols = imap imaps
+protocols = imap

 # A space separated list of IP or host addresses where to listen in for
 # connections. "*" listens in all IPv4 interfaces. "[::]" listens in all IPv6
@@ -45,7 +45,7 @@
 # SSL/TLS is used (LOGINDISABLED capability). Note that if the remote IP
 # matches the local IP (ie. you're connecting from the same computer), the
 # connection is considered secure and plaintext authentication is allowed.
-#disable_plaintext_auth = yes
+disable_plaintext_auth = no

 # Should all IMAP and POP3 processes be killed when Dovecot master process
 # shuts down. Setting this to "no" means that Dovecot can be upgraded without
@@ -221,7 +221,7 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-#mail_location =
+mail_location = maildir:~/Maildir

 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.

Before starting Dovecot, we have to create a dummy SSL certificate:

mkdir -p /opt/local/etc/ssl/{certs,private}
openssl req -new -x509 -days 3650 -nodes -out /opt/local/etc/ssl/certs/dovecot.pem -keyout /opt/local/etc/ssl/private/dovecot.pem

And finally, we can launch the Dovecot server itself as root:

dovecot

That’s all !

You can now access your local IMAP server with any client. Here is an example with Thunderbird:

And if you have problems, the first reflex is to read dovecot’s logs:

tail -F /var/log/mail.log

Fuse and sshfs on MacOSX Leopard

I’m used to access distant machine’s file systems via ssh. My favorite environment, KDE, makes things easy thanks to the support of sftp:// URLs via a kio_slave. MacOSX is not as friendly and don’t have any built-in mechanism of that kind.

To get similar features in Leopard, we have to rely on MacFuse and sshfs. I’ll explain here how I’ve installed these components on MacOSX 10.5.

MacFUSE_Banner

First, download the latest MacFuse dmg and install it. FYI, the version I’ve got was MacFuse 2.0.3,2.

Then, download the sshfs executable for Leopard, either the gzipped version or the binary from the SVN as explained in the MacFuse wiki.

From a terminal, rename the binary:

sudo mv ./sshfs-static-leopard ./sshfs

Then allow the binary to be executed and place it in the system:

sudo chmod +x sshfs
sudo install sshfs /usr/local/bin

From now you can test sshfs mounting with the following command:

sshfs user@myserver.net:/folder/ /Network/distant-folder -p 22

I personally had a problem here: sshfs complained about a missing library. I fixed this by downloading the required file from the MacFusion project and copying it beside the sshfs binary:

sudo wget http://www.macfusionapp.org/trac/export/86/trunk/SSHFS/sshnodelay.so
sudo mv ./sshnodelay.so /usr/local/bin/
sudo chmod +x /usr/local/bin/sshnodelay.so

If this fail you can also check:

  • that the current user you’re logged with has access to the distant server with the ssh user@myserver.net command;
  • or that the local mount point exists (you can create it with mkdir -p /Network/distant-folder);
  • and finally, you can add the -o debug option to the sshfs command above to get additional clues.

Now we will automate the mounting of sshfs at every start.

At this point I recommend you to register the root user of your MacOSX system to the distant server:

sudo cat ~/.ssh/id_rsa.pub | sudo ssh -p 22 user@myserver.net "cat >> ~/.ssh/authorized_keys"

If doesn’t exists, we have to create the /etc/fstab to edit it:

sudo touch /etc/fstab
sudo vi /etc/fstab

And add the following directives:

dummy:user@myserver.net:/folder/ /Network/distant-folder sshfs allow_other,auto_cache,reconnect,port=22,follow_symlinks,volname="Distant folder" 0 0

As you can see I’ve added lots of options to accommodate my uses. You can get more informations about sshfs options through traditional help pages:

sshfs --help

MacOSX’s automount daemon will look for a script called mount_sshfs at start. Actually it doesn’t exists on your system, but sshfs command line is compatible with what automount expect. So creating a symbolic link will do the trick:

sudo ln -s /usr/local/bin/sshfs /sbin/mount_sshfs

Finally, we can tell automount to acknowledge all our modifications:

sudo automount -vc

Plone 3.2 (and Python 2.4) on Mac OS X Leopard

In this post I’ll show you how I installed Python 2.4 on Mac OS X Leopard and how, starting from a bare Python environnement, we can build a stand-alone Zope 2.10 instance with Plone 3.2 thanks to zc.buildout.

If your goal is to play with or evaluate Plone (or if you don’t know what zc.buildout is), then this article will lead you to some troubles and pain. The Plone community maintain a collection of out-of-the-box and ready-to-use installers for all major platforms. So before going further, I strongly advise you to use the official Plone installer for Mac OS X. This is much simpler and faster than the process described below.

Now that all Plone newcomers are redirected to the right place, I can start to talk about how to install Python 2.4 on Mac OS X. Why the 2.4 release ? That’s simple: Mac OS X 10.5 (Leopard) ships with Python 2.5, but Plone 3 requires Python 2.4.

To get Python 2.4 on your machine, you can install it from its source code. But this is too much work for me. There should be a way to do it easier and faster… And there is.

Browsing the net, I found the repository of the “fat python” project, were you can find a universal binary installer for Panther. I’ve just installed it on my brand new Mac OS X 10.5.7 and it seems to works perfectly:
python-2.4-shell-in-mac-os-x-leopard

Now that the most annoying part (to me) is done, we can install Plone via zc.buildout.

Before going further, you need to have a machine that is able to compile code, which mean Apple’s developer tools must be installed locally. These softwares are available for free on the second DVD that ships with every Mac OS X copy.

First we create our project directory, then we download, from its SVN repository, the bootstrap code of buildout:

$ mkdir -p ~/plone-vanilla
$ cd ~/plone-vanilla
$ curl http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py --output ./bootstrap.py

Then we create our buildout config file and edit it:

$ touch ./buildout.cfg
$ vi ./buildout.cfg

The buildout.cfg file should contain the following directives, which tell buildout to install Plone 3.2.3, Zope 2.10.8 and all their dependencies:

[buildout]
find-links = http://dist.plone.org

http://download.zope.org/ppix

http://download.zope.org/distribution

http://effbot.org/downloads

http://dist.plone.org/release/3.2.3

extends    = http://dist.plone.org/release/3.2.3/versions.cfg
versions   = versions
parts      = zope-server
             zope-instance
eggs       = PIL
             Plone

[zope-server]
recipe               = plone.recipe.zope2install
url                  = http://www.zope.org/Products/Zope/2.10.8/Zope-2.10.8-final.tgz
fake-zope-eggs       = true
additional-fake-eggs = ZConfig
                       pytz

[zope-instance]
recipe           = plone.recipe.zope2instance
zope2-location   = ${zope-server:location}
user             = admin:admin
debug-mode       = on
verbose-security = on
eggs             = ${buildout:eggs}

Now let’s build our Plone and Zope environnement:

$ python2.4 ./bootstrap.py
$ ./bin/buildout

At the end, if your build process didn’t fail, you’ll be able to start your Zope server:

$ ./bin/zope-instance
program: /Users/kevin/plone-vanilla/parts/zope-instance/bin/runzope
daemon manager not running
zopectl> start
. daemon process started, pid=17585
zopectl> logtail
------
2009-07-20T20:42:26 INFO ZServer HTTP server started at Mon Jul 20 20:42:26 2009
	Hostname: 0.0.0.0
	Port: 8080
------
2009-07-20T20:42:35 INFO Marshall libxml2-python not available. Unable to register libxml2 based marshallers.
------
2009-07-20T20:42:59 INFO Zope Ready to handle requests

Then you can fire up Safari, go to http://localhost:8080/manage (default Zope config), and login as the admin user (password: admin):
safari-zope-login

Create a Plone site:
plone-site-creation

Check that your using the right version of Plone in the control panel:
plone-323-control-panel