Tag Archives: E-mail

How-to setup Mailman + Nginx + Exim on Debian Squeeze

Posted on by
1

Before going further, please take note that I start this tutorial assuming that you already have a minimal Exim setup running on your Debian machine.

Mailman

Now that you have the context, let’s proceed with Mailman install:

$ aptitude install mailman

During the installation, you’ll be prompted about the languages files you want Mailman web interface support. English is enough for me.

Now Mailman requires a meta-mailing-list from which it will send all mails related to subscription, reminders and all:

$ newlist mailman kevin@deldycke.com

You’ll then be prompted for a password.

After that, Mailman will provide you with a list of directives to add to /etc/aliases:

mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

This update is not necessary, as Exim will handle them automatically.

You can now restart the Mailman server:

$ /etc/init.d/mailman start

Oh, and the first time you’ll run Mailman, do a start as above, not a restart, else you’ll end up with this error:

Restarting Mailman master qrunner: mailmanctl PID unreadable in: /var/run/mailman/mailman.pid
[Errno 2] No such file or directory: '/var/run/mailman/mailman.pid'
Is qrunner even running?

If everything is alright, you’ll receive a mail similar to this one:

Nginx

Now we have to configure our HTTP server to make the administration interface available from the web. If Apache is the recommended server to use with Mailman, Nginx is already running on my machine, so let’s use it instead.

First, as explained on Nginx wiki we need to install fcgiwrap:

$ aptitude install fcgiwrap

Then we have to create an Nginx configuration file dedicated to Mailman. Assuming we want all mailing-lists managed under the lists.example.com domain, here are the directives you have to put in a new /etc/nginx/sites-available/mailman file:

server {
  server_name lists.example.com;

  root /usr/lib/cgi-bin;

  location = / {
    rewrite ^ /mailman/listinfo permanent;
  }

  location / {
    rewrite ^ /mailman$uri;
  }

  location /mailman {
    include /etc/nginx/fastcgi_params;
    # Fastcgi socket
    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
    # Disable gzip (it makes scripts feel slower since they have to complete
    # before getting gzipped)
    gzip off;
  }

  location /images/mailman {
    alias /var/lib/mailman/icons;
  }

  location /pipermail {
    alias /var/lib/mailman/archives/public;
    autoindex on;
  }
}

server {
  server_name *.lists.example.com .lists.example.org .lists.example.net;
  rewrite ^ http://lists.example.com$request_uri? permanent;
}

The configuration above is a mix between the one available on Nginx wiki and the /usr/share/doc/fcgiwrap/examples/nginx.conf example file that come with the Debian package.

All we have to do now is to activate the configuration above and restart our CGI and HTTP server:

$ ln -s /etc/nginx/sites-available/mailman /etc/nginx/sites-enabled/
$ /etc/init.d/fcgiwrap restart
$ /etc/init.d/nginx restart

If everything’s OK, going to http://lists.example.com will show you this:

Exim

Now we have to setup the MTA. All informations here are coming from the documentation you can find on your Debian system in /usr/share/doc/mailman/README.Exim4.Debian.gz.

First, we have to update /etc/mailman/mm_cfg.py (the global Mailman configuration file). We’ll aligned there the default URLs, hosts and MTA-related parameters:

--- /etc/mailman/mm_cfg.py.orig    2011-08-31 22:28:53.000000000 +0200
+++ /etc/mailman/mm_cfg.py 2011-09-07 22:43:41.000000000 +0200
@@ -57,16 +57,16 @@
 #-------------------------------------------------------------
 # If you change these, you have to configure your http server
 # accordingly (Alias and ScriptAlias directives in most httpds)
-DEFAULT_URL_PATTERN = 'http://%s/cgi-bin/mailman/'
-PRIVATE_ARCHIVE_URL = '/cgi-bin/mailman/private'
+DEFAULT_URL_PATTERN = 'http://%s/mailman/'
+PRIVATE_ARCHIVE_URL = '/mailman/private'
 IMAGE_LOGOS         = '/images/mailman/'

 #-------------------------------------------------------------
 # Default domain for email addresses of newly created MLs
-DEFAULT_EMAIL_HOST = 'server123.example.net'
+DEFAULT_EMAIL_HOST = 'lists.example.com'
 #-------------------------------------------------------------
 # Default host for web interface of newly created MLs
-DEFAULT_URL_HOST   = 'server123.example.net'
+DEFAULT_URL_HOST   = 'lists.example.com'
 #-------------------------------------------------------------
 # Required when setting any of its arguments.
 add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST)
@@ -94,7 +94,10 @@
 # Uncomment if you use Postfix virtual domains (but not
 # postfix-to-mailman.py), but be sure to see
 # /usr/share/doc/mailman/README.Debian first.
-# MTA='Postfix'
+MTA = 'Postfix'
+POSTFIX_ALIAS_CMD = '/bin/true'
+POSTFIX_MAP_CMD = 'chgrp Debian-exim'
+POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.example.com']

 #-------------------------------------------------------------
 # Uncomment if you want to filter mail with SpamAssassin. For

Then we have to update the Exim configuration template. If like me you haven’t choose to split configuration into small files, here are the modifications you have to add to /etc/exim4/exim4.conf.template:

--- /etc/exim4/exim4.conf.template.orig 2011-09-07 23:34:53.000000000 +0200
+++ /etc/exim4/exim4.conf.template       2011-09-07 23:44:45.000000000 +0200
@@ -395,6 +395,21 @@
 ### end main/03_exim4-config_tlsoptions
 #####################################################
 #####################################################
+### main/04_local_mailman_macros
+#####################################################
+# Home dir for your Mailman installation -- aka Mailman's prefix
+# directory.
+MAILMAN_HOME=/var/lib/mailman
+MAILMAN_WRAP=MAILMAN_HOME/mail/mailman
+
+# User and group for Mailman, should match your --with-mail-gid
+# switch to Mailman's configure script.
+MAILMAN_USER=list
+MAILMAN_GROUP=daemon
+#####################################################
+### end main/04_local_mailman_macros
+#####################################################
+#####################################################
 ### main/90_exim4-config_log_selector
 #####################################################

@@ -1371,6 +1386,44 @@
 ### end router/900_exim4-config_local_user
 #####################################################
 #####################################################
+### router/970_local_mailman
+#####################################################
+# Messages get sent out with
+# envelope from "mailman-bounces@virtual_domain"
+# But mailman doesn't put such addresses
+# in the aliases. Recognise these here.
+mailman_workaround:
+  debug_print = "R: mailman_workaround for $local_part@$domain"
+  domains = +local_domains
+  require_files = MAILMAN_HOME/lists/$local_part/config.pck
+  driver = accept
+  local_parts = mailman
+  local_part_suffix_optional
+  local_part_suffix = -bounces : -bounces+* : \
+           -confirm+* : -join : -leave : \
+           -subscribe : -unsubscribe : \
+           -owner : -request : -admin : -loop
+  transport = mailman_transport
+  group = MAILMAN_GROUP
+
+# Mailman lists
+mailman_router:
+  debug_print = "R: mailman_router for $local_part@$domain"
+  domains = +local_domains
+  condition = ${lookup{$local_part@$domain}lsearch{MAILMAN_HOME/data/virtual-mailman}{1}{0}}
+  require_files = MAILMAN_HOME/lists/$local_part/config.pck
+  driver = accept
+  local_part_suffix_optional
+  local_part_suffix = -bounces : -bounces+* : \
+                      -confirm+* : -join : -leave : \
+                      -subscribe : -unsubscribe : \
+                      -owner : -request : -admin : -loop
+  transport = mailman_transport
+  group = MAILMAN_GROUP
+#####################################################
+### end router/970_local_mailman
+#####################################################
+#####################################################
 ### router/mmm_mail4root
 #####################################################

@@ -1689,6 +1742,25 @@
 ### end transport/35_exim4-config_address_directory
 #####################################################
 #####################################################
+### transport/40_local_mailman
+#####################################################
+mailman_transport:
+  debug_print = "T: mailman_transport for $local_part@$domain"
+  driver = pipe
+  command = MAILMAN_WRAP \
+            '${if def:local_part_suffix \
+                  {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
+                  {post}}' \
+            $local_part
+  current_directory = MAILMAN_HOME
+  home_directory = MAILMAN_HOME
+  user = MAILMAN_USER
+  group = MAILMAN_GROUP
+  freeze_exec_fail = true
+#####################################################
+### end transport/40_local_mailman
+#####################################################
+#####################################################
 ### retry/00_exim4-config_header
 #####################################################

Don’t apply this diff as-is, as the original file contain the modifications I previously made to let Exim use Gmail to send mails.

Then we have to update the Exim meta-configuration that is stored in /etc/exim4/update-exim4.conf.conf. There we specify our host (lists.example.com) and public IP address (123.456.78.90):

dc_eximconfig_configtype='smarthost'
dc_other_hostnames='lists.example.com'
dc_local_interfaces='127.0.0.1 ; ::1 ; 123.456.78.90'
dc_readhost='lists.example.com'
dc_relay_domains='lists.example.com'
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='smtp.gmail.com:587'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='mail_spool'

Finally, our hostname must be a FQDN, so we have to add it to /etc/hosts:

--- /etc/hosts.orig        2011-09-12 13:52:19.000000000 +0200
+++ /etc/hosts     2011-09-12 12:21:31.000000000 +0200
@@ -1,7 +1,7 @@
 # Do not remove the following line, or various programs
 # that require network functionality will fail.
 127.0.0.1      localhost.localdomain localhost
-123.456.78.90   server123.example.net
+123.456.78.90   server123.example.net lists.example.com
 # The following lines are desirable for IPv6 capable hosts
 #(added automatically by netbase upgrade)
 ::1     ip6-localhost ip6-loopback

Then we have to regenerate Exim’s configuration before restarting Mailman:

$ update-exim4.conf --verbose
$ /etc/init.d/exim4 restart
$ /etc/init.d/mailman restart

Testing

You can now test your setup by creating a test mailing-list:

$ newlist kev-test

Now subscribe some test users and play with this mailing-list.

By monitoring /var/log/mailman/error, you’ll maybe run into this error:

IOError: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/kev-test.mbox/kev-test.mbox'

This can be easily fixed with:

$ chown -R list /var/lib/mailman/archives/private/

Once you’re convinced that Mailman is working as expected, you can remove your temporary test mailing-list, and regenerate aliases to clean things up:

$ rmlist -a  kev-test
$ /var/lib/mailman/bin/genaliases

Munin monitoring

Finally, if like me you use Munin to monitor your machine, then it’s a good idea to let it graph some Mailman usage:

$ wget http://exchange.munin-monitoring.org/plugins/mailman-queue-check/version/2/download --output-document=/usr/share/munin/plugins/mailman-queue-check
$ wget http://exchange.munin-monitoring.org/plugins/mailman_subscribers/version/3/download --output-document=/usr/share/munin/plugins/mailman_subscribers
$ ln -s /usr/share/munin/plugins/mailman-queue-check /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mailman_subscribers /etc/munin/plugins/
$ echo "[mailman*]
user root
" > /etc/munin/plugin-conf.d/mailman
$ chmod 755 /usr/share/munin/plugins/mailman*
$ /etc/init.d/munin-node restart

Ultimate guide of Lotus Notes mail migration

Posted on by
4

The title may sounds pretentious but extracting mails out of Lotus Notes is soooo hard and complicated, that achieving such a task feels like winning an epic battle against the forces of evil.

Anyway. The goal of this post is to help you migrate all your Lotus Notes mails to a more convenient and standard format like maildir or mailbox.

There are several ways of extracting all your mail trapped in Lotus Notes’ proprietary databases.

Method #1: using integrated IMAP service

This is probably the simplest method. It consists in using the Lotus Notes desktop client (were your emails currently resides) as an IMAP client.

Essentially, what you have to do is just to create a secondary account linked to an IMAP server, like Gmail, etc. This works well and is explained in details in this tutorial.

But sometimes your Notes client is behind firewalls and proxys. So you can’t reach the Internet.

And some other times, Lotus Notes clients are crippled and don’t let you create an IMAP connexion. Unfortunately this happened to me:

So I had to found another approach.

Method #2: using nlconverter

This method is explored in details in my previous article.

But again, it seems that the Lotus Notes on my machine was crippled and/or corrupted. The nlconverter GUI gave me this error:

And the command line gave me this:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

After these two failed attempts, I was quite depressed and not far from surrender to the evil power of proprietary software. Then I managed to setup a new (but complicated) strategy.

Method #3: using Lotus Notes client for Mac OS X

This is the only method that worked for me, and basically, is the same as the first one, but on Mac OS X. Additionally, it involves a local mail server. This procedure was tested several times on Leopard (OSX 10.5).

  1. First things first, download the trial version of Lotus Notes client for Mac OS X (unfortunately you have to register) and install it. I used Lotus Notes 7.0.3 for Mac OS X Leopard (10.5):
  2. You’ll be welcomed by a wizard:
  3. On the next screen, enter a dummy name and uncheck the “I want to connect to a Domino server” box:
  4. Then proceed to the next step in which you’ll uncheck “Setup instant messaging“:
  5. In the last screen, uncheck all boxes. We don’t want to setup any other service yet:
  6. Initial setup is now complete:

    You can now launch Lotus Notes:

  7. On first run, there will be the following screen, where you should click on the “No thanks, just give me the defaults“:
  8. You’ll end up on what will be your default Lotus Notes main page from now on:
  9. The next step is to go back to the machine (Windows for me) from which you’re running the Notes client containing all the mails you plan to migrate. From there, export your mail database:

    All details of this operation can be found in this dedicated article.
  10. Then go back to your Mac OS X machine and import your freshly exported .nsf database. This is as simple as opening the file via the File > Database > Open... menu and dialog:

  11. When Notes attempts to open the file, you’ll be prompted by several dialogs regarding the security attached to the database. If you get the “Create Cross Certificate” screen, then just answer “Yes” as below:

    And every time you get an “Execution Security Alert” message, always check the “Start trusting the signer to execute this action” option before clicking “OK“:
  12. The client will then rebuild the index before giving you a plain view of your inbox:

  13. Next step is to setup a local IMAP server:

    As you can see I used Dovecot, and all is explained here.
  14. Now it’s time to create a new account in Lotus Notes to access this local IMAP server. Click on the Address Book in your toolbar and add a new Account:

  15. Here is where you configure Notes to let it be aware of our local server existence. Only the first tab must be changed to your local parameters. You can left the last two tabs untouched:
  16. Open within Notes your local IMAP mailbox. It is found in the workspace, which you can access via the Databases icon on your toolbar:

  17. You’ll be welcomed by a useless help screen:

    Just close it to get your local IMAP mail view:
  18. While trying to opening the local IMAP mailbox, you may encounter this TCPIP port error:

    In this case, please have a look at my other article explaining how to open TCPIP port in Lotus Notes.
  19. For this step, just copy or cut, then paste, mails from your local .nsf database to your local IMAP account:




  20. While playing with copy’n'paste, you may encounter this error:

    A workaround can be found in this article.
  21. Finally, if like me you’ve played a lot with mails during the transfer step above, you may ends up with loads of duplicate mails. In this case have a look at the deduplication script I wrote. It will help you clean-up your Maildir folder.
  22. That’s it ! You now have a standard Maildir of your Lotus Notes mails, located in your user home directory (~/Maildir):

My ultimate action was to convert the Dovecot maildir to Kmail maildir, as I wanted to use Kmail to finally upload everything in Gmail. But you can use anything that suit your needs, like thunderbird or any mail conversion tools.

Conclusion

  • Lotus Notes sucks. Everybody knows that, but I feel liberated saying that ! ;)
  • The smartest thing to do is to avoid Notes like the plague in the first place. Sadly when working for the man, it’s not always possible… :(
  • The only method I found to work for me (the third solution in this article) is far from perfect from my point of view. What I dream about is a 100% automated solution, like a command line utility we can name nsf2maildir. And as I don’t plan to own Apple hardware and software in a near future, such a command should be 100% free software and running on Linux. I really think there is a “market” for a free software component able to read and understand .nsf files. Any motivated volunteer ? ;)

Maildir deduplication script in Python

Posted on by
16

Some months ago I wrote a tiny Python script which scan all folders and sub-folders of a Maildir, then remove duplicate mails.

You can give the script a list of email headers to ignore while it compares mails between each others. This is particularly helpful to find duplicate mails having the exact same content but different headers/metadatas.

I created this script to clean up a Maildir folder I messed up after moving repeatedly tons of mails from a Lotus Notes database. As you can see below, the same mail imported twice contain a variable header based on the date and time the import was performed:

This variable header make mails looks different from the point of view of the script. That’s explain why I implemented the HEADERS_TO_IGNORE parameter with the default set to X-MIMETrack.

The script is available on my GitHub repository. It was tested on MacOS X 10.6 with python 2.6.2 but should work on other systems and versions as the code is really simple (and stupid).

How-to export/backup Lotus Notes mails

Posted on by
4

You are using Lotus Notes as your mail platform. Unfortunately your mailbox has a quota you’ve already reached and you need space. A solution consist in exporting regularly your mails on your local machine to free up your inbox. Here is a little article documenting the export procedure using the fat desktop client.

If screenshots were taken with a french version, instructions given here are for the english one. This will give you enough clues to perform the export whatever the localisation is. The Lotus Notes version I used was the 7.0.2 release.

So first, let’s start Notes and open your mailbox. You should be on a screen similar to this one:

Then, go to the File › Database › New Copy menu:

And you’ll get an export screen that’ll let you choose where to create a local copy of your database:

This will generate a .nsf file containing all your current mail.

Now that you have a backup, you are free to delete all your mails in Lotus Notes. By following this procedure regulary, you can create yearly or monthly archives of you mails without reaching the mailbox quota ! For example, this is how my local archive folder looks like:

Lotus Notes’ Rich Text to MIME conversion error

Posted on by
4

Today I encountered a strange error while using Lotus Notes. I had a “Cannot convert Notes Rich Text message to MIME message” error:

This was triggered when I tried to move certain mails from one account to another. And to add insult to injury, this nasty and dangerous error will make you loose data.

Let’s say you want to cut and paste a batch of 10 mails. Then that error occurs while Notes paste the 3rd message. It means you’ll loose the last 7 messages of your batch. Why ? The 10 messages will be removed from their original location on cutting, and the last 7 messages will be trapped in the copy buffer. Isn’t that a reasonable reason to hate Lotus Notes ?

Anyway. After several tests and experiments, I finally found the common property shared by all those reluctant messages. They all have inline images embedded in the body of the mail, like the one below:

In mail edit mode, you can get properties of these objects and get confirmation that they are inline images:

As you can see above, the edit mode lets you manipulate (cut, copy, paste, …) these embedded pictures. Let’s take advantage of this to fix our initial issue.

Here is my procedure to make these mails pass through the conversion error:

  1. In Notes’ edit mode, cut all inline pictures, one picture at a time;
  2. For each cutted picture, paste it as a new image in the image editor of your choice (Gimp did the trick for me);
  3. Save each image on your local disk;
  4. Now that all inline images are removed from the original mail, attach (but don’t paste) to it all the images you saved in the previous step;
  5. Finally, save mail modifications in Notes and you’ll be able to move the mail without the conversion error.

This is really dirty, and isn’t bearable past a few mails. But that’s the only solution I found so far. Of course if you have a superior/automated way to address this lame bug, I’ll be happy to hear that ! :)