How-to merge Mailman mailing-lists

Let’s say I have an old inactive mailing list (which ID is old-ml) for which I want to merge its archives to another one (called active-ml).

To do so, I have to merge the two mbox files holding all mails since the creation of these mailing-lists. I first tried to use cat to concatenate the two mbox files be it didn’t work.

Luckily, I found a Python script to merge 2 mbox files while sorting all mails by date. Here is how I uses it:

$ cd /var/lib/mailman/archives/private
$ wget http://mail.python.org/pipermail/mailman-users/attachments/20080322/80455064/attachment.txt --output-document=mbmerge.py
$ python ./mbmerge.py ./old-ml.mbox/old-ml.mbox ./active-ml.mbox/active-ml.mbox > ./active-ml.mbox/active-ml.mbox.new

Then I switched the current mbox with the one generated above and asked mailman to regenerate the static HTML archives:

$ cd /var/lib/mailman/archives/private/active-ml.mbox/
$ mv active-ml.mbox active-ml.mbox.backup
$ mv active-ml.mbox.new active-ml.mbox
$ chown list:list active-ml.mbox*
$ /usr/lib/mailman/bin/arch --wipe active-ml

Of course this will only merge mail archives. You still have to merge your old mailing lists parameters (including membership) manually.

At last, when everything is clean to you, you can safely remove your old mailing-list:

$ rmlist -a old-ml
$ /var/lib/mailman/bin/genaliases

Mailman migration

Last week I detailed how I configured Mailman with Exim and Nginx on a Debian Squeeze. Here are some more notes on how I migrated my mailing lists from my old server (Lenny with Mailman 2.1.11) to the new Mailman installation (Squeeze with Mailman 2.1.13).

First, I remove the default mailman meta-list as I will retrieve the one from the old server:

$ /etc/init.d/mailman stop
$ rmlist -a mailman
$ /var/lib/mailman/bin/genaliases

Then I copy mailing-list data from the old server to the new:

$ rsync --progress -vrae "ssh -C" /var/lib/mailman/lists    root@new.example.com:/var/lib/mailman/
$ rsync --progress -vrae "ssh -C" /var/lib/mailman/archives root@new.example.com:/var/lib/mailman/
$ rsync --progress -vrae "ssh -C" /var/lib/mailman/data     root@new.example.com:/var/lib/mailman/

Back to our new server, fix some rights, check all lists are there, and run the automatic update:

$ chown -R list:list /var/lib/mailman/
$ /etc/init.d/mailman start
$ list_lists
$ /var/lib/mailman/bin/update

Now let Mailman check its databases and fix permission:

$ check_db -a -v
$ check_perms -f -v

At this point you may get this error in your /var/log/exim4/mainlog:

2011-09-13 10:06:09 failed to expand condition "${lookup{$local_part@$domain}lsearch{/var/lib/mailman/data/virtual-mailman}{1}{0}}" for mailman_router router: failed to open /var/lib/mailman/data/virtual-mailman for linear search: Permission denied (euid=101 egid=103)

This can be fixed with (source):

$ chgrp Debian-exim /var/lib/mailman/data/virtual-mailman

You may also encounter this error:

2011-09-13 10:06:09 H=mail-xxx-xxxx.google.com [209.85.000.000] F=<kevin@example.com> rejected RCPT <kev-test@lists.example.com>: Unrouteable address

In this case regenerating Mailman aliases should fix the issue:

$ /var/lib/mailman/bin/genaliases

By the way, to test that Exim is routing mails as expected, your can use the following command:

$ exim -bt kev-test@lists.example.com
R: system_aliases for kev-test@lists.example.com
R: mailman_router for kev-test@lists.example.com
kev-test@lists.example.com
  router = mailman_router, transport = mailman_transport

Last problem I had was mails did not reached my server. Everytime I send something from Gmail to a list, I got back error mails saying this:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 relay not permitted (state 14).

I fixed this issue by updating my SPF record on the example.com domain from:

v=spf1 a mx ~all

to:

v=spf1 a mx ptr ~all

How-to setup Mailman + Nginx + Exim on Debian Squeeze

Before going further, please take note that I start this tutorial assuming that you already have a minimal Exim setup running on your Debian machine.

Mailman

Now that you have the context, let’s proceed with Mailman install:

$ aptitude install mailman

During the installation, you’ll be prompted about the languages files you want Mailman web interface support. English is enough for me.

Now Mailman requires a meta-mailing-list from which it will send all mails related to subscription, reminders and all:

$ newlist mailman kevin@deldycke.com

You’ll then be prompted for a password.

After that, Mailman will provide you with a list of directives to add to /etc/aliases:

mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

This update is not necessary, as Exim will handle them automatically.

You can now restart the Mailman server:

$ /etc/init.d/mailman start

Oh, and the first time you’ll run Mailman, do a start as above, not a restart, else you’ll end up with this error:

Restarting Mailman master qrunner: mailmanctl PID unreadable in: /var/run/mailman/mailman.pid
[Errno 2] No such file or directory: '/var/run/mailman/mailman.pid'
Is qrunner even running?

If everything is alright, you’ll receive a mail similar to this one:

Nginx

Now we have to configure our HTTP server to make the administration interface available from the web. If Apache is the recommended server to use with Mailman, Nginx is already running on my machine, so let’s use it instead.

First, as explained on Nginx wiki we need to install fcgiwrap:

$ aptitude install fcgiwrap

Then we have to create an Nginx configuration file dedicated to Mailman. Assuming we want all mailing-lists managed under the lists.example.com domain, here are the directives you have to put in a new /etc/nginx/sites-available/mailman file:

server {
  server_name lists.example.com;

  root /usr/lib/cgi-bin;

  location = / {
    rewrite ^ /mailman/listinfo permanent;
  }

  location / {
    rewrite ^ /mailman$uri;
  }

  location /mailman {
    include /etc/nginx/fastcgi_params;
    # Fastcgi socket
    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
    # Disable gzip (it makes scripts feel slower since they have to complete
    # before getting gzipped)
    gzip off;
  }

  location /images/mailman {
    alias /var/lib/mailman/icons;
  }

  location /pipermail {
    alias /var/lib/mailman/archives/public;
    autoindex on;
  }
}

server {
  server_name *.lists.example.com .lists.example.org .lists.example.net;
  rewrite ^ http://lists.example.com$request_uri? permanent;
}

The configuration above is a mix between the one available on Nginx wiki and the /usr/share/doc/fcgiwrap/examples/nginx.conf example file that come with the Debian package.

All we have to do now is to activate the configuration above and restart our CGI and HTTP server:

$ ln -s /etc/nginx/sites-available/mailman /etc/nginx/sites-enabled/
$ /etc/init.d/fcgiwrap restart
$ /etc/init.d/nginx restart

If everything’s OK, going to http://lists.example.com will show you this:

Exim

Now we have to setup the MTA. All informations here are coming from the documentation you can find on your Debian system in /usr/share/doc/mailman/README.Exim4.Debian.gz.

First, we have to update /etc/mailman/mm_cfg.py (the global Mailman configuration file). We’ll aligned there the default URLs, hosts and MTA-related parameters:

--- /etc/mailman/mm_cfg.py.orig    2011-08-31 22:28:53.000000000 +0200
+++ /etc/mailman/mm_cfg.py 2011-09-07 22:43:41.000000000 +0200
@@ -57,16 +57,16 @@
 #-------------------------------------------------------------
 # If you change these, you have to configure your http server
 # accordingly (Alias and ScriptAlias directives in most httpds)
-DEFAULT_URL_PATTERN = 'http://%s/cgi-bin/mailman/'
-PRIVATE_ARCHIVE_URL = '/cgi-bin/mailman/private'
+DEFAULT_URL_PATTERN = 'http://%s/mailman/'
+PRIVATE_ARCHIVE_URL = '/mailman/private'
 IMAGE_LOGOS         = '/images/mailman/'

 #-------------------------------------------------------------
 # Default domain for email addresses of newly created MLs
-DEFAULT_EMAIL_HOST = 'server123.example.net'
+DEFAULT_EMAIL_HOST = 'lists.example.com'
 #-------------------------------------------------------------
 # Default host for web interface of newly created MLs
-DEFAULT_URL_HOST   = 'server123.example.net'
+DEFAULT_URL_HOST   = 'lists.example.com'
 #-------------------------------------------------------------
 # Required when setting any of its arguments.
 add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST)
@@ -94,7 +94,10 @@
 # Uncomment if you use Postfix virtual domains (but not
 # postfix-to-mailman.py), but be sure to see
 # /usr/share/doc/mailman/README.Debian first.
-# MTA='Postfix'
+MTA = 'Postfix'
+POSTFIX_ALIAS_CMD = '/bin/true'
+POSTFIX_MAP_CMD = 'chgrp Debian-exim'
+POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.example.com']

 #-------------------------------------------------------------
 # Uncomment if you want to filter mail with SpamAssassin. For

Then we have to update the Exim configuration template. If like me you haven’t choose to split configuration into small files, here are the modifications you have to add to /etc/exim4/exim4.conf.template:

--- /etc/exim4/exim4.conf.template.orig 2011-09-07 23:34:53.000000000 +0200
+++ /etc/exim4/exim4.conf.template       2011-09-07 23:44:45.000000000 +0200
@@ -395,6 +395,21 @@
 ### end main/03_exim4-config_tlsoptions
 #####################################################
 #####################################################
+### main/04_local_mailman_macros
+#####################################################
+# Home dir for your Mailman installation -- aka Mailman's prefix
+# directory.
+MAILMAN_HOME=/var/lib/mailman
+MAILMAN_WRAP=MAILMAN_HOME/mail/mailman
+
+# User and group for Mailman, should match your --with-mail-gid
+# switch to Mailman's configure script.
+MAILMAN_USER=list
+MAILMAN_GROUP=daemon
+#####################################################
+### end main/04_local_mailman_macros
+#####################################################
+#####################################################
 ### main/90_exim4-config_log_selector
 #####################################################

@@ -1371,6 +1386,44 @@
 ### end router/900_exim4-config_local_user
 #####################################################
 #####################################################
+### router/970_local_mailman
+#####################################################
+# Messages get sent out with
+# envelope from "mailman-bounces@virtual_domain"
+# But mailman doesn't put such addresses
+# in the aliases. Recognise these here.
+mailman_workaround:
+  debug_print = "R: mailman_workaround for $local_part@$domain"
+  domains = +local_domains
+  require_files = MAILMAN_HOME/lists/$local_part/config.pck
+  driver = accept
+  local_parts = mailman
+  local_part_suffix_optional
+  local_part_suffix = -bounces : -bounces+* : \
+           -confirm+* : -join : -leave : \
+           -subscribe : -unsubscribe : \
+           -owner : -request : -admin : -loop
+  transport = mailman_transport
+  group = MAILMAN_GROUP
+
+# Mailman lists
+mailman_router:
+  debug_print = "R: mailman_router for $local_part@$domain"
+  domains = +local_domains
+  condition = ${lookup{$local_part@$domain}lsearch{MAILMAN_HOME/data/virtual-mailman}{1}{0}}
+  require_files = MAILMAN_HOME/lists/$local_part/config.pck
+  driver = accept
+  local_part_suffix_optional
+  local_part_suffix = -bounces : -bounces+* : \
+                      -confirm+* : -join : -leave : \
+                      -subscribe : -unsubscribe : \
+                      -owner : -request : -admin : -loop
+  transport = mailman_transport
+  group = MAILMAN_GROUP
+#####################################################
+### end router/970_local_mailman
+#####################################################
+#####################################################
 ### router/mmm_mail4root
 #####################################################

@@ -1689,6 +1742,25 @@
 ### end transport/35_exim4-config_address_directory
 #####################################################
 #####################################################
+### transport/40_local_mailman
+#####################################################
+mailman_transport:
+  debug_print = "T: mailman_transport for $local_part@$domain"
+  driver = pipe
+  command = MAILMAN_WRAP \
+            '${if def:local_part_suffix \
+                  {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
+                  {post}}' \
+            $local_part
+  current_directory = MAILMAN_HOME
+  home_directory = MAILMAN_HOME
+  user = MAILMAN_USER
+  group = MAILMAN_GROUP
+  freeze_exec_fail = true
+#####################################################
+### end transport/40_local_mailman
+#####################################################
+#####################################################
 ### retry/00_exim4-config_header
 #####################################################

Don’t apply this diff as-is, as the original file contain the modifications I previously made to let Exim use Gmail to send mails.

Then we have to update the Exim meta-configuration that is stored in /etc/exim4/update-exim4.conf.conf. There we specify our host (lists.example.com) and public IP address (123.456.78.90):

dc_eximconfig_configtype='smarthost'
dc_other_hostnames='lists.example.com'
dc_local_interfaces='127.0.0.1 ; ::1 ; 123.456.78.90'
dc_readhost='lists.example.com'
dc_relay_domains='lists.example.com'
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='smtp.gmail.com:587'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='mail_spool'

Finally, our hostname must be a FQDN, so we have to add it to /etc/hosts:

--- /etc/hosts.orig        2011-09-12 13:52:19.000000000 +0200
+++ /etc/hosts     2011-09-12 12:21:31.000000000 +0200
@@ -1,7 +1,7 @@
 # Do not remove the following line, or various programs
 # that require network functionality will fail.
 127.0.0.1      localhost.localdomain localhost
-123.456.78.90   server123.example.net
+123.456.78.90   server123.example.net lists.example.com
 # The following lines are desirable for IPv6 capable hosts
 #(added automatically by netbase upgrade)
 ::1     ip6-localhost ip6-loopback

Then we have to regenerate Exim’s configuration before restarting Mailman:

$ update-exim4.conf --verbose
$ /etc/init.d/exim4 restart
$ /etc/init.d/mailman restart

Testing

You can now test your setup by creating a test mailing-list:

$ newlist kev-test

Now subscribe some test users and play with this mailing-list.

By monitoring /var/log/mailman/error, you’ll maybe run into this error:

IOError: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/kev-test.mbox/kev-test.mbox'

This can be easily fixed with:

$ chown -R list /var/lib/mailman/archives/private/

Once you’re convinced that Mailman is working as expected, you can remove your temporary test mailing-list, and regenerate aliases to clean things up:

$ rmlist -a  kev-test
$ /var/lib/mailman/bin/genaliases

Munin monitoring

Finally, if like me you use Munin to monitor your machine, then it’s a good idea to let it graph some Mailman usage:

$ wget http://exchange.munin-monitoring.org/plugins/mailman-queue-check/version/2/download --output-document=/usr/share/munin/plugins/mailman-queue-check
$ wget http://exchange.munin-monitoring.org/plugins/mailman_subscribers/version/3/download --output-document=/usr/share/munin/plugins/mailman_subscribers
$ ln -s /usr/share/munin/plugins/mailman-queue-check /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mailman_subscribers /etc/munin/plugins/
$ echo "[mailman*]
user root
" > /etc/munin/plugin-conf.d/mailman
$ chmod 755 /usr/share/munin/plugins/mailman*
$ /etc/init.d/munin-node restart

Got “unsized object” errors with Debian’s Mailman ? Try this patch !

Last week I came across a showstopper bug on Mailman 2.1.9-7, the current version of Mailman package distributed with Debian Etch.

Here is the python traceback (from /var/log/mailman/error logfile) I get each time I’ve sent a mail to my brand new mailing-list:

Dec 20 01:20:04 2008 (14275) Uncaught runner exception: len() of unsized object
Dec 20 01:20:04 2008 (14275) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
  File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Handlers/ToDigest.py", line 81, in process
    mbox.AppendMessage(msg)
  File "/usr/lib/mailman/Mailman/Mailbox.py", line 69, in AppendMessage
    g.flatten(msg, unixfrom=True)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten
    self._write(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 136, in _write
    self._write_headers(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 182, in _write_headers
    header_name=h, continuation_ws='\t').encode()
  File "/usr/lib/mailman/pythonlib/email/Header.py", line 412, in encode
    newchunks += self._split(s, charset, targetlen, splitchars)
  File "/usr/lib/mailman/pythonlib/email/Header.py", line 297, in _split
    elen = charset.encoded_header_len(encoded)
  File "/usr/lib/mailman/pythonlib/email/Charset.py", line 354, in encoded_header_len
    raise repr(s)
TypeError: len() of unsized object

Dec 20 01:20:04 2008 (14275) SHUNTING: 1229732404.1069181+dcd89a08bf7911dac2db804b76cd42d20564c71c

Here is the corresponding (anonymized) mail sent to the mailing list from a Gmail account:

Received: by 10.180.244.13 with HTTP; Fri, 19 Dec 2008 16:32:22 -0800 (PST)
Message-ID: <1f7b086f0812192632x7427c0f7u2048609ddd50673@mail.gmail.com>
Date: Sat, 20 Dec 2008 01:32:22 +0100
From: "Kevin" <kevin@my-domain.com>
To: my-ml@lists.my-domain.com
Subject: sqdfqsdfqsfd
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64
Content-Disposition: inline
Delivered-To: kevin@my-domain.com

LS0KS2V2LgogIOKAoiBiYW5kOiBodHRwOi8vY29vbGNhdmVtZW4uY29tCiAg4oCiIGJsb2c6IGh0
dHA6Ly9rZXZpbi5kZWxkeWNrZS5jb20K

And now my hackish tale. Based on a quick look at Mailman’s source code, I made an educated guess that this error is just a side effect of the wrong assumption that the s variable in the Charset.encoded_header_len() method is always a string. So I came up with the following evil patch to handle (gracefully, I hope) the case of s being None.

Here is the resulting patch of my python-fu:

--- /usr/lib/mailman/pythonlib/email/Charset.py.orig   2008-12-28 19:46:23.000000000 +0100
+++ /usr/lib/mailman/pythonlib/email/Charset.py        2008-12-20 01:42:37.000000000 +0100
@@ -351,6 +351,7 @@
             lenqp = email.quopriMIME.header_quopri_len(s)
             return min(lenb64, lenqp) + len(cset) + MISC_LEN
         else:
+            return s is not None and len(str(s)) or 0
             return len(s)

     def header_encode(self, s, convert=False):

And it do the trick ! Of course I can’t guarantee that this patch is the way to definitely fix the bug. And it may corrupt data. So use it only if you’re as crazy as me ! :D

But I know, I know… As a responsible and serious hacker (sigh), I should report this bug to the Debian or Mailman project. But I’m still not familiar with Dedian’s way of reporting bugs (and to be honest, I feel lazy these days :p ). Maybe, one day…