Got “unsized object” errors with Debian’s Mailman ? Try this patch !

Last week I came across a showstopper bug on Mailman 2.1.9-7, the current version of Mailman package distributed with Debian Etch.

Here is the python traceback (from /var/log/mailman/error logfile) I get each time I’ve sent a mail to my brand new mailing-list:

Dec 20 01:20:04 2008 (14275) Uncaught runner exception: len() of unsized object
Dec 20 01:20:04 2008 (14275) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
  File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Handlers/ToDigest.py", line 81, in process
    mbox.AppendMessage(msg)
  File "/usr/lib/mailman/Mailman/Mailbox.py", line 69, in AppendMessage
    g.flatten(msg, unixfrom=True)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten
    self._write(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 136, in _write
    self._write_headers(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 182, in _write_headers
    header_name=h, continuation_ws='\t').encode()
  File "/usr/lib/mailman/pythonlib/email/Header.py", line 412, in encode
    newchunks += self._split(s, charset, targetlen, splitchars)
  File "/usr/lib/mailman/pythonlib/email/Header.py", line 297, in _split
    elen = charset.encoded_header_len(encoded)
  File "/usr/lib/mailman/pythonlib/email/Charset.py", line 354, in encoded_header_len
    raise repr(s)
TypeError: len() of unsized object

Dec 20 01:20:04 2008 (14275) SHUNTING: 1229732404.1069181+dcd89a08bf7911dac2db804b76cd42d20564c71c

Here is the corresponding (anonymized) mail sent to the mailing list from a Gmail account:

Received: by 10.180.244.13 with HTTP; Fri, 19 Dec 2008 16:32:22 -0800 (PST)
Message-ID: <1f7b086f0812192632x7427c0f7u2048609ddd50673@mail.gmail.com>
Date: Sat, 20 Dec 2008 01:32:22 +0100
From: "Kevin" <kevin@my-domain.com>
To: my-ml@lists.my-domain.com
Subject: sqdfqsdfqsfd
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64
Content-Disposition: inline
Delivered-To: kevin@my-domain.com

LS0KS2V2LgogIOKAoiBiYW5kOiBodHRwOi8vY29vbGNhdmVtZW4uY29tCiAg4oCiIGJsb2c6IGh0
dHA6Ly9rZXZpbi5kZWxkeWNrZS5jb20K

And now my hackish tale. Based on a quick look at Mailman’s source code, I made an educated guess that this error is just a side effect of the wrong assumption that the s variable in the Charset.encoded_header_len() method is always a string. So I came up with the following evil patch to handle (gracefully, I hope) the case of s being None.

Here is the resulting patch of my python-fu:

--- /usr/lib/mailman/pythonlib/email/Charset.py.orig   2008-12-28 19:46:23.000000000 +0100
+++ /usr/lib/mailman/pythonlib/email/Charset.py        2008-12-20 01:42:37.000000000 +0100
@@ -351,6 +351,7 @@
             lenqp = email.quopriMIME.header_quopri_len(s)
             return min(lenb64, lenqp) + len(cset) + MISC_LEN
         else:
+            return s is not None and len(str(s)) or 0
             return len(s)

     def header_encode(self, s, convert=False):

And it do the trick ! Of course I can’t guarantee that this patch is the way to definitely fix the bug. And it may corrupt data. So use it only if you’re as crazy as me ! :D

But I know, I know… As a responsible and serious hacker (sigh), I should report this bug to the Debian or Mailman project. But I’m still not familiar with Dedian’s way of reporting bugs (and to be honest, I feel lazy these days :p ). Maybe, one day…

dpkg, APT & Aptitude commands

  • List all installed packages:
    dpkg -l
    
  • List all recently installed packages:
    zcat -f /var/log/dpkg.log* | grep "\ install\ " | sort
    
  • Install a package from a lower-priority repository, like the backport repository:
    apt-get -t squeeze-backports install my-package
    
  • Force reinstallation of a package:
    apt-get -d --reinstall install my-package
    dpkg --install --force-confmiss /var/cache/apt/archives/my-package.deb
    
  • Clean aptitude local cache:
    apt-get clean
    
  • Uninstall a package throughly (both program files and configuration):
    apt-get remove --purge my_package
    
  • Force removal of a package while ignoring all dependencies:
    dpkg --remove --force-depends libsomething
    
  • Remove orphaned pakages:
    deborphan | xargs apt-get -y remove --purge
    
  • Show the changelog of a package (here, the linux kernel of Ubuntu):
    aptitude changelog linux-generic
    
  • Which package contain a given file:
    apt-file search file_to_search
    
  • Get the list of files of a package:
    apt-file list package_name
    
  • Remove dpkg lock file:
    rm /var/lib/dpkg/lock
    
  • Hold a package with either dpkg or aptitude:
    echo "kdenlive hold" | dpkg --set-selections
    
    aptitude hold kdenlive
    
  • Unhold a package:
    echo "kdenlive install" | dpkg --set-selections
    
    aptitude unhold kdenlive
    
  • List holded packages:
    dpkg --get-selections | grep hold
    

Give away of the day: free 23andMe $100 vouchers !

I ordered a 23andMe v1 kit (Time’s invention of the year 2008) in early march. The version 2 of the “personal genetic kit” was released some months ago.

As a late v1 customer, they offered me a free v2 upgrade. Even if I had to pay the shipping ($70 as european), that’s really fair as it accounts for $399. They also gave me two $100 coupons. So if you do the maths, they offered me the difference between the price I paid for the v1 ($999) and the actual value of the v2 kit ($399). That’s an honest commercial act ! :)

Alas, the coupons expires December 31 2008, and I don’t plan to take advantages of the discount. So I’ll give them away to the firsts of you who send me a mail or post a comment.

How-to fix wireless DHCP on Mandriva 2009.0

In two words: dhcp_client sucks !

And now the long story…

Since the upgrade to Mandriva 2008.1, wifi stopped working on my laptop. I tried to install the 2008.1 on several machines. I tried to connect on different access points. I lowered security on the access point. I tried eveything. On desperation, I even tried to boot Windows to check that hardware was ok ! And the only log I had was this:

SIOCETHTOOL: Operation not supported

After all these tests, I was convinced that the problem had something to do with the distribution itself. Maybe a firmware issue or a bad combination of packages…

Then came the 2009.0 release. I though that an upgrade will cure my malediction. Indeed. Nothing new on wireless side. My wifi was still broken. Until I came across a tip on a random forum (I don’t remember which one) suggesting that dhcp_client could be the culprit.

So I replaced it by dhcpcd, and against all expectations, it worked !

And to not be annoyed by dhcp_client in the future, it’s wise to definitely remove it:

urpmi dhcp_client

Google Apps’ video chat comes with secure Gmail sessions

The story was spread by all top tech blogs last week: Google’s Gmail now features a video chat. And it requires the installation of a dedicated plugin.

Alas, there is no such plugin for any other platform except “Windows XP and later” (according the official website) and Macs (as read on the official blog). So it’s a quite sad news for us Linux users. Indeed, I’m confident about a future seamless integration into the free software ecosystem, as the Gmail’s video chat is based on a stack of open (or soon-to-be, according the recent controversy) standards and protocols: XMPP/Jingle, h264/SVC & RTP.

Anyways, that’s not the main purpose of this post.

I just wanted to point out an update that was not reported by the news: as soon as it was officially made available for the public, the brand new video feature was released for Google Apps’ Gmail too.

Not only that, Google also backported to Apps’ Gmail the much awaited HTTPs option that allow you to force secure encryption of your sessions:

These two updates are quite interesting to note. I long as I remember (and I might be wrong), Google Apps components were always out-of-sync with their legacy equivalent. So this maybe a sign of change in a really good direction for Google Apps users ! :)

How-to: e107 autogallery to Zenphoto migration

These past few days I was working on the Cool Cavemen’s photo gallery to move it to a shiny new one, powered by Zenphoto. In this post I will roughly describe how I’ve done it, code and commands included.

The old gallery was based on autogallery, a e107 plugin. We assume here that both e107 and Zenphoto are well configured and installed at the root of you web hosting space (/www in this case).

The first step is to copy the autogallery album structure, with all its content, to Zenphoto:

cd /www
cp -ax ./e107_plugins/autogallery/Gallery/* ./zenphoto/albums/

Then we delete all previews, thumbnails and XML metadatas, to keep in Zenphoto original assets only:

find ./zenphoto/albums/ -iname "*.xml" | xargs rm -f
find ./zenphoto/albums/ -iname "pv_*" | xargs rm -f
find ./zenphoto/albums/ -iname "th_*" | xargs rm -f

By now, you should be able to play with your medias using Zenphoto’s admin interface.

But if you’re unlucky as I was, you will find a strange bug which break down drag’n'drop album sorting. The fix I found was to remove, in photo filenames, the numerical prefix (and the following dot) set by autogallery to define the sort order. This operation should be performed, before the copy from autogallery to Zenphoto (= the first command in this post). By the way, if you know a one-liner to do this, please, please… share ! :)

To migrate comments, I have no automatic solution. I choose to do this manually, editing the database by hand. In my case it was the quickest way as I only had a dozen of comments to migrate.

And last but not least, if you care about measuring the popularity of your photos, you should consider migrating the view counter associated with each of your media. Don’t worry, this time I wrote a script to take care of it automagically. It will generate a bunch of SQL statements you’ll have to execute on your Zenphoto MySQL database. Here is my “e107 autogallery to Zenphoto hit counter migration script” (nice name isn’t it ? ;) ) that do the job:

#!/usr/bin/python

##############################################################################
#
# Copyright (C) 2008 Kevin Deldycke <kevin@deldycke.com>
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
##############################################################################

"""
  Last update: 2008 aug 21
"""

########### User config ###########

AUTOGALLERY_ALBUM_PATH = "/www/e107_plugins/autogallery/Gallery"
ZENPHOTO_ALBUM_PATH    = "/www/zenphoto/albums"
ZENPHOTO_TABLE_PREFIX  = "zenphoto_"

######## End of user config #######

import os, hashlib
import xml.etree.ElementTree as ET

# Calculate hash of a given file
def getHash(path):
  # Calculate the hash from file raw data
  if not os.path.isfile(path):
    return None
  try:
    file_object = open(path, 'r')
    data = file_object.read()
  except:
    return None
  if not len(data):
    return None
  return hashlib.sha224(data).hexdigest()

# Associate each autogallery photo having a hitcounter greater than 0 with its MD5 hash
def populateHashTable(arg, dirname, names):
  global hash_table
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Get hit count for %s" % file_path
    # Check that the file as a positive hit counter associated with
    xml_file_path = "%s.xml" % file_path
    if not os.path.isfile(xml_file_path):
      continue
    try:
      tree = ET.parse(xml_file_path)
    except:
      continue
    node = tree.find("viewhits")
    if node is None:
      continue
    try:
      hits = int(node.text)
    except:
      continue
    if not hits > 0:
      continue
    # Update hash table with data we care about
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    hash_table[file_hash] = hits + hash_table.get(file_hash, 0)

# Generate hitcount SQL request for each matching file
def generateSQL(arg, dirname, names):
  global sql
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Search hitcounter matching file %s" % file_path
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    if file_hash in hash_table:
      sql += "UPDATE `%simages` SET `hitcounter`=`hitcounter`+%d WHERE `filename`=%r;\n" % (ZENPHOTO_TABLE_PREFIX, hash_table[file_hash], name)

# Core of the script
hash_table = {}
sql        = ""
# Normalize path
source_path = os.path.abspath(AUTOGALLERY_ALBUM_PATH)
dest_path   = os.path.abspath(ZENPHOTO_ALBUM_PATH)

os.path.walk(source_path, populateHashTable, None)
# print repr(hash_table)
os.path.walk(dest_path, generateSQL, None)
print sql

I think code and comments are self-explainatory. And do not forget to update constants at the top of the script to match your installation paths and database’s tables prefix.

And finally, for your information, I tested all of this on following versions:

  • e107 0.7.11
  • autogallery 2.61
  • Zenphoto 1.2
  • Python 2.5.2
  • Linux server