e107 Importer WordPress plugin v1.0 released !

After 3 years in limbo, here is a new stable version of my e107 Importer plugin for WordPress, proudly numbered 1.0 ! :)

This is the first time this plugin is available on the official WordPress plugin repository. This means easy upgrades for you ! :)

I’ve heavily updated the plugin to use the latest WordPress import framework, so everything is now cleanly integrated.

This version was tested with e107 v0.7.24 and WordPress 3.1-RC3.

Here is the changelog:

A note for developpers: the reference code base is now located on GitHub. That’s were all new code must be commited. WordPress’ Subversion repository is only a mirror.

If any question, please read the FAQ first.

Blocking e107 dDOS attack with fail2ban

Last month, a new security vulnerability was discovered in e107. If a fix was released quickly, some instances on the web were left unpatched. These sites are easy target for hackers script-kiddies, and a generalized dDOS attack was carry out on every e107 websites out there.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

How-to: e107 autogallery to Zenphoto migration

These past few days I was working on the Cool Cavemen’s photo gallery to move it to a shiny new one, powered by Zenphoto. In this post I will roughly describe how I’ve done it, code and commands included.

The old gallery was based on autogallery, a e107 plugin. We assume here that both e107 and Zenphoto are well configured and installed at the root of you web hosting space (/www in this case).

The first step is to copy the autogallery album structure, with all its content, to Zenphoto:

cd /www
cp -ax ./e107_plugins/autogallery/Gallery/* ./zenphoto/albums/

Then we delete all previews, thumbnails and XML metadatas, to keep in Zenphoto original assets only:

find ./zenphoto/albums/ -iname "*.xml" | xargs rm -f
find ./zenphoto/albums/ -iname "pv_*" | xargs rm -f
find ./zenphoto/albums/ -iname "th_*" | xargs rm -f

By now, you should be able to play with your medias using Zenphoto’s admin interface.

But if you’re unlucky as I was, you will find a strange bug which break down drag’n'drop album sorting. The fix I found was to remove, in photo filenames, the numerical prefix (and the following dot) set by autogallery to define the sort order. This operation should be performed, before the copy from autogallery to Zenphoto (= the first command in this post). By the way, if you know a one-liner to do this, please, please… share ! :)

To migrate comments, I have no automatic solution. I choose to do this manually, editing the database by hand. In my case it was the quickest way as I only had a dozen of comments to migrate.

And last but not least, if you care about measuring the popularity of your photos, you should consider migrating the view counter associated with each of your media. Don’t worry, this time I wrote a script to take care of it automagically. It will generate a bunch of SQL statements you’ll have to execute on your Zenphoto MySQL database. Here is my “e107 autogallery to Zenphoto hit counter migration script” (nice name isn’t it ? ;) ) that do the job:

#!/usr/bin/python

##############################################################################
#
# Copyright (C) 2008 Kevin Deldycke <kevin@deldycke.com>
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
##############################################################################

"""
  Last update: 2008 aug 21
"""

########### User config ###########

AUTOGALLERY_ALBUM_PATH = "/www/e107_plugins/autogallery/Gallery"
ZENPHOTO_ALBUM_PATH    = "/www/zenphoto/albums"
ZENPHOTO_TABLE_PREFIX  = "zenphoto_"

######## End of user config #######

import os, hashlib
import xml.etree.ElementTree as ET

# Calculate hash of a given file
def getHash(path):
  # Calculate the hash from file raw data
  if not os.path.isfile(path):
    return None
  try:
    file_object = open(path, 'r')
    data = file_object.read()
  except:
    return None
  if not len(data):
    return None
  return hashlib.sha224(data).hexdigest()

# Associate each autogallery photo having a hitcounter greater than 0 with its MD5 hash
def populateHashTable(arg, dirname, names):
  global hash_table
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Get hit count for %s" % file_path
    # Check that the file as a positive hit counter associated with
    xml_file_path = "%s.xml" % file_path
    if not os.path.isfile(xml_file_path):
      continue
    try:
      tree = ET.parse(xml_file_path)
    except:
      continue
    node = tree.find("viewhits")
    if node is None:
      continue
    try:
      hits = int(node.text)
    except:
      continue
    if not hits > 0:
      continue
    # Update hash table with data we care about
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    hash_table[file_hash] = hits + hash_table.get(file_hash, 0)

# Generate hitcount SQL request for each matching file
def generateSQL(arg, dirname, names):
  global sql
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Search hitcounter matching file %s" % file_path
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    if file_hash in hash_table:
      sql += "UPDATE `%simages` SET `hitcounter`=`hitcounter`+%d WHERE `filename`=%r;\n" % (ZENPHOTO_TABLE_PREFIX, hash_table[file_hash], name)

# Core of the script
hash_table = {}
sql        = ""
# Normalize path
source_path = os.path.abspath(AUTOGALLERY_ALBUM_PATH)
dest_path   = os.path.abspath(ZENPHOTO_ALBUM_PATH)

os.path.walk(source_path, populateHashTable, None)
# print repr(hash_table)
os.path.walk(dest_path, generateSQL, None)
print sql

I think code and comments are self-explainatory. And do not forget to update constants at the top of the script to match your installation paths and database’s tables prefix.

And finally, for your information, I tested all of this on following versions:

  • e107 0.7.11
  • autogallery 2.61
  • Zenphoto 1.2
  • Python 2.5.2
  • Linux server

e107 to WordPress migration : v0.9 plug-in released

9 months after the last one, here is the new version (v0.9) of my e107 to WordPress import plug-in !

Change log:

  • “One-click migration” instead of multiple step process (more user-friendly),
  • Better error management (a must-have for precise bug reports),
  • Replace all links to old content with permalinks (increased SEO),
  • Better database management,
  • Work with latest WordPress v2.3.2 and e107 v0.7.11,
  • Code cleaned up ! ;)

FeedBurner and e107 integration

In the context of my plan to move an e107-based website to WordPress, I need to take care of my RSS subscribers. To let people (and search engines) get my content via old URLs, I will use Apache redirections to do this transparently and permanently. My final goal is to have a WordPress website with all RSS feeds (blog posts and comments) managed by FeedBurner, to gather statistics about my audience.

Actually there is plenty of feeds format available in e107 (RSS 1.0, RSS 2.0, Atom and RDF) and one feed can be accessed through multiple URLs. We will reduce this incredible mess by using RSS 2.0 feeds only and redirect all others to it.

First, check that the e107 RSS feed plugin is activated. Then create an account on FeedBurner and setup there two feeds, one for your website’s news and another one for comments. Based on default e107 parameters, your news feed URL is like http://www.my-domain.com/e107_plugins/rss_menu/rss.php?1.2 and comments feed like http://www.my-domain.com/e107_plugins/rss_menu/rss.php?5.2.

Then, create (or edit) the http://www.my-domain.com/.htaccess file, and add following code:

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !FeedBurner [NC]
RewriteCond %{QUERY_STRING} ^(5|Comments)
RewriteRule e107_plugins/rss_menu/rss\.php http://feeds.feedburner.com/myfeed-comments? [R=301,L]

RewriteCond %{HTTP_USER_AGENT} !FeedBurner [NC]
RewriteCond %{QUERY_STRING} ^(1|News|.*)
RewriteRule e107_plugins/rss_menu/rss\.php http://feeds.feedburner.com/myfeed? [R=301,L]

This code is inspired by the one written by Mike Atlas, who had a similar issue and wanted to outsource his e107 forum RSS feeds to FeedBurner.

The first rewrite rule will redirect all URLs that start with http://www.my-domain.com/e107_plugins/rss_menu/rss.php?5 or http://www.my-domain.com/e107_plugins/rss_menu/rss.php?Comments to http://feeds.feedburner.com/myfeed-comments.

The second rewrite rule will redirect all other URLs that start with http://www.my-domain.com/e107_plugins/rss_menu/rss.php (including http://www.my-domain.com/e107_plugins/rss_menu/rss.php?1 and http://www.my-domain.com/e107_plugins/rss_menu/rss.php?News) to http://feeds.feedburner.com/myfeed.

That’s all ! Thanks to this server-side redirection, nobody will notice that the feeds have moved and no subscriber will be bothered to update their aggregator.

In my case, the only remaining task to do is to move my e107 website to WordPress then install FeedSmith plugin. But that’s another story… ;)

e107 to WordPress v0.8: import images and preferences

The 8th version of my e107 to WordPress import script is out ! This version is quite special because this is the first one that support all features I planned to implement in the road map I write for the first alpha release.

Here is the changelog:

  • Import images embedded in e107 news and custom pages,
  • Import e107 site preferences (like site name and description),
  • Better import of user profile data,
  • An existing user on the blog can be detected and updated automatically,
  • Fix the profanity filter bug,
  • Tested with latest WordPress 2.1.3.