This is a special release as it ends the active development cycle. I officially declare this plugin unmaintained. I don’t plan to work on it again.

The 1.4 will be the last version. Unless you send me code contribution, you’ll not see a 1.5 version.

The reason behind this decision is simple and straightforward: I’ve recently migrated my last e107 website to WordPress. Now that e107 is no longer part of my life, I have no interest in spending time and energy working on this plugin.

So if you plan to get rid of your e107 sites in favor of WordPress, do the migration now, as the plugin will self-deteriorate as WordPress API gets updated.

Before settling on its current name, the Cool Cavemen project was referred to by its members as The Ultimate Band (talk about rock-star egos…). Here is a screenshot of the theme I did for e107:

In fact the original HTML mockup this theme is based on still exists. It is dated back to November 1st, 2004, which is now the official Cool Cavemen anniversary. The theme above was created two weeks later.

When I created the Cool Cavemen’s site, I choose e107. Back then I perceived it to be the only Open Source PHP-based CMS having the best balance between a clean and a powerful theme engine. That was my opinion before decided to switch to WordPress.

At the end of November ’04, our theme was updated to this:

The header above is based on a photo of a green laser, that was taken by Cool Cavemen’s guitarist.

2005 started with an updated version of the theme, featuring a photo of Cool Cavemen’s first gig. They were only three on stage, our bass player was still drumming at the time:

In February we finally had our official photo featuring all members of the band ! But it was cold outside so we added some fur to keep our website warm:

I spent the next months trying to build my own version of the Holy Grail: a perfect CSS-based 3-columns fluid layout (with a middle column placed in the top of the HTML). This explain Eric Mayer‘s quote in these mockups and the references to the Skidoo Too template:

I never found the Holy Grail, and the tests above remained unseen by the public. Tired by this journey, I never touched the theme again.

Until September 2005 when I updated it to this:

Notice the box in the top of the right column, which was designed to publish a new track every week. The code behind this box is available in another article.

So that was the last major version of the theme. Basically our e107 site looked that way for most of its life.

In November 2005 I attempted to reboot the theme. I made these 3 propositions to the band:

The last one had an interactive header, with tiny sketches showing up on mouse over:

Unfortunately we didn’t found any of these themes matching the Cool Cavemen spirit (whatever that is). If these alternatives were publicly discussed, we decided that no one was going to replace our previous theme.

The final update we made was when our Raw EP was released. We basically applied filters on the header to match Raw’s cover. We also updated our logo to use the one designed for us by QPX:

This version add loads of polishing and is not far from being feature-complete. I think I’m reaching the end of the active development of this plugin. I don’t see the need to add new features.

I also feel this way because last week, I succeeded in moving to WordPress all news and pages from the old e107′s Cool Cavemen website. I now only need to import all forums to definitively get rid of e107 from my life. At that point, I will declare the plugin no longer active. This mean I will no longer update it, but will still integrate code other developers are willing to contribute.

Before that happen, I will of course release one or two revisions of this plugin in the next few months. But expect bug fixes and tiny enhancements, not big new features.

That being said, here is the changelog of the brand new e107 Importer 1.3:

• Upgrade embedded e107 code with latest 0.7.25.
• Redirect imported images to attachments.
• Purge invalid mapping entries on import.
• Replace old e107 URLs in content by new WordPress permalinks.
• Allow both imported and already-existing content to by updated with new permalinks.
• Let user specify the list of e107 forums to import.
• Phased imports should work without major problems.

As usual, my plugin is available on the official WordPress plugin directory.

Here is the detailed changelog:

• Upgrade e107 code to match latest 0.7.25-rc1.
• Fix variable bleeding when importing items in batches.
• Add a new way of handling e107 extended news using WordPress’ excerpts.
• Parse BBCode and replace e107 constants in news excerpt.
• Use internal WordPress library (kses) to parse HTML in the image upload step.
• Do not upload the same images more than once.
• Add a new enhanced BBCode parser on top of the one from e107. Make it the default parser.
• Each time we alter the original imported content, we create a post revision.

e107 migrated to full UTF-8 years ago, but I must have messed the upgrade process at the time. That was my conclusion when I took a close look to my tables: all of them seems to be set to Latin-1 but contain UTF-8 data. Here are screenshots from SQLBuddy (a great light-weight MySQL manager) showing just that:

To fix this, I first tried to use the following command I found on the web:

mysql --database=e107db -B -N -e "SHOW TABLES"  | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;"}' | mysql --database=e107db

But this doesn’t work, as it not only change the encoding of the table, but also transcode the data inside the table.

Let’s try something else. First, we’ll export the database to a dump file, of which the encoding is forced to Latin-1:

mysqldump -a -c -e --no-create-db --add-drop-table --default-character-set=latin1 --databases 'e107db' > ./e107-data.sql

Now the trick is to change the CHARSET parameter of all CREATE TABLE directives to UTF-8:

sed -i 's/CHARSET=latin1/CHARSET=utf8/g' ./e107-data.sql

We’ll also change the NAMES directive to force MySQL to handle imported data as UTF-8:

sed -i 's/SET NAMES latin1/SET NAMES utf8/g' ./e107-data.sql

Then we’re free to import the result in a new UTF-8 database:

sed -i 's/USE `e107db`;/#USE `e107db`;/g' ./e107-data.sql
mysql --execute="CREATE DATABASE e107db_new CHARACTER SET=utf8"
mysql --database=e107db_new < ./e107-data.sql

And now, accentuated characters appears as they should in our database, meaning we’ve fixed all the mess !

PS: I found another alternative method (look at the end of the linked page) which consists of temporarily handling TEXT fields as BLOB, to have MySQL treat them as binary content (thus skipping character transcoding). Haven’t tested this but sounds tricky.

The biggest new feature is support of e107 forum import to the bbPress WordPress plugin. This plugin is still in alpha and was not released yet. This mean you have to fetch it from its Subversion repository. And be careful to get the recommended version (SVN release 2942).

Because of the experimental status of forum import, the default option of e107 Importer is to not import forums. I decided to include this feature anyway to get feedbacks. So please don’t consider forum import as a highly reliable. It may work for you or may not. And please write detailed bug reports.

Here is a detailed changelog between 1.0 and 1.1:

• Add import of forums and threads to bbPress WordPress plugin.
• Parse BBCode and e107 constants in forums and thread.
• Add forums and threads redirections.
• Make e107 user import optional. This needs you to set a pre-existing WordPress user that will take ownership of all imported content.
• Parse BBCode in titles too.
• Import images embedded in comments and forum threads.
• Description update of existing users is no longer destructive.
• Add an entry in the FAQ regarding script ending prematurely.
• Disable all extra HTML rendering hooks like the one coming from e107 linkwords plugin.
• Allow news and pages import to be skipped.
• Add missing news category redirects.
• Minimal requirement set to WordPress 3.1.
• Some pages are not tied to a user. In this case, default to the current user.

This is the first time this plugin is available on the official WordPress plugin repository. This means easy upgrades for you !

I’ve heavily updated the plugin to use the latest WordPress import framework, so everything is now cleanly integrated.

This version was tested with e107 v0.7.24 and WordPress 3.1-RC3.

Here is the changelog:

• Upgrade e107 code from e107 v0.7.24.
• Minimal requirement set to WordPress 3.0.0.
• Use new WordPress importer framework.
• Add an e107 to WordPress 301 redirector plugin (support news, pages, users and feeds).
• Disable the URL rewriting feature introduced in v0.9.
• Make image import optional.
• Add an option to upload images from allowed domains only.
• Align naming conventions with other WordPress importer.
• Add a complete WordPress plugin hosting compatible readme file with full metadatas.
• Add screenshots.
• List all versions of e107 and WordPress I tested this plugin with.
• Add a PayPal donation link.
• Add a minimal FAQ.
• Add an overview of features in description.
• Update source code repository location.
• Remove patching of Kubrick theme to support comments on static pages.

A note for developpers: the reference code base is now located on GitHub. That’s were all new code must be commited. WordPress’ Subversion repository is only a mirror.

If any question, please read the FAQ first.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban…

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

The old gallery was based on autogallery, a e107 plugin. We assume here that both e107 and Zenphoto are well configured and installed at the root of you web hosting space (/www in this case).

The first step is to copy the autogallery album structure, with all its content, to Zenphoto:

cd /www
cp -ax ./e107_plugins/autogallery/Gallery/* ./zenphoto/albums/

Then we delete all previews, thumbnails and XML metadatas, to keep in Zenphoto original assets only:

find ./zenphoto/albums/ -iname "*.xml" | xargs rm -f
find ./zenphoto/albums/ -iname "pv_*" | xargs rm -f
find ./zenphoto/albums/ -iname "th_*" | xargs rm -f

By now, you should be able to play with your medias using Zenphoto’s admin interface.

But if you’re unlucky as I was, you will find a strange bug which break down drag’n'drop album sorting. The fix I found was to remove, in photo filenames, the numerical prefix (and the following dot) set by autogallery to define the sort order. This operation should be performed, before the copy from autogallery to Zenphoto (= the first command in this post). By the way, if you know a one-liner to do this, please, please… share !

To migrate comments, I have no automatic solution. I choose to do this manually, editing the database by hand. In my case it was the quickest way as I only had a dozen of comments to migrate.

And last but not least, if you care about measuring the popularity of your photos, you should consider migrating the view counter associated with each of your media. Don’t worry, this time I wrote a script to take care of it automagically. It will generate a bunch of SQL statements you’ll have to execute on your Zenphoto MySQL database. Here is my “e107 autogallery to Zenphoto hit counter migration script” (nice name isn’t it ? ) that do the job:

#!/usr/bin/python

##############################################################################
#
# Copyright (C) 2008 Kevin Deldycke <kevin@deldycke.com>
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
##############################################################################

"""
  Last update: 2008 aug 21
"""

########### User config ###########

AUTOGALLERY_ALBUM_PATH = "/www/e107_plugins/autogallery/Gallery"
ZENPHOTO_ALBUM_PATH    = "/www/zenphoto/albums"
ZENPHOTO_TABLE_PREFIX  = "zenphoto_"

######## End of user config #######

import os, hashlib
import xml.etree.ElementTree as ET

# Calculate hash of a given file
def getHash(path):
  # Calculate the hash from file raw data
  if not os.path.isfile(path):
    return None
  try:
    file_object = open(path, 'r')
    data = file_object.read()
  except:
    return None
  if not len(data):
    return None
  return hashlib.sha224(data).hexdigest()

# Associate each autogallery photo having a hitcounter greater than 0 with its MD5 hash
def populateHashTable(arg, dirname, names):
  global hash_table
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Get hit count for %s" % file_path
    # Check that the file as a positive hit counter associated with
    xml_file_path = "%s.xml" % file_path
    if not os.path.isfile(xml_file_path):
      continue
    try:
      tree = ET.parse(xml_file_path)
    except:
      continue
    node = tree.find("viewhits")
    if node is None:
      continue
    try:
      hits = int(node.text)
    except:
      continue
    if not hits > 0:
      continue
    # Update hash table with data we care about
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    hash_table[file_hash] = hits + hash_table.get(file_hash, 0)

# Generate hitcount SQL request for each matching file
def generateSQL(arg, dirname, names):
  global sql
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Search hitcounter matching file %s" % file_path
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    if file_hash in hash_table:
      sql += "UPDATE `%simages` SET `hitcounter`=`hitcounter`+%d WHERE `filename`=%r;\n" % (ZENPHOTO_TABLE_PREFIX, hash_table[file_hash], name)

# Core of the script
hash_table = {}
sql        = ""
# Normalize path
source_path = os.path.abspath(AUTOGALLERY_ALBUM_PATH)
dest_path   = os.path.abspath(ZENPHOTO_ALBUM_PATH)

os.path.walk(source_path, populateHashTable, None)
# print repr(hash_table)
os.path.walk(dest_path, generateSQL, None)
print sql

I think code and comments are self-explainatory. And do not forget to update constants at the top of the script to match your installation paths and database’s tables prefix.

And finally, for your information, I tested all of this on following versions:

• e107 0.7.11
• autogallery 2.61
• Zenphoto 1.2
• Python 2.5.2
• Linux server

9 months after the last one, here is the new version (v0.9) of my e107 to WordPress import plug-in !

Change log:

• “One-click migration” instead of multiple step process (more user-friendly),
• Better error management (a must-have for precise bug reports),
• Replace all links to old content with permalinks (increased SEO),
• Better database management,
• Work with latest WordPress v2.3.2 and e107 v0.7.11,
• Code cleaned up !