Tag Archives: PHP

e107 Importer v1.3 released

Posted on by
6

A month after the last one, here is a brand new version of my e107 Importer, numbered 1.3.

This version add loads of polishing and is not far from being feature-complete. I think I’m reaching the end of the active development of this plugin. I don’t see the need to add new features.

I also feel this way because last week, I succeeded in moving to WordPress all news and pages from the old e107′s Cool Cavemen website. I now only need to import all forums to definitively get rid of e107 from my life. At that point, I will declare the plugin no longer active. This mean I will no longer update it, but will still integrate code other developers are willing to contribute.

Before that happen, I will of course release one or two revisions of this plugin in the next few months. But expect bug fixes and tiny enhancements, not big new features.

That being said, here is the changelog of the brand new e107 Importer 1.3:

  • Upgrade embedded e107 code with latest 0.7.25.
  • Redirect imported images to attachments.
  • Purge invalid mapping entries on import.
  • Replace old e107 URLs in content by new WordPress permalinks.
  • Allow both imported and already-existing content to by updated with new permalinks.
  • Let user specify the list of e107 forums to import.
  • Phased imports should work without major problems.

e107 Importer 1.2 is out, with an enhanced BBCode parser.

Posted on by
3

Here is a brand new version of my e107 Importer plugin. This release fix lots of nasty bugs. Better, I added an enhanced BBCode parser which try to clean-up what e107′s parser output. This new parser also try to align the final HTML with what WordPress produce by default.

As usual, my plugin is available on the official WordPress plugin directory.

Here is the detailed changelog:

  • Upgrade e107 code to match latest 0.7.25-rc1.
  • Fix variable bleeding when importing items in batches.
  • Add a new way of handling e107 extended news using WordPress’ excerpts.
  • Parse BBCode and replace e107 constants in news excerpt.
  • Use internal WordPress library (kses) to parse HTML in the image upload step.
  • Do not upload the same images more than once.
  • Add a new enhanced BBCode parser on top of the one from e107. Make it the default parser.
  • Each time we alter the original imported content, we create a post revision.

Fixing messed-up encoding in MySQL

Posted on by
Reply

Currently working on my e107 Importer plugin, I was confronted today with badly-encoded data coming from my databases.

e107 migrated to full UTF-8 years ago, but I must have messed the upgrade process at the time. That was my conclusion when I took a close look to my tables: all of them seems to be set to Latin-1 but contain UTF-8 data. Here are screenshots from SQLBuddy (a great light-weight MySQL manager) showing just that:

To fix this, I first tried to use the following command I found on the web:

mysql --database=e107db -B -N -e "SHOW TABLES"  | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;"}' | mysql --database=e107db

But this doesn’t work, as it not only change the encoding of the table, but also transcode the data inside the table.

Let’s try something else. First, we’ll export the database to a dump file, of which the encoding is forced to Latin-1:

mysqldump -a -c -e --no-create-db --add-drop-table --default-character-set=latin1 --databases 'e107db' > ./e107-data.sql

Now the trick is to change the CHARSET parameter of all CREATE TABLE directives to UTF-8:

sed -i 's/CHARSET=latin1/CHARSET=utf8/g' ./e107-data.sql

We’ll also change the NAMES directive to force MySQL to handle imported data as UTF-8:

sed -i 's/SET NAMES latin1/SET NAMES utf8/g' ./e107-data.sql

Then we’re free to import the result in a new UTF-8 database:

sed -i 's/USE `e107db`;/#USE `e107db`;/g' ./e107-data.sql
mysql --execute="CREATE DATABASE e107db_new CHARACTER SET=utf8"
mysql --database=e107db_new < ./e107-data.sql

And now, accentuated characters appears as they should in our database, meaning we’ve fixed all the mess ! :)


PS: I found another alternative method (look at the end of the linked page) which consists of temporarily handling TEXT fields as BLOB, to have MySQL treat them as binary content (thus skipping character transcoding). Haven’t tested this but sounds tricky.

e107 Importer 1.1 available !

Posted on by
2

A month after the 1.0 release, here is my e107 Importer v1.1 for WordPress !

The biggest new feature is support of e107 forum import to the bbPress WordPress plugin. This plugin is still in alpha and was not released yet. This mean you have to fetch it from its Subversion repository. And be careful to get the recommended version (SVN release 2942).

Because of the experimental status of forum import, the default option of e107 Importer is to not import forums. I decided to include this feature anyway to get feedbacks. So please don’t consider forum import as a highly reliable. It may work for you or may not. And please write detailed bug reports.

Here is a detailed changelog between 1.0 and 1.1:

  • Add import of forums and threads to bbPress WordPress plugin.
  • Parse BBCode and e107 constants in forums and thread.
  • Add forums and threads redirections.
  • Make e107 user import optional. This needs you to set a pre-existing WordPress user that will take ownership of all imported content.
  • Parse BBCode in titles too.
  • Import images embedded in comments and forum threads.
  • Description update of existing users is no longer destructive.
  • Add an entry in the FAQ regarding script ending prematurely.
  • Disable all extra HTML rendering hooks like the one coming from e107 linkwords plugin.
  • Allow news and pages import to be skipped.
  • Add missing news category redirects.
  • Minimal requirement set to WordPress 3.1.
  • Some pages are not tied to a user. In this case, default to the current user.

e107 Importer WordPress plugin v1.0 released !

Posted on by
19

After 3 years in limbo, here is a new stable version of my e107 Importer plugin for WordPress, proudly numbered 1.0 ! :)

This is the first time this plugin is available on the official WordPress plugin repository. This means easy upgrades for you ! :)

I’ve heavily updated the plugin to use the latest WordPress import framework, so everything is now cleanly integrated.

This version was tested with e107 v0.7.24 and WordPress 3.1-RC3.

Here is the changelog:

A note for developpers: the reference code base is now located on GitHub. That’s were all new code must be commited. WordPress’ Subversion repository is only a mirror.

If any question, please read the FAQ first.

Blocking e107 dDOS attack with fail2ban

Posted on by
3

Last month, a new security vulnerability was discovered in e107. If a fix was released quickly, some instances on the web were left unpatched. These sites are easy target for hackers script-kiddies, and a generalized dDOS attack was carry out on every e107 websites out there.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban…

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74