e107 Importer v1.3 released

A month after the last one, here is a brand new version of my e107 Importer, numbered 1.3.

This version add loads of polishing and is not far from being feature-complete. I think I’m reaching the end of the active development of this plugin. I don’t see the need to add new features.

I also feel this way because last week, I succeeded in moving to WordPress all news and pages from the old e107′s Cool Cavemen website. I now only need to import all forums to definitively get rid of e107 from my life. At that point, I will declare the plugin no longer active. This mean I will no longer update it, but will still integrate code other developers are willing to contribute.

Before that happen, I will of course release one or two revisions of this plugin in the next few months. But expect bug fixes and tiny enhancements, not big new features.

That being said, here is the changelog of the brand new e107 Importer 1.3:

  • Upgrade embedded e107 code with latest 0.7.25.
  • Redirect imported images to attachments.
  • Purge invalid mapping entries on import.
  • Replace old e107 URLs in content by new WordPress permalinks.
  • Allow both imported and already-existing content to by updated with new permalinks.
  • Let user specify the list of e107 forums to import.
  • Phased imports should work without major problems.

Simple video backdrop with Quartz Composer

While playing with Quartz Composer, the only composition I made with a clear intention of using it was this one:

As you can see, it’s just my band‘s logo, subtly glowing (or breathing, as you wish). It’s a simple but effective animated stage backdrop. I haven’t used it in any of our concert, but next time a video projector is present on stage, I will ! :)

As usual, here is the source composition, and a preview of the patch:

QLC effects generator for LED panels

Here is another script I wrote some months ago. It’s called qlc-effects-generator.py. It’s a quick and dirty hack that generate chasers, groups and scenes for QLC (a QT-based DMX lighting software). It just produce XML statements you copy’n'paste in your .qxw QLC workspace file.

I used it to create some effects for my 4 el-cheapo Mac-Mah LED wider panels. This script helped me generate column and row presets of my 4×8 pixels LED matrix for some basic colors:

Here are some photos of my preliminary tests at home:

And finally photos of the panels on stage (taken by Toma Heroow during Cool Cavemen’s concert in last november):

As usual, use and hack this script at you own risks, and feel free to send me bug reports and contributions ! :)

Blocking e107 dDOS attack with fail2ban

Last month, a new security vulnerability was discovered in e107. If a fix was released quickly, some instances on the web were left unpatched. These sites are easy target for hackers script-kiddies, and a generalized dDOS attack was carry out on every e107 websites out there.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

Cool Cavemen live at Gayant Expo, part II.

Here is “Funky Cops”, the second Cool Cavemen’s live song at Gayant Expo:

I released this video two weeks ago for Cool Cavemen. As I try to release one video every week, I give a high priority to the editing work. This leaves me with little time to write on this blog.

But starting from now, I plan to publish a blog post for each video. I’ll use these articles to write about one aspect of the work involved behind the scene.

In the first post of the series, I gave you the context in which the concert was performed. Today the post is dedicated to video formats. First, let’s talk about the video sources…

The concert was shot with 4 cameras. Among them, only two were of the same kinds. Those were part of the live broadcasting system of the event. This explain the “mise en abyme” effect in the background screen:

At the end of the gig, I wasn’t be able to retrieve two continuous feeds. Instead I got an already-edited video corresponding to what was projected live (*sigh*).

As a result, I ended with 3 video sources:

  1. A DVD-like video stream (576i) produced by my consumer-grade camera (now for sale at 0.01€ on ebay). It produces 720×576 pixels interlaced frames at 25 fps, with a pixel ratio of 16:15 (giving 768×576 pixels frames at 1:1) and a final display ratio of 4:3. All encoded as a 9 Mbps MPEG-2 stream in a MPEG-PS container.
  2. A 720p video stream: 1280×720 pixels progressive frames at 30 fps, with 1:1 pixel ratio and 16:9 display ratio, encoded as variable bitrate MJPEG stream in a QuickTime container.
  3. The already-edited video stream (Half-D1) from unidentified Sony cameras: 352×576 pixels interlaced frames at 25 fps, with a pixel ratio of 24:11 (giving 768×576 pixels frames at 1:1) and a final display ratio of 4:3. The file was a 6 Mbps MPEG-2 stream in a MPEG-PS container.

All those informations were extracted thanks to ffmeg, mplayer and tcprobe (see all the command lines involved).

As you can see, this is an absolute mess ! There is no consistency ! And now, before starting the video editing itself, I have this important decision to make: choose the final video format, in which my project will be rendered.

Let me explain how I did it. But before, I have to tell you something. To me, an interlaced video at 25 fps is just a 50 fps stream with half the vertical resolution. This is important for you to know if you want to understand how I perceive quality. I’ll probably explain it in details in a future article. But for now, this should give you enough insights on how I came up with my two strategies.

The first one is the “maximizing” strategy. It consists of keeping the best parts from all video sources. Based on formats described above, this means 1280×720 pixels progressive frames at 50 fps, with 1:1 pixel ratio and 16:9 display ratio. In this process we create non-existent informations by scaling and interpolating spatial and temporal data.

The second strategy is the “minimizing” strategy which, you can guess from its name, is the exact opposite of the first one. Here we discard spatial and temporal informations until we reach a sub-format shared by all sources. In our example, this gives 352×288 pixels frames at 30 fps, with a pixel ratio of 24:22 and a display ratio of 4:3. There, 288 is half 576, which is the result of using a deinterlacing “bob” filter on video streams #1 and #3 to get 50 fps. And for the pixel ratio, as we “bobbed” the interlaced videos, we keep the worst horizontal scaling and multiply the vertical scaling by two, which give us 24:22.

For this project, I finally went by the first stategy. I choosed to render the project to a 720p video at 25 fps, with a 1:1 pixel ratio and 16:9 display ratio. Also known as… HD-Ready !

Why this format ? It’s the most popular one that closely match the characteristics we established three paragraphs above. It’s also quite standard, and “gives a chance” to the second video source to display in full resolution. I also felt that it will cause less pain when confronted to the wide range of software video players out there.

Now that I have decided which format to use, I can create a project in my video editor with the right parameters and start the editing process. But this is another topic for another post !

Cool Cavemen live at Gayant Expo: first video released !

After several months of work, nit-picking and drama, I’ve finally released Cool Cavemen’s “Pump the Funk Up” video:

This video was taken during the biggest Cool Cavemen’s concert in 2009. We were playing at Gayant Expo (“largest french venue in the north of Paris” as they said in their commercial leaflets…). It was April 17th, during Cartel des Mines, a student festival organized by a group of engineering schools.

As the whole concert was filmed, I plan to release a new song every one or two weeks. I can’t promise a regular release cycle as I edit videos along the way. And of course, it also depends on my available free time…

By the way, this post also mark the opening of a section dedicated to my video projects. Currently it’s quite empty and brief, but I hope to populate it with more substantial stuff soon…