WebPing is a script I started to work on in 2009 while working at EDF. Back then, I needed a monitoring tool to keep an eye on the 80+ Plone instances that my team managed. For several corporate reasons, I wasn’t allowed to use a proper monitoring tool like Munin or Nagios. So I created a small script to fill this need. That’s how WebPing came to be.

WebPing is just a stupid Python script that is designed to be ticked regularly by a cron job. It try to fetch a list of URLs and store response times in an SQLite database. Then it create a static HTML report you’re free to serve with any HTTP server (an example Apache configuration is provided). The configuration of WebPing and the list of URLs it monitor is stored in a YAML file.

The produced HTML report use the Flot jQuery plugin to render graphs. Here is how the dashboard looks like:

Finally, WebPing is able to send reports and alerts by emails. Here is how a mail alert looks like:

Since I created WebPing, I found several other projects more or less developed around the same idea. See Kong, which is based on Django and Twill, a web-oriented DSL. Another project I spotted after the facts was multi-mechanize. Like Kong, it’s written in Python. But I never played with one or the other.

$ aptitude install fail2ban

Then I simply create a local configuration file where I’ll put all my custom config:

$ touch /etc/fail2ban/jail.local

Here is the content of that file:

[DEFAULT]
# Do not filter connexion from my apartment and from the server itself
ignoreip  = 127.0.0.1 88.123.123.123 91.123.123.123
# Ban for a week
bantime   = 604800
maxretry  = 3
destemail = kevin@deldycke.com
banaction = iptables-allports
action    = %(action_mwl)s

[ssh]
enabled  = true
port     = 22
maxretry = 2

[ssh-ddos]
enabled = true
port     = 22

[apache]
# Apache basic auth
enabled   = true
maxretry  = 3
# Ban for 1 hour
bantime   = 3600

[apache-noscript]
enabled = true

[apache-overflows]
enabled = true

[apache-badbots]
enabled  = true
filter   = apache-badbots
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[apache-nohome]
enabled  = true
filter   = apache-nohome
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[exim]
enabled  = true
filter   = exim
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

[exim-relay]
enabled  = true
filter   = exim-relay
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

While adjusting Fail2Ban, I was surprised by how sensitive this software is. It can just refuse to start without any notice in the log or on the command line. Even if its log_level variable is set to 4 (= DEBUG) in /etc/fail2ban/fail2ban.conf.

In such a case, a sure way to find the culprit is to use a brute force debugging method: first set all the enabled variable of your jail.local‘s sections to false. Then activate one section after another until Fail2Ban refuse to restart.

For me, the problem was that I forgot to add my custom exim-relay filter to Fail2Ban. So I fixed my issue by creating an empty file at /etc/fail2ban/filter.d/exim-relay.conf in which I pasted the following content:

# Based on default exim.conf filter by Cyril Jaquier
# Real life exemaple:
# 2009-07-02 08:16:42 H=118-167-129-21.dynamic.hinet.net (91.121.198.84) [118.167.129.21] F=<titieueue@hotmail.com> rejected RCPT <s2288@mail2000.com.tw>: relay not permitted

[Definition]

# Option:  failregex
# Notes.:  regex to match use of my exim mail server as a relay it does not
#          allow.
# Values:  TEXT
#
failregex = \[<HOST>\] .*(?:relay not permitted)

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Speaking of custom filters, here is one to filter DFind scans (file located at /etc/fail2ban/filter.d/apache-w00tw00t.conf):

# Based on http://howflow.com/tricks/block_w00tw00t_scan_hosts_with_fail2ban
# Real life exemaple:
# [Sat Jun 27 16:43:08 2009] [error] [client 94.23.57.77] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Definition]

# Option:  failregex
# Notes.:  regex to match the w00tw00t scan messages in the logfile.
# Values:  TEXT
failregex = ^.*\[client <HOST>\].*w00tw00t\.at\.ISC\.SANS\.DFind.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

And here is the corresponding section from my jail.local file:

[apache-w00tw00t]
enabled  = true
filter   = apache-w00tw00t
action   = iptables-allports
logpath  = /var/log/apache*/*error.log
maxretry = 1

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban…

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

But since this blog moved from one location to another in the last past years, I doubt everyone use the right URLs. This update is a good opportunity to check that you are using the “official” feed URLs:

• For blog posts: http://kevin.deldycke.com/feed
• For blog comments: http://kevin.deldycke.com/comments/feed

In the future, I plan to support these URLs only. So please update your feed aggregation settings !

If Lich’ti’s domain name is dead, QPX’s personal blog is not. His website is powered by WordPress and was available at http://qpx.lich-ti.fr. My job is now to move it to http://qpx.coolcavemen.com. In this post, I’ll tell you how I’ve done it.

Before going further, backup everything, and be ready to revert back to your original situation at any moment ! What works for me will not necessary works for you…

To play nice with your visitors, you can setup a temporary maintenance page while we’re performing the migration.

Let’s start the migration by replacing, in the files served by Apache, all occurrences of the old domain name by the new one:

find /var/www/qpx-blog -mount -print -type f -exec sed -i 's/qpx.lich-ti.fr/qpx.coolcavemen.com/g' "{}" \;

If you have doubts about the efficiency of the command above, you can check the presence of the string we’re looking to replace via this command:

grep -RIi "qpx.lich-ti.fr" ./*

Then, we dump the database containing all WordPress content and config to a local file (the command will prompt for password):

mysqldump -p --host=localhost --port=3306 --user=root --opt --databases "qpx_blog" > qpx_dump.sql

And we replace all strings of the old domain by the new one:

sed 's/qpx.lich-ti.fr/qpx.coolcavemen.com/g' qpx_dump.sql > new_qpx.sql

Finally, we re-inject the modified database content after clearing the original:

mysql -p --host=localhost --port=3306 --user=root --execute='DROP DATABASE `qpx_blog`;'
mysql -p --host=localhost --port=3306 --user=root < new_qpx.sql

Now you can disable the maintenance page and test the blog to check nothing’s broken.

Again, to play nice with your visitors (and search engines), you can redirect old URLs to the new domain, with apache directives similar to this one:

<VirtualHost *:80>
  ServerName qpx.lich-ti.fr
  RedirectMatch permanent (.*) http://qpx.coolcavemen.com$1
</VirtualHost>

As Wikipedia tells you, several tools of this kind exists. Why choosing eAccelerator in particular ? I really have no clue… I’ve never used any of these tools, so I had to start somewhere. That’s as simple as that !

Now, I have a Debian server as a target system. Unfortunately, eAccelerator is not bundled in Lenny. Browsing the web, I found some personal repositories of people kindly sharing their deb packages, like Andrew McMillan and schnuckelig.eu. The former provides a version of eAccelerator for the i386, the latter for the amd64 architecture. In this how-to, I’ve combined the 2 repositories to give both 32 bits and 64 bits users a chance to use eAccelerator on Lenny.

Let’s start the installation ! First, add the following lines to your /etc/apt/sources.list file:

deb http://debian.mcmillan.net.nz/debian lenny awm
deb-src http://debian.mcmillan.net.nz/debian lenny awm
deb http://debian.schnuckelig.eu/ lenny main contrib non-free

To kill annoying warning messages, register the cryptographic fingerprint of each repository:

$ gpg --keyserver keyring.debian.org --recv-keys 0x8f068012;
$ gpg --export --armor 0x8f068012 | apt-key add -
$ wget -O - http://debian.schnuckelig.eu/repository-key.gpg | apt-key add -

Then, update your package database:

$ aptitude update

And finally, you can install eAccelerator for PHP5 without any pain:

$ apt-get install php5-eaccelerator

Happy fine-tunning !

One of them, Maomium, was hacked last night. Someone created a user account on it then stole my admin identity to post content. As soon as I discovered the hack, I’ve put the blog down and changed all passwords which may have been exposed to the hacker (database, etc…).

Before the hack happened, my apache log show me that a person was looking for blogs powered by WordPress 2.2 and open to registration:

123.76-136-217.adsl-dyn.isp.belgacom.be www.maomium.com - [07/Jun/2007:00:51:55 +0200] "GET /category/wordpress/ HTTP/1.1" 200 2960 "http://www.google.be/search?hl=fr&q=%22powered+by+wordpress+2.2%22+Register&btnG=Rechercher&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4"

This person was my hacker. As you can see he’s a belgian guy and his broadband provider is Belgacom, to which I sent an abuse request. He register himself as Waryas with his myv4you@hotmail.com email. I know that, thanks to the email WordPress send me each time someone register. Then google told me that this hack was not his first.

If you want to disect his behaviour, you can download my apache log.

This event show us that the WordPress vulnerablility regarding guest account registration is still there. So the advice given by CountZero must be applied !

In the context of my plan to move an e107-based website to WordPress, I need to take care of my RSS subscribers. To let people (and search engines) get my content via old URLs, I will use Apache redirections to do this transparently and permanently. My final goal is to have a WordPress website with all RSS feeds (blog posts and comments) managed by FeedBurner, to gather statistics about my audience.

Actually there is plenty of feeds format available in e107 (RSS 1.0, RSS 2.0, Atom and RDF) and one feed can be accessed through multiple URLs. We will reduce this incredible mess by using RSS 2.0 feeds only and redirect all others to it.

First, check that the e107 RSS feed plugin is activated. Then create an account on FeedBurner and setup there two feeds, one for your website’s news and another one for comments. Based on default e107 parameters, your news feed URL is like http://www.my-domain.com/e107_plugins/rss_menu/rss.php?1.2 and comments feed like http://www.my-domain.com/e107_plugins/rss_menu/rss.php?5.2.

Then, create (or edit) the http://www.my-domain.com/.htaccess file, and add following code:

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !FeedBurner [NC]
RewriteCond %{QUERY_STRING} ^(5|Comments)
RewriteRule e107_plugins/rss_menu/rss\.php http://feeds.feedburner.com/myfeed-comments? [R=301,L]

RewriteCond %{HTTP_USER_AGENT} !FeedBurner [NC]
RewriteCond %{QUERY_STRING} ^(1|News|.*)
RewriteRule e107_plugins/rss_menu/rss\.php http://feeds.feedburner.com/myfeed? [R=301,L]

This code is inspired by the one written by Mike Atlas, who had a similar issue and wanted to outsource his e107 forum RSS feeds to FeedBurner.

The first rewrite rule will redirect all URLs that start with http://www.my-domain.com/e107_plugins/rss_menu/rss.php?5 or http://www.my-domain.com/e107_plugins/rss_menu/rss.php?Comments to http://feeds.feedburner.com/myfeed-comments.

The second rewrite rule will redirect all other URLs that start with http://www.my-domain.com/e107_plugins/rss_menu/rss.php (including http://www.my-domain.com/e107_plugins/rss_menu/rss.php?1 and http://www.my-domain.com/e107_plugins/rss_menu/rss.php?News) to http://feeds.feedburner.com/myfeed.

That’s all ! Thanks to this server-side redirection, nobody will notice that the feeds have moved and no subscriber will be bothered to update their aggregator.

In my case, the only remaining task to do is to move my e107 website to WordPress then install FeedSmith plugin. But that’s another story…

So I decided to move the http://kev.coolcavemen.com/static/repository folder to http://kevin.deldycke.free.fr/repository, which is a 10GiB (yes, ten gibibyte, this is not a typo) free web space offered by Free, my ISP.

To do this smoothly, I’ve just set up a generic 301 redirection thanks to Apache. This is the only line I added to my root .htaccess file to enable this:

Redirect permanent /static/repository http://kevin.deldycke.free.fr/repository

This move will normally be completely silent for you. So please, let me now if something bad happend while you play with my repository.