WebPing Open-sourced !

I’ve just released WebPing under a GPL license. It’s available right now on a GitHub repository.

WebPing is a script I started to work on in 2009 while working at EDF. Back then, I needed a monitoring tool to keep an eye on the 80+ Plone instances that my team managed. For several corporate reasons, I wasn’t allowed to use a proper monitoring tool like Munin or Nagios. So I created a small script to fill this need. That’s how WebPing came to be.

WebPing is just a stupid Python script that is designed to be ticked regularly by a cron job. It try to fetch a list of URLs and store response times in an SQLite database. Then it create a static HTML report you’re free to serve with any HTTP server (an example Apache configuration is provided). The configuration of WebPing and the list of URLs it monitor is stored in a YAML file.

The produced HTML report use the Flot jQuery plugin to render graphs. Here is how the dashboard looks like:

Finally, WebPing is able to send reports and alerts by emails. Here is how a mail alert looks like:

Since I created WebPing, I found several other projects more or less developed around the same idea. See Kong, which is based on Django and Twill, a web-oriented DSL. Another project I spotted after the facts was multi-mechanize. Like Kong, it’s written in Python. But I never played with one or the other.

Configuring Fail2Ban on Debian Squeeze

This always start with a package installation:

$ aptitude install fail2ban

Then I simply create a local configuration file where I’ll put all my custom config:

$ touch /etc/fail2ban/jail.local

Here is the content of that file:

[DEFAULT]
# Do not filter connexion from my apartment and from the server itself
ignoreip  = 127.0.0.1 88.123.123.123 91.123.123.123
# Ban for a week
bantime   = 604800
maxretry  = 3
destemail = kevin@deldycke.com
banaction = iptables-allports
action    = %(action_mwl)s

[ssh]
enabled  = true
port     = 22
maxretry = 2

[ssh-ddos]
enabled = true
port     = 22

[apache]
# Apache basic auth
enabled   = true
maxretry  = 3
# Ban for 1 hour
bantime   = 3600

[apache-noscript]
enabled = true

[apache-overflows]
enabled = true

[apache-badbots]
enabled  = true
filter   = apache-badbots
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[apache-nohome]
enabled  = true
filter   = apache-nohome
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[exim]
enabled  = true
filter   = exim
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

[exim-relay]
enabled  = true
filter   = exim-relay
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

While adjusting Fail2Ban, I was surprised by how sensitive this software is. It can just refuse to start without any notice in the log or on the command line. Even if its log_level variable is set to 4 (= DEBUG) in /etc/fail2ban/fail2ban.conf.

In such a case, a sure way to find the culprit is to use a brute force debugging method: first set all the enabled variable of your jail.local‘s sections to false. Then activate one section after another until Fail2Ban refuse to restart.

For me, the problem was that I forgot to add my custom exim-relay filter to Fail2Ban. So I fixed my issue by creating an empty file at /etc/fail2ban/filter.d/exim-relay.conf in which I pasted the following content:

# Based on default exim.conf filter by Cyril Jaquier
# Real life exemaple:
# 2009-07-02 08:16:42 H=118-167-129-21.dynamic.hinet.net (91.121.198.84) [118.167.129.21] F=<titieueue@hotmail.com> rejected RCPT <s2288@mail2000.com.tw>: relay not permitted

[Definition]

# Option:  failregex
# Notes.:  regex to match use of my exim mail server as a relay it does not
#          allow.
# Values:  TEXT
#
failregex = \[<HOST>\] .*(?:relay not permitted)

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Speaking of custom filters, here is one to filter DFind scans (file located at /etc/fail2ban/filter.d/apache-w00tw00t.conf):

# Based on http://howflow.com/tricks/block_w00tw00t_scan_hosts_with_fail2ban
# Real life exemaple:
# [Sat Jun 27 16:43:08 2009] [error] [client 94.23.57.77] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Definition]

# Option:  failregex
# Notes.:  regex to match the w00tw00t scan messages in the logfile.
# Values:  TEXT
failregex = ^.*\[client <HOST>\].*w00tw00t\.at\.ISC\.SANS\.DFind.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

And here is the corresponding section from my jail.local file:

[apache-w00tw00t]
enabled  = true
filter   = apache-w00tw00t
action   = iptables-allports
logpath  = /var/log/apache*/*error.log
maxretry = 1

Apache commands

  • Hide Subversion and Git directories content (source):
    RedirectMatch 404 /\.(svn|git)(/|$)
    
  • Disable rendering of PHP files coming from imported third party Javascript submodules (context):
    RedirectMatch 404 js-(.*)\.php$
    
  • Redirect any request to current year sub-directory (I used this for a yearly-updated static web page):
    RewriteEngine on
    RewriteRule !^/2010/ /2010/ [R=301,L]
    
  • Here is my template for domain-based virtual host routing:
    # Setup the main website access
    <VirtualHost *:80>
      ServerName example.com
      DocumentRoot /var/www/example
      # Add extra capabilities to let CMS like WordPress manage redirections
      <Directory /var/www/example>
        Options +FollowSymLinks +SymLinksIfOwnerMatch
      </Directory>
    </VirtualHost>
    # Redirect all other access to the website from different domains to the canonical URL
    <VirtualHost *:80>
      ServerName www.example.com
      ServerAlias *.example.com
      ServerAlias example.net *.example.net
      ServerAlias example.org *.example.org
      RedirectMatch permanent (.*) http://example.com$1
    </VirtualHost>
    
  • Insert dynamic headers in HTTP responses depending on the browser:
    BrowserMatchNoCase ".*MSIE\s[1-6].*" IS_DISGUSTING_BROWSER
    Header add X-advice-of-the-day "Save a kitten: use Firefox !" env=IS_DISGUSTING_BROWSER
    
  • Prevent WebDAV connexions (thanks Guillaume!):
    <Location />
      <Limit PROPFIND PROPPATCH MKCOL COPY MOVE LOCK UNLOCK PATCH>
        # Leaves GET (and HEAD), POST, PUT, DELETE, CONNECT, OPTIONS and TRACE alone
        Order allow,deny
        Deny from all
      </Limit>
    </Location>
    SetEnvIf Request_Method "OPTIONS" CLIENT_PROBE
    Header set Allow "GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE" env=CLIENT_PROBE
    
  • At work, we had to engineer a convoluted software architecture for our intranet to fit the network security policy of our customer. This had a bad side effect of letting the web statistic collector delete all cookies but its own, thus breaking intranet’s authentication. So we (thanks Matthieu!) came up with this unmaintainable hack on Apache side to hide our intranet’s cookies to NedStat’s Javascript embedded code:
    <LocationMatch "/(.*)">
      LoadModule headers_module modules/mod_headers.so
      RequestHeader edit Cookie "(app_cookie_001=[^;]*(; )*)" ""
      RequestHeader edit Cookie "(app_cookie_002=[^;]*(; )*)" ""
      RequestHeader edit Cookie "(app_cookie_003=[^;]*(; )*)" ""
    </LocationMatch>
    
  • Kill all apache processes and restart the service:
    /etc/init.d/apache2 stop ; pkill -9 -u www-data ; /etc/init.d/apache2 restart
    
  • Restart Apache service if no process found:
    [ `ps axu | grep -v "grep" | grep --count "www-data"` -le 0 ] && /etc/init.d/apache2 restart
    

Blocking e107 dDOS attack with fail2ban

Last month, a new security vulnerability was discovered in e107. If a fix was released quickly, some instances on the web were left unpatched. These sites are easy target for hackers script-kiddies, and a generalized dDOS attack was carry out on every e107 websites out there.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

Feeds updated !

Last night I’ve changed the way my feeds are handled on this blog. I’ve taken care of all redirections with a mix of WordPress plugins, Apache’s 301 redirects and Feedburner’s “My Brand” service. So everything should be transparent from your (and your feed reader) point of you.

But since this blog moved from one location to another in the last past years, I doubt everyone use the right URLs. This update is a good opportunity to check that you are using the “official” feed URLs:

In the future, I plan to support these URLs only. So please update your feed aggregation settings ! :)

Moving a WordPress blog to another domain

qpx-site-domain-migration I provide hosting for free to some of my friends. One of them, QPX, had a side project called Lich’ti. But the latter is no longer active, so he decided to not renew the lich-ti.fr domain.

If Lich’ti’s domain name is dead, QPX’s personal blog is not. His website is powered by WordPress and was available at http://qpx.lich-ti.fr. My job is now to move it to http://qpx.coolcavemen.com. In this post, I’ll tell you how I’ve done it.

Before going further, backup everything, and be ready to revert back to your original situation at any moment ! What works for me will not necessary works for you…

To play nice with your visitors, you can setup a temporary maintenance page while we’re performing the migration.

Let’s start the migration by replacing, in the files served by Apache, all occurrences of the old domain name by the new one:

find /var/www/qpx-blog -mount -print -type f -exec sed -i 's/qpx.lich-ti.fr/qpx.coolcavemen.com/g' "{}" \;

If you have doubts about the efficiency of the command above, you can check the presence of the string we’re looking to replace via this command:

grep -RIi "qpx.lich-ti.fr" ./*

Then, we dump the database containing all WordPress content and config to a local file (the command will prompt for password):

mysqldump -p --host=localhost --port=3306 --user=root --opt --databases "qpx_blog" > qpx_dump.sql

And we replace all strings of the old domain by the new one:

sed 's/qpx.lich-ti.fr/qpx.coolcavemen.com/g' qpx_dump.sql > new_qpx.sql

Finally, we re-inject the modified database content after clearing the original:

mysql -p --host=localhost --port=3306 --user=root --execute='DROP DATABASE `qpx_blog`;'
mysql -p --host=localhost --port=3306 --user=root < new_qpx.sql

Now you can disable the maintenance page and test the blog to check nothing’s broken.

Again, to play nice with your visitors (and search engines), you can redirect old URLs to the new domain, with apache directives similar to this one:

<VirtualHost *:80>
  ServerName qpx.lich-ti.fr
  RedirectMatch permanent (.*) http://qpx.coolcavemen.com$1
</VirtualHost>