Installation Guide for a full-featured Debian server

Featured

Here is a collection of articles I wrote during the past year. Together they form a guide that will let you setup a full-featured Debian server. All of these tutorials are based on the recent work I did to setup my personal server on Debian Squeeze.

These articles are independent with each other, meaning you can pick the one your interested in to customize your server and ignore the others.

  1. Setup SMART monitoring tool for HDDs.
  2. Setup Nut to manage the UPS.
  3. Setup Duplicity and Amazon S3 for cloud-based backups.
  4. Setup Exim to relay mails via Gmail.
  5. Setup cron-apt to keep our distribution up to date.
  6. Add a fail2ban deamon.
  7. Setup Munin to monitor our machine.
  8. Basic setup of Nginx + PHP-FPM + MySQL web stack.
  9. Optimizing Nginx + PHP-FPM + MySQL for performances.
  10. Setup PHP APC op-code cache.
  11. Install haveged to get lots of entropy.
  12. Setup a WebDAVs server with Lighttpd.
  13. Setup Mailman + Nginx + Exim for mailing-lists.
  14. Mailman mailing-list migration and merging.

Configuring Fail2Ban on Debian Squeeze

This always start with a package installation:

$ aptitude install fail2ban

Then I simply create a local configuration file where I’ll put all my custom config:

$ touch /etc/fail2ban/jail.local

Here is the content of that file:

[DEFAULT]
# Do not filter connexion from my apartment and from the server itself
ignoreip  = 127.0.0.1 88.123.123.123 91.123.123.123
# Ban for a week
bantime   = 604800
maxretry  = 3
destemail = kevin@deldycke.com
banaction = iptables-allports
action    = %(action_mwl)s

[ssh]
enabled  = true
port     = 22
maxretry = 2

[ssh-ddos]
enabled = true
port     = 22

[apache]
# Apache basic auth
enabled   = true
maxretry  = 3
# Ban for 1 hour
bantime   = 3600

[apache-noscript]
enabled = true

[apache-overflows]
enabled = true

[apache-badbots]
enabled  = true
filter   = apache-badbots
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[apache-nohome]
enabled  = true
filter   = apache-nohome
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[exim]
enabled  = true
filter   = exim
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

[exim-relay]
enabled  = true
filter   = exim-relay
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

While adjusting Fail2Ban, I was surprised by how sensitive this software is. It can just refuse to start without any notice in the log or on the command line. Even if its log_level variable is set to 4 (= DEBUG) in /etc/fail2ban/fail2ban.conf.

In such a case, a sure way to find the culprit is to use a brute force debugging method: first set all the enabled variable of your jail.local‘s sections to false. Then activate one section after another until Fail2Ban refuse to restart.

For me, the problem was that I forgot to add my custom exim-relay filter to Fail2Ban. So I fixed my issue by creating an empty file at /etc/fail2ban/filter.d/exim-relay.conf in which I pasted the following content:

# Based on default exim.conf filter by Cyril Jaquier
# Real life exemaple:
# 2009-07-02 08:16:42 H=118-167-129-21.dynamic.hinet.net (91.121.198.84) [118.167.129.21] F=<titieueue@hotmail.com> rejected RCPT <s2288@mail2000.com.tw>: relay not permitted

[Definition]

# Option:  failregex
# Notes.:  regex to match use of my exim mail server as a relay it does not
#          allow.
# Values:  TEXT
#
failregex = \[<HOST>\] .*(?:relay not permitted)

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Speaking of custom filters, here is one to filter DFind scans (file located at /etc/fail2ban/filter.d/apache-w00tw00t.conf):

# Based on http://howflow.com/tricks/block_w00tw00t_scan_hosts_with_fail2ban
# Real life exemaple:
# [Sat Jun 27 16:43:08 2009] [error] [client 94.23.57.77] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Definition]

# Option:  failregex
# Notes.:  regex to match the w00tw00t scan messages in the logfile.
# Values:  TEXT
failregex = ^.*\[client <HOST>\].*w00tw00t\.at\.ISC\.SANS\.DFind.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

And here is the corresponding section from my jail.local file:

[apache-w00tw00t]
enabled  = true
filter   = apache-w00tw00t
action   = iptables-allports
logpath  = /var/log/apache*/*error.log
maxretry = 1

Using Munin to monitor a Debian Squeeze server

Again, here is a tutorial article exposing the recipe I use to cook a Munin on a Debian Squeeze.

As usual, let’s start by installing the main Munin package:

$ aptitude install munin

FYI, the version that aptitude choose to install was Munin 1.4.5. The default configuration coming along will make it produce graphs and HTML content to /var/cache/munin/www. Now we need to serve these pages via a web server.

As I wanted to play with nginx for a long time, I will use this opportunity to serve Munin’s content. The default version coming with Squeeze is quite old, so we’ll get the latest version from the Dotdeb repository:

$ echo "deb http://packages.dotdeb.org squeeze all" > /etc/apt/sources.list.d/squeeze-dotdeb.list
$ aptitude update
$ aptitude install nginx

And if you don’t want to get those error messages about untrusted packages, don’t forget to add Dotdeb’s keys to your keyring.

We can now test that nginx is working by starting it up then fetch the default served page:

$ /etc/init.d/nginx start
$ wget --output-document=- http://localhost | grep "Welcome to nginx"

Then we’ll disable the default nginx config and create a new one for Munin:

$ rm /etc/nginx/sites-enabled/default
$ touch /etc/nginx/sites-available/munin

In the latter, we put this minimal configuration:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  location / {
    index index.html;
    access_log off;
  }
}

Now we have to activate it before restarting nginx:

$ ln -s  /etc/nginx/sites-available/munin /etc/nginx/sites-enabled/munin
$ /etc/init.d/nginx restart

Now we are free to point our browser to the http://munin.example.com URL to get our graphs.

You’ll see that by default, Munin refer to your machine as localhost.localdomain. It’s time to tweak Munin a little to get nice reports:

$ sed -i 's/\[localhost\.localdomain\]/\[munin\.example\.com\]/g' /etc/munin/munin.conf

By default Munin activate a lot of great graphs. But I always find that some crucial monitoring are missing. Let’s add some more monitoring scripts:

$ aptitude install munin-plugins-extra

Here is a collection of general purpose graphs I automatically add to Munin:

$ ln -s /usr/share/munin/plugins/df_abs  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/netstat /etc/munin/plugins/
$ echo "[netstat]
user root
" > /etc/munin/plugin-conf.d/netstat

It’s also good to have a clue about your connectivity to the rest of the world:

$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_google.com
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_ovh.fr
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_example.com

I also like to have insight about my automated backups:

$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_duplicity
$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_sshd

Monitoring temperatures, voltages and other hardware metrics is a must, unless your machine is a virtual server :) :

$ ln -s /usr/share/munin/plugins/cpuspeed         /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/acpi             /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/hddtemp_smartctl /etc/munin/plugins/
$ aptitude install i2c-tools lm-sensors
$ sensors-detect
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_temp
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_volt

I sometimes have a Fail2Ban deamon running on a server, so that’s a good thing to monitor it:

$ ln -s /usr/share/munin/plugins/fail2ban /etc/munin/plugins/
$ echo "[fail2ban*]
user root
" > /etc/munin/plugin-conf.d/fail2ban

Having an UPS, it’s good to monitor it too. Here is for the UPS on the local system having the MGE-Ellipse750 ID (as defined in your /etc/nut/ups.conf file):

$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_voltages
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_charge
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_freq
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_current
$ ln -s /usr/share/munin/plugins/nut_misc  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nut_volts /etc/munin/plugins/
$ echo "[nut*]
user root

[nut_*]
env.upsname MGE-Ellipse750@localhost
" > /etc/munin/plugin-conf.d/nut

And if you have a MySQL server running on the machine, that’s a good idea to get stats:

$ ln -s /usr/share/munin/plugins/mysql_threads     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_slowqueries /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_queries     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_bytes       /etc/munin/plugins/

I also use some other Munin plugins coming from Munin exchange:

$ wget http://exchange.munin-monitoring.org/plugins/mysql_size_all/version/1/download --output-document=/usr/share/munin/plugins/mysql_size_all
$ ln -s /usr/share/munin/plugins/mysql_size_all /etc/munin/plugins/

An here is how I monitor my RAID array:

$ wget http://exchange.munin-monitoring.org/plugins/raid/version/3/download --output-document=/usr/share/munin/plugins/raid
$ ln -s /usr/share/munin/plugins/raid /etc/munin/plugins/
$ echo "[raid]
user root
" > /etc/munin/plugin-conf.d/raid

Finally, it’s time to monitor nginx itself:

$ ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
$ echo "[nginx_*]
env.url http://localhost/nginx_status
" > /etc/munin/plugin-conf.d/nginx

These two scripts above have some Perl module dependencies:

$ aptitude install libio-all-lwp-perl

If you don’t install the libraries above, you’ll get these kind of errors in /var/log/munin/munin-node.log:

2011/05/03-17:50:10 [2009] Error output from nginx_request:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_request line 106.
2011/05/03-17:50:10 [2009] Service 'nginx_request' exited with status 9/0.
2011/05/03-17:50:10 [2009] Error output from nginx_status:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_status line 109.
2011/05/03-17:50:10 [2009] Service 'nginx_status' exited with status 2/0.

But for this to work, we have to update the /etc/nginx/sites-enabled/munin file. Now it looks like this:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  # Restrict Munin access
  auth_basic "Restricted";
  auth_basic_user_file /etc/nginx/htpasswd;
  location / {
    index index.html;
    access_log off;
  }
}
server {
  allow 127.0.0.1;
  deny all;
  location /nginx_status {
    stub_status on;
    access_log off;
  }
}

Note that I’ve added a simple HTTP authentication to Munin webpages and restricted access to nginx statistics from the local machine only.

At last, before rebooting Munin and Nginx, make sure all downloaded plugins are executables. This is important and always forgotten:

$ chmod -R 755 /usr/share/munin/plugins/
$ /etc/init.d/nginx restart
$ /etc/init.d/munin-node restart

Blocking e107 dDOS attack with fail2ban

Last month, a new security vulnerability was discovered in e107. If a fix was released quickly, some instances on the web were left unpatched. These sites are easy target for hackers script-kiddies, and a generalized dDOS attack was carry out on every e107 websites out there.

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74