These articles are independent with each other, meaning you can pick the one your interested in to customize your server and ignore the others.

1. Setup SMART monitoring tool for HDDs.
2. Setup Nut to manage the UPS.
3. Setup Duplicity and Amazon S3 for cloud-based backups.
4. Setup Exim to relay mails via Gmail.
5. Setup cron-apt to keep our distribution up to date.
6. Add a fail2ban deamon.
7. Setup Munin to monitor our machine.
8. Basic setup of Nginx + PHP-FPM + MySQL web stack.
9. Optimizing Nginx + PHP-FPM + MySQL for performances.
10. Setup PHP APC op-code cache.
11. Install haveged to get lots of entropy.
12. Setup a WebDAVs server with Lighttpd.
13. Setup Mailman + Nginx + Exim for mailing-lists.
14. Mailman mailing-list migration and merging.

The setup I’ll detail below runs on an OVH VPS instance. This virtual server has 4 CPU cores at 1.5GHz, 1 Go RAM and 50 Gb HDD.

I’m mostly running WordPress instances on that server, so you’ll see some reference of it in this post.

MySQL

First, let’s tune MySQL. That’s the easiest part of that article, as you only need to create a .cnf file in /etc/mysql/conf.d/ and place there all your custom parameters. Here is the content of my /etc/mysql/conf.d/kev.cnf:

[mysqld]
interactive_timeout = 50
join_buffer = 1M
key_buffer = 250M
max_connections = 100
max_heap_table_size = 32M
myisam_sort_buffer_size = 96M
query_cache_limit = 4M
query_cache_size = 250M
query_prealloc_size = 65K
query_alloc_block_size = 128K
read_buffer_size = 1M
read_rnd_buffer_size = 768K
sort_buffer_size = 1M
table_cache = 4096
thread_cache_size = 1024
tmp_table_size = 32M
wait_timeout = 500
# Debug
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
# InnoDBinnodb_buffer_pool_size = 256Minnodb_additional_mem_pool_size = 10Minnodb_log_file_size = 32Minnodb_flush_method = O_DIRECTinnodb_file_per_table = 1innodb_flush_log_at_trx_commit = 0
[mysqld_safe]
nice = -5
open_files_limit = 8192

[isamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

Most of these parameters were set for my particular usage and with insights from the MySQL Tuning Primer Script.

PHP-FPM

Unlike MySQL, the structure of PHP configuration files on Debian Squeeze doesn’t let us easily add our customizations. We have to modify the default files provided at the package installation.

Here is my setup of the PHP processes pool:

--- /etc/php5/fpm/pool.d/www.conf.orig     2011-06-07 08:14:30.000000000 +0200
+++ /etc/php5/fpm/pool.d/www.conf  2011-08-15 17:34:09.000000000 +0200
@@ -237,3 +237,10 @@
 ;php_admin_value[error_log] = /var/log/fpm-php.www.log
 ;php_admin_flag[log_errors] = on
 ;php_admin_value[memory_limit] = 32M
+
+pm.max_children = 25
+pm.start_servers = 4
+pm.min_spare_servers = 2
+pm.max_spare_servers = 10
+pm.max_requests = 500
+request_terminate_timeout = 30

The second customization I made is not about performances but convenience. It just allow my WordPress’ users to upload larger files:

--- /etc/php5/fpm/php.ini.orig      2011-06-18 13:32:37.000000000 +0200
+++ /etc/php5/fpm/php.ini   2011-06-22 22:50:49.000000000 +0200
@@ -725,7 +725,7 @@

 ; Maximum size of POST data that PHP will accept.
 ; http://php.net/post-max-size
-post_max_size = 8M
+post_max_size = 15M

 ; Magic quotes are a preprocessing feature of PHP where PHP will attempt to
 ; escape any character sequences in GET, POST, COOKIE and ENV data which might
@@ -876,7 +876,7 @@

 ; Maximum allowed size for uploaded files.
 ; http://php.net/upload-max-filesize
-upload_max_filesize = 2M
+upload_max_filesize = 15M

 ; Maximum number of files that can be uploaded via a single request
 max_file_uploads = 20

Nginx

Let’s say my WordPress blog is installed in /var/www/my_wordpress. To let it be served by Nginx, we add a configuration file for this site in /etc/nginx/sites-available/my_wordpress:

server {
  server_name blog.example.com;
  root /var/www/my_wordpress/;
  include /etc/nginx/wordpress.conf;
  location /static {
    autoindex on;
  }
}

server {
  listen 80 default_server;
  server_name .example.com .example.org .example.net;
  rewrite ^ http://blog.example.com$request_uri? permanent;
}

In the configuration above, you can see that I want my blog to be served at http://blog.example.com. I also added some domain redirections in the form of a second server section, and a way to better display my static file repository by letting Nginx generate index pages.

Then don’t forget to activate this site:

$ ln -s /etc/nginx/sites-available/my_wordpress /etc/nginx/sites-enabled/

The file above refer to /etc/nginx/wordpress.conf which is where I place all the configuration directives common to all the WordPress blogs on my server. Here is the content of that file:

# This order might seem weird - this is attempted to match last if rules below fail.
# See: http://wiki.nginx.org/HttpCoreModule
location / {
  try_files $uri $uri/ /index.php?q=$uri&$args;
}

# Add trailing slash to */wp-admin requests.
rewrite /wp-admin$ $scheme://$host$uri/ permanent;

include global.conf;

include php.conf;

Again, this file make a reference to php.conf, which is the same as the one featured in my previous article. I only removed the index directive to place it elsewhere, and added a limit on the number of PHP requests a client can make:

location ~ \.php$ {
  # Throttle requests to prevent abuse
  limit_req zone=antidos burst=5;

  # Zero-day exploit defense.
  # http://forum.nginx.org/read.php?2,88845,page=3
  # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
  # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
  try_files $uri =404;

  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  include /etc/nginx/fastcgi_params;

  # As explained in http://kbeezie.com/view/php-self-path-nginx/ some fastcgi_param are missing from fastcgi_params.
  # Keep these parameters for compatibility with old PHP scripts using them.
  fastcgi_param PATH_INFO       $fastcgi_path_info;
  fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

  # Some default config
  fastcgi_connect_timeout        60;
  fastcgi_send_timeout          180;
  fastcgi_read_timeout          180;
  fastcgi_buffer_size          128k;
  fastcgi_buffers            4 256k;
  fastcgi_busy_buffers_size    256k;
  fastcgi_temp_file_write_size 256k;

  fastcgi_intercept_errors    on;
  fastcgi_ignore_client_abort off;

  fastcgi_pass 127.0.0.1:9000;
}

Here is where the index directive moved: /etc/nginx/conf.d/kev.conf. I also added there some tweaks and the global request throttling configuration:

# Hide Nginx version
server_tokens off;

# Set default index file names
index index.php index.html index.htm;

# Allow uploads up to 15 Mo
client_max_body_size 15m;

# Create a global request accounting pool to prevent DOS
limit_req_zone $binary_remote_addr zone=antidos:10m rate=3r/s;

The global.conf file we saw in /etc/nginx/wordpress.conf refer to /etc/nginx/global.conf, which contain additional measures to remove cruft from log files and enhance security:

# Do not log excessive request on common web content like favicon and robots.txt
location = /favicon.ico {
  log_not_found off;
  access_log off;
}
location = /robots.txt {
  allow all;
  log_not_found off;
  access_log off;
}

# Deny all attempts to access any dotfile (=hidden files) such as .htaccess, .htpasswd, .DS_Store, .directory, .svn, .git, ...
location ~ /\. {
  deny all;
  access_log off;
  log_not_found off;
}

All of default Nginx configuration can’t be overridden by additional files. We have to change /etc/nginx/nginx.conf itself:

--- /etc/nginx/nginx.conf.orig   2011-06-06 00:46:56.000000000 +0200
+++ /etc/nginx/nginx.conf        2011-08-15 17:44:58.000000000 +0200
@@ -3,8 +3,9 @@
 pid /var/run/nginx.pid;

 events {
-       worker_connections 768;
-       # multi_accept on;
+       use epoll;
+       worker_connections 1024;
+       multi_accept on;
 }

 http {
@@ -16,7 +17,7 @@
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
-       keepalive_timeout 65;
+       keepalive_timeout 3;
        types_hash_max_size 2048;
        # server_tokens off;

That’s all for our customizations. We can now restart all our servers:

$ /etc/init.d/mysql restart
$ /etc/init.d/php5-fpm restart
$ /etc/init.d/nginx restart

Conclusion

I’m running my websties under this configuration for about 3 months and I’m really happy with the results. I’m sure I can push optimizations further, but it may require lots of time and effort compared to the marginal gain I’ll get. My websites are responsive enough to me. And if they collapse in the future under the load of the Reddit crowd, I’ll still have the option to move to a bigger virtual server (vertical scaling FTW!).

WebPing is a script I started to work on in 2009 while working at EDF. Back then, I needed a monitoring tool to keep an eye on the 80+ Plone instances that my team managed. For several corporate reasons, I wasn’t allowed to use a proper monitoring tool like Munin or Nagios. So I created a small script to fill this need. That’s how WebPing came to be.

WebPing is just a stupid Python script that is designed to be ticked regularly by a cron job. It try to fetch a list of URLs and store response times in an SQLite database. Then it create a static HTML report you’re free to serve with any HTTP server (an example Apache configuration is provided). The configuration of WebPing and the list of URLs it monitor is stored in a YAML file.

The produced HTML report use the Flot jQuery plugin to render graphs. Here is how the dashboard looks like:

Finally, WebPing is able to send reports and alerts by emails. Here is how a mail alert looks like:

Since I created WebPing, I found several other projects more or less developed around the same idea. See Kong, which is based on Django and Twill, a web-oriented DSL. Another project I spotted after the facts was multi-mechanize. Like Kong, it’s written in Python. But I never played with one or the other.

First, install the required packages:

$ aptitude install lighttpd-mod-webdav

As we want to provide a secure WebDAV access, we need to install OpenSSL:

$ aptitude install openssl

Then we create the file /etc/lighttpd/clear-creds.lst, that will contain credentials required for authentication, under the following form:

user1:password1
user2:password2
user3:password3

Logins and passwords are stored here in clear. This is stupid, but for this project I was looking to setup a quick and dirty server. For temporary tests this setup is OK, but I encourage you to switch to a better credential storage system.

Now I want to serve WebDAV content within a secure channel. A self-signed SSL certificate will be enough. Let’s generate one:

$ cd /etc/lighttpd/
$ openssl req -x509 -nodes -subj '/' -days 3650 -newkey rsa:2048 -keyout server.pem -out server.pem

We’ll configure Lighttpd by loading the default parameters of modules we use:

$ cd /etc/lighttpd/conf-enabled/
$ ln -s ../conf-available/05-auth.log
$ ln -s ../conf-available/10-ssl.conf
$ ln -s ../conf-available/10-webdav.conf

Now I create a custom configuration file:

$ touch /etc/lighttpd/conf-available/99-custom.conf
$ cd /etc/lighttpd/conf-enabled/
$ ln -s ../conf-available/99-custom.conf

Here is the content of that 99-custom.conf configuration file:

# Hide server version
server.tag = "lighttpd"

# Force all request to be in HTTPs
# This also redirects all WebDAV requests to WebDAVs
$HTTP["scheme"] == "http" {
  $HTTP["host"] =~ "(.*)" {
    url.redirect = ( "^/(.*)" => "https://%1/$1" )
  }
}

# Valid credentials are required for any request
auth.backend = "plain"
auth.backend.plain.userfile = "/etc/lighttpd/clear-creds.lst"
auth.require = (
  "/" => (
    "method" => "digest",
    "realm" => "My WebDAV server",
    "require" => "valid-user"
  )
)

# Enable WebDAV in read and write mode
webdav.activate = "enable"
webdav.is-readonly = "disable"

# Customize directory listings a bit
dir-listing.set-footer = "<a href='http://example.com'>Company</a>'s document repository."

And do not forget to restart the server:

$ /etc/init.d/lighttpd restart

As you can see in the screenshot above, you can now:

• Browse the file system in read/write mode with a WebDAV client via a webdavs://12.34.56.78/ URL;
• Access content in read-only mode with a browser by a classic https://12.34.56.78/ URL.

$ aptitude install php5-apc

In my case this package come from the PHP bundle distributed by the Dotdeb repository.

If installing APC is easy, monitoring it with Munin requires some extra manipulations. There is currently no good APC plugin available on Munin Exhange. So we’ll use the external munin-php-apc project instead.

The latter can’t get APC statistics by itself: it need an extra PHP file to be served locally. As you can read in my previous article, my Munin is powered by Nginx. So now we’ll setup Nginx to serve this extra PHP file:

$ mkdir -p /var/www/apc
$ cd /var/www/apc
$ wget http://munin-php-apc.googlecode.com/svn/trunk/php_apc/apc_info.php
$ chown -R www-data:www-data /var/www/apc

Then I need to update my /etc/nginx/sites-available/munin file (see details about this file on my previous article) to have the second server section look like this:

server {
  server_name localhost;
  include /etc/nginx/php.conf;
  root /var/www/apc;
  allow 127.0.0.1;
  deny all;
  location / {
    access_log off;
  }
  location /nginx_status {
    stub_status on;
    access_log off;
  }
}

Here the included /etc/nginx/php.conf file is the one in which I’ve concentrate all the Nginx directives required to activate PHP file parsing. The content and the mechanism behind this file is describe in my article on setting up Nginx with PHP-FPM.

Let’s get back to our Munin monitoring setup. I can restart now Nginx and check that I can access locally to my raw statistics:

$ /etc/init.d/nginx reload
$ wget http://localhost/apc_info.php
$ wget http://localhost/nginx_status

The last step is to install and configure the Munin plugin:

$ aptitude install libwww-perl
$ wget http://munin-php-apc.googlecode.com/svn/trunk/php_apc/php_apc_ --output-document=/usr/share/munin/plugins/php_apc_
$ chmod -R 755 /usr/share/munin/plugins/
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_usage
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_hit_miss
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_purge
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_fragmentation
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_files
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_rates
$ echo "[php_apc_*]
user root
env.url http://localhost/apc_info.php?auto
" > /etc/munin/plugin-conf.d/php_apc
$ /etc/init.d/munin-node restart

And finally, after a while, you’ll get those beautiful graphs:

First, we’ll get all our packages from an up-to-date DotDeb repository. If this is not already done, add those repositories to aptitude:

$ echo "deb http://packages.dotdeb.org squeeze all" > /etc/apt/sources.list.d/squeeze-dotdeb.list
$ gpg --keyserver keys.gnupg.net --recv-key 89DF5277
$ gpg -a --export 89DF5277 | apt-key add -
$ aptitude update

Now we can install the whole stack:

$ aptitude install nginx
$ aptitude install php5-fpm php5-mysql php5-gd php5-curl
$ aptitude install mysql-server

FYI, here is the list of versions I installed:

• Nginx 1.0.2
• PHP 5.3.6
• MySQL 5.1.57

As a way to test that our setup is working, we’ll serve a simple PHP file:

$ mkdir -p /var/www/example.com/
$ cd /var/www/example.com/
$ echo "
<?php phpinfo(); ?>
" > ./index.php
$ chown -R www-data:www-data /var/www

Now let’s create a minimal Nginx configuration file for this site:

$ touch /etc/nginx/sites-available/example.com

In this brand new file, put the following directives:

server {
  server_name example.com;
  include /etc/nginx/php.conf;
  location / {
    root /var/www/example.com/;
    access_log on;
  }
}

This will only work if you’ve updated your DNS with an A record having example.com redirecting to the IP address of your Nginx server.

Now it’s time to create the /etc/nginx/php.conf file referenced in the Nginx configuration above. This file is where I put the generic setup making the bridge between Nginx and PHP-FPM. Here is what it should contain:

index index.php index.html index.htm;

location ~ \.php$ {
  # Zero-day exploit defense.
  # http://forum.nginx.org/read.php?2,88845,page=3
  # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
  # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
  try_files $uri =404;

  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  include /etc/nginx/fastcgi_params;

  # As explained in http://kbeezie.com/view/php-self-path-nginx/ some fastcgi_param are missing from fastcgi_params.
  # Keep these parameters for compatibility with old PHP scripts using them.
  fastcgi_param PATH_INFO       $fastcgi_path_info;
  fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

  # Some default config
  fastcgi_connect_timeout        60;
  fastcgi_send_timeout          180;
  fastcgi_read_timeout          180;
  fastcgi_buffer_size          128k;
  fastcgi_buffers            4 256k;
  fastcgi_busy_buffers_size    256k;
  fastcgi_temp_file_write_size 256k;

  fastcgi_intercept_errors    on;
  fastcgi_ignore_client_abort off;

  fastcgi_pass 127.0.0.1:9000;
}

Finally you can activate the site configuration and restart the whole stack:

$ ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/
$ /etc/init.d/mysql restart
$ /etc/init.d/php5-fpm restart
$ /etc/init.d/nginx restart

If everything’s OK on your DNS, pointing your browser to http://example.com will show you the famous page produced by phpinfo():

Note that MySQL doesn’t need any special attention to make it work out of the box. But again, if you plan to use it in production, its configuration needs special care, as for Nginx and PHP.

$ aptitude install fail2ban

Then I simply create a local configuration file where I’ll put all my custom config:

$ touch /etc/fail2ban/jail.local

Here is the content of that file:

[DEFAULT]
# Do not filter connexion from my apartment and from the server itself
ignoreip  = 127.0.0.1 88.123.123.123 91.123.123.123
# Ban for a week
bantime   = 604800
maxretry  = 3
destemail = kevin@deldycke.com
banaction = iptables-allports
action    = %(action_mwl)s

[ssh]
enabled  = true
port     = 22
maxretry = 2

[ssh-ddos]
enabled = true
port     = 22

[apache]
# Apache basic auth
enabled   = true
maxretry  = 3
# Ban for 1 hour
bantime   = 3600

[apache-noscript]
enabled = true

[apache-overflows]
enabled = true

[apache-badbots]
enabled  = true
filter   = apache-badbots
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[apache-nohome]
enabled  = true
filter   = apache-nohome
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[exim]
enabled  = true
filter   = exim
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

[exim-relay]
enabled  = true
filter   = exim-relay
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

While adjusting Fail2Ban, I was surprised by how sensitive this software is. It can just refuse to start without any notice in the log or on the command line. Even if its log_level variable is set to 4 (= DEBUG) in /etc/fail2ban/fail2ban.conf.

In such a case, a sure way to find the culprit is to use a brute force debugging method: first set all the enabled variable of your jail.local‘s sections to false. Then activate one section after another until Fail2Ban refuse to restart.

For me, the problem was that I forgot to add my custom exim-relay filter to Fail2Ban. So I fixed my issue by creating an empty file at /etc/fail2ban/filter.d/exim-relay.conf in which I pasted the following content:

# Based on default exim.conf filter by Cyril Jaquier
# Real life exemaple:
# 2009-07-02 08:16:42 H=118-167-129-21.dynamic.hinet.net (91.121.198.84) [118.167.129.21] F=<titieueue@hotmail.com> rejected RCPT <s2288@mail2000.com.tw>: relay not permitted

[Definition]

# Option:  failregex
# Notes.:  regex to match use of my exim mail server as a relay it does not
#          allow.
# Values:  TEXT
#
failregex = \[<HOST>\] .*(?:relay not permitted)

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Speaking of custom filters, here is one to filter DFind scans (file located at /etc/fail2ban/filter.d/apache-w00tw00t.conf):

# Based on http://howflow.com/tricks/block_w00tw00t_scan_hosts_with_fail2ban
# Real life exemaple:
# [Sat Jun 27 16:43:08 2009] [error] [client 94.23.57.77] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Definition]

# Option:  failregex
# Notes.:  regex to match the w00tw00t scan messages in the logfile.
# Values:  TEXT
failregex = ^.*\[client <HOST>\].*w00tw00t\.at\.ISC\.SANS\.DFind.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

And here is the corresponding section from my jail.local file:

[apache-w00tw00t]
enabled  = true
filter   = apache-w00tw00t
action   = iptables-allports
logpath  = /var/log/apache*/*error.log
maxretry = 1

As usual, let’s start by installing the main Munin package:

$ aptitude install munin

FYI, the version that aptitude choose to install was Munin 1.4.5. The default configuration coming along will make it produce graphs and HTML content to /var/cache/munin/www. Now we need to serve these pages via a web server.

As I wanted to play with nginx for a long time, I will use this opportunity to serve Munin’s content. The default version coming with Squeeze is quite old, so we’ll get the latest version from the Dotdeb repository:

$ echo "deb http://packages.dotdeb.org squeeze all" > /etc/apt/sources.list.d/squeeze-dotdeb.list
$ aptitude update
$ aptitude install nginx

And if you don’t want to get those error messages about untrusted packages, don’t forget to add Dotdeb’s keys to your keyring.

We can now test that nginx is working by starting it up then fetch the default served page:

$ /etc/init.d/nginx start
$ wget --output-document=- http://localhost | grep "Welcome to nginx"

Then we’ll disable the default nginx config and create a new one for Munin:

$ rm /etc/nginx/sites-enabled/default
$ touch /etc/nginx/sites-available/munin

In the latter, we put this minimal configuration:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  location / {
    index index.html;
    access_log off;
  }
}

Now we have to activate it before restarting nginx:

$ ln -s  /etc/nginx/sites-available/munin /etc/nginx/sites-enabled/munin
$ /etc/init.d/nginx restart

Now we are free to point our browser to the http://munin.example.com URL to get our graphs.

You’ll see that by default, Munin refer to your machine as localhost.localdomain. It’s time to tweak Munin a little to get nice reports:

$ sed -i 's/\[localhost\.localdomain\]/\[munin\.example\.com\]/g' /etc/munin/munin.conf

By default Munin activate a lot of great graphs. But I always find that some crucial monitoring are missing. Let’s add some more monitoring scripts:

$ aptitude install munin-plugins-extra

Here is a collection of general purpose graphs I automatically add to Munin:

$ ln -s /usr/share/munin/plugins/df_abs  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/netstat /etc/munin/plugins/
$ echo "[netstat]
user root
" > /etc/munin/plugin-conf.d/netstat

It’s also good to have a clue about your connectivity to the rest of the world:

$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_google.com
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_ovh.fr
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_example.com

I also like to have insight about my automated backups:

$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_duplicity
$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_sshd

Monitoring temperatures, voltages and other hardware metrics is a must, unless your machine is a virtual server :

$ ln -s /usr/share/munin/plugins/cpuspeed         /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/acpi             /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/hddtemp_smartctl /etc/munin/plugins/
$ aptitude install i2c-tools lm-sensors
$ sensors-detect
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_temp
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_volt

I sometimes have a Fail2Ban deamon running on a server, so that’s a good thing to monitor it:

$ ln -s /usr/share/munin/plugins/fail2ban /etc/munin/plugins/
$ echo "[fail2ban*]
user root
" > /etc/munin/plugin-conf.d/fail2ban

Having an UPS, it’s good to monitor it too. Here is for the UPS on the local system having the MGE-Ellipse750 ID (as defined in your /etc/nut/ups.conf file):

$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_voltages
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_charge
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_freq
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_current
$ ln -s /usr/share/munin/plugins/nut_misc  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nut_volts /etc/munin/plugins/
$ echo "[nut*]
user root

[nut_*]
env.upsname MGE-Ellipse750@localhost
" > /etc/munin/plugin-conf.d/nut

And if you have a MySQL server running on the machine, that’s a good idea to get stats:

$ ln -s /usr/share/munin/plugins/mysql_threads     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_slowqueries /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_queries     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_bytes       /etc/munin/plugins/

I also use some other Munin plugins coming from Munin exchange:

$ wget http://exchange.munin-monitoring.org/plugins/mysql_size_all/version/1/download --output-document=/usr/share/munin/plugins/mysql_size_all
$ ln -s /usr/share/munin/plugins/mysql_size_all /etc/munin/plugins/

An here is how I monitor my RAID array:

$ wget http://exchange.munin-monitoring.org/plugins/raid/version/3/download --output-document=/usr/share/munin/plugins/raid
$ ln -s /usr/share/munin/plugins/raid /etc/munin/plugins/
$ echo "[raid]
user root
" > /etc/munin/plugin-conf.d/raid

Finally, it’s time to monitor nginx itself:

$ ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
$ echo "[nginx_*]
env.url http://localhost/nginx_status
" > /etc/munin/plugin-conf.d/nginx

These two scripts above have some Perl module dependencies:

$ aptitude install libio-all-lwp-perl

If you don’t install the libraries above, you’ll get these kind of errors in /var/log/munin/munin-node.log:

2011/05/03-17:50:10 [2009] Error output from nginx_request:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_request line 106.
2011/05/03-17:50:10 [2009] Service 'nginx_request' exited with status 9/0.
2011/05/03-17:50:10 [2009] Error output from nginx_status:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_status line 109.
2011/05/03-17:50:10 [2009] Service 'nginx_status' exited with status 2/0.

But for this to work, we have to update the /etc/nginx/sites-enabled/munin file. Now it looks like this:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  # Restrict Munin access
  auth_basic "Restricted";
  auth_basic_user_file /etc/nginx/htpasswd;
  location / {
    index index.html;
    access_log off;
  }
}
server {
  allow 127.0.0.1;
  deny all;
  location /nginx_status {
    stub_status on;
    access_log off;
  }
}

Note that I’ve added a simple HTTP authentication to Munin webpages and restricted access to nginx statistics from the local machine only.

At last, before rebooting Munin and Nginx, make sure all downloaded plugins are executables. This is important and always forgotten:

$ chmod -R 755 /usr/share/munin/plugins/
$ /etc/init.d/nginx restart
$ /etc/init.d/munin-node restart

I also read back then rumors implying that Nut’s maintainer was employed by MGE. Having a hardware manufacturer employing a fellow open-source hacker has certainly influenced my purchase decision.

MGE is no more and has been merged with EATON. But my UPS is still supported, and the release of Debian Squeeze is a good opportunity to consolidate my knowledge in the form of this tutorial.

So here is how I setup Nut on Debian Squeeze to monitor my UPS.

First things first, we have to install the main package and its USB driver:

$ aptitude install nut nut-usb

Now let’s configure Nut and run it:

$ sed -i 's/MODE=none/MODE=standalone/g' /etc/nut/nut.conf
$ echo '
[MGE-Ellipse750]
driver = usbhid-ups
port = auto
desc = "MGE UPS Systems"
' >> /etc/nut/ups.conf
$ sed -i 's/# LISTEN 127\.0\.0\.1 3493/LISTEN 127\.0\.0\.1/g' /etc/nut/upsd.conf
$ echo '
[kevin]
password = badpassword
upsmon master
' >> /etc/nut/upsd.users
$ sed -i 's/# NOTIFYCMD \/usr\/local\/ups\/bin\/notifyme/NOTIFYCMD \/sbin\/upssched/g' /etc/nut/upsmon.conf
$ echo '
MONITOR MGE-Ellipse750@localhost 1 kevin badpassword master
NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
' >> /etc/nut/upsmon.conf
$ sed -i 's/CMDSCRIPT \/upssched-cmd/CMDSCRIPT \/etc\/nut\/upssched-cmd/g' /etc/nut/upssched.conf
$ sed -i 's/# PIPEFN \/var\/run\/nut\/upssched\/upssched.pipe/PIPEFN \/var\/run\/nut\/upssched.pipe/g' /etc/nut/upssched.conf
$ sed -i 's/# LOCKFN \/var\/run\/nut\/upssched\/upssched.lock/LOCKFN \/var\/run\/nut\/upssched.lock/g' /etc/nut/upssched.conf
$ echo '
AT ONBATT * START-TIMER onbatt 30
AT ONLINE * CANCEL-TIMER onbatt
' >> /etc/nut/upssched.conf
$ echo '
#!/bin/sh
exit 0
' > /etc/nut/upssched-cmd
$ /etc/init.d/nut restart

As you can see you have lots of stuff to configure before Nut can do what it was designed for. But after all of these commands, you should have a working UPS.

You can now test that your system works by using the command below, which list statistics of a given UPS:

$ upsc MGE-Ellipse750@localhost

But in some rare cases, your UPS will not be recognized and you’ll have like me the following messages in your /var/log/syslog:

May  5 16:12:36 paris-server upsmon[10773]: Poll UPS [MGE-Ellipse750@127.0.0.1] failed - Driver not connected

In this case, you should run Nut’s driver in debug mode:

$ /lib/nut/usbhid-ups -DDD -a MGE-Ellipse750
Network UPS Tools - Generic HID driver 0.34 (2.4.3)
USB communication driver 0.31
   0.000000     debug level is '3'
   0.013911     upsdrv_initups...
   0.189541     Checking device (0463/FFFF) (005/003)
   0.189705     - VendorID: 0463
   0.189741     - ProductID: ffff
   0.189767     - Manufacturer: unknown
   0.189794     - Product: unknown
   0.189819     - Serial Number: unknown
   0.189842     - Bus: 005
   0.189862     Trying to match device
   0.189906     Device matches
   0.189954     failed to claim USB device: could not claim interface 0: Operation not permitted
   0.189995     failed to detach kernel driver from USB device: could not detach kernel driver from interface 0: Operation not permitted
   0.190033     failed to claim USB device: could not claim interface 0: Operation not permitted
   0.190070     failed to detach kernel driver from USB device: could not detach kernel driver from interface 0: Operation not permitted
   0.190108     failed to claim USB device: could not claim interface 0: Operation not permitted
   0.190145     failed to detach kernel driver from USB device: could not detach kernel driver from interface 0: Operation not permitted
   0.190181     failed to claim USB device: could not claim interface 0: Operation not permitted
   0.190217     failed to detach kernel driver from USB device: could not detach kernel driver from interface 0: Operation not permitted
   0.190252     Can't claim USB device [0463:ffff]: could not detach kernel driver from interface 0: Operation not permitted

As you can see in messages above, Nut can’t see my UPS. By chance, forcing nut to use the root user let it see my UPS:

$ /lib/nut/usbhid-ups -DDD -u root -a MGE-Ellipse750
Network UPS Tools - Generic HID driver 0.34 (2.4.3)
USB communication driver 0.31
   0.000000     debug level is '3'
   0.001678     upsdrv_initups...
   0.172877     Checking device (0463/FFFF) (005/003)
   1.112408     - VendorID: 0463
   1.112464     - ProductID: ffff
   1.112489     - Manufacturer: MGE OPS SYSTEMS
   1.112516     - Product: ELLIPSE
   1.112542     - Serial Number: BDCJ3800Q
   1.112569     - Bus: 005
   1.112595     Trying to match device
   1.112647     Device matches
   1.112726     failed to claim USB device: could not claim interface 0: Device or resource busy
   1.113239     detached kernel driver from USB device...
   1.251394     HID descriptor, method 1: (9 bytes) => 09 21 00 01 21 01 22 01 03
   1.251460     HID descriptor, method 2: (9 bytes) => 09 21 00 01 21 01 22 01 03
   1.251491     HID descriptor length 769
   1.351379     Report Descriptor size = 769
   1.351456     Report Descriptor: (769 bytes) => 05 84 09 04 a1 01 09 24 a1 00 09 02 a1 00
   1.351509      55 00 65 00 85 01 75 01 95 05 15 00 25 01 05 85 09 d0 09 44 09 45 09 42 0b
(...)

So the issue is now clear and is related to permissions. I was able to fix this issue by changing the permissions on the USB device corresponding to my UPS:

$ chmod 0666 /dev/bus/usb/005/003

Another working way to fix this is to change the group of the device to nut:

$ chown :nut /dev/bus/usb/005/003

BTW, to get the bus number (005 here) and device number (003 in my case) of your UPS, run lsudb:

$ lsusb
Bus 005 Device 003: ID 0463:ffff MGE UPS Systems UPS
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Of course this fix is absolutely temporary, as you’ll need to perform the change above after every reboot. This is far from practical. In fact, as describe in this Fedora 10 bug report, but also in some other Debian bug report, this issue is directly tied to conflicting Udev rules.

Based on clues from these bug reports you can fix Udev using different strategies. As I can’t decide which one is the cleanest, I just did something that is quite brutal, but works. It consist of replacing in /lib/udev/rules.d/91-permissions.rules the line setting rights for USBfs-like devices:

--- /lib/udev/rules.d/91-permissions.rules-orig 2011-05-05 18:49:08.015538434 +0200
+++ /lib/udev/rules.d/91-permissions.rules      2011-05-05 18:49:16.663537978 +0200
@@ -33,7 +33,7 @@

 # usbfs-like devices
 SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", \
-                               MODE="0664"
+                               MODE="0666"

 # serial devices
 SUBSYSTEM=="tty",

Now all you have to do is to unplug the power cord and wait until your machine gracefully shut down as soon as batteries are low !

First, let’s install the smartmontools package:

$ aptitude install smartmontools

We now have to check that SMART is activated on our drives:

$ smartctl -a /dev/sdb | grep "SMART support is: Enabled"
$ smartctl -a /dev/sdc | grep "SMART support is: Enabled"
$ smartctl -a /dev/sdd | grep "SMART support is: Enabled"
$ smartctl -a /dev/sde | grep "SMART support is: Enabled"

Nowadays all modern drives are already activated in factories. But in case they’re not you can activate this feature with the following command:

$ smartctl -s on -a /dev/sdb

Now we have to activate the smartd daemon, and let it start automatically when the machine boot:

$ sed -i 's/#start_smartd=yes/start_smartd=yes/g'                           /etc/default/smartmontools
$ sed -i 's/#smartd_opts="--interval=1800"/smartd_opts="--interval=1800"/g' /etc/default/smartmontools

For some reason, I prefer to not let smartd select by itself drives it should check. Here is how to deactivate the auto-detection:

$ sed -i 's/^DEVICESCAN /#DEVICESCAN /g' /etc/smartd.conf

Now let’s specify by hand the way our drives should be monitored and maintained. Here are the lines I added to /etc/smartd.conf:

/dev/sdb -a -o on -S on -s (S/../.././02|L/../../6/03) -m kevin@deldycke.com
/dev/sdc -a -o on -S on -s (S/../.././04|L/../../6/05) -m kevin@deldycke.com
/dev/sdd -a -o on -S on -s (S/../.././06|L/../../6/07) -m kevin@deldycke.com
/dev/sde -a -o on -S on -s (S/../.././08|L/../../6/09) -m kevin@deldycke.com

Finally, we need to restart the SMART service to take into account all our changes:

$ /etc/init.d/smartmontools restart