The goal was to create a back-end talking to Microsoft’s Live Connect API and keep a cache of user profiles. Then a front-end demonstrating my HTML/CSS/JS know-how was to be built. User authentication was supposed to use OAuth.

The only technological constraint was to use Python. I decided to use CherryPy and Mako to leverage the boilerplate code I just released back then. For the persistent layer, my first intention was to use SQLAlchemy, but quickly switched to MongoDB as I never played with it and this project was a great opportunity to.

If my web app was far from finished, it was still well-received by the team. After other interviews I was made an competitive offer. I finally declined as I wanted to finish what I stated at my current company.

What’s left of this experience is Live Browser, the web app I created, which source code is now available on GitHub.

These articles are independent with each other, meaning you can pick the one your interested in to customize your server and ignore the others.

1. Setup SMART monitoring tool for HDDs.
2. Setup Nut to manage the UPS.
3. Setup Duplicity and Amazon S3 for cloud-based backups.
4. Setup Exim to relay mails via Gmail.
5. Setup cron-apt to keep our distribution up to date.
6. Add a fail2ban deamon.
7. Setup Munin to monitor our machine.
8. Basic setup of Nginx + PHP-FPM + MySQL web stack.
9. Optimizing Nginx + PHP-FPM + MySQL for performances.
10. Setup PHP APC op-code cache.
11. Install haveged to get lots of entropy.
12. Setup a WebDAVs server with Lighttpd.
13. Setup Mailman + Nginx + Exim for mailing-lists.
14. Mailman mailing-list migration and merging.

The setup I’ll detail below runs on an OVH VPS instance. This virtual server has 4 CPU cores at 1.5GHz, 1 Go RAM and 50 Gb HDD.

I’m mostly running WordPress instances on that server, so you’ll see some reference of it in this post.

MySQL

First, let’s tune MySQL. That’s the easiest part of that article, as you only need to create a .cnf file in /etc/mysql/conf.d/ and place there all your custom parameters. Here is the content of my /etc/mysql/conf.d/kev.cnf:

[mysqld]
interactive_timeout = 50
join_buffer = 1M
key_buffer = 250M
max_connections = 100
max_heap_table_size = 32M
myisam_sort_buffer_size = 96M
query_cache_limit = 4M
query_cache_size = 250M
query_prealloc_size = 65K
query_alloc_block_size = 128K
read_buffer_size = 1M
read_rnd_buffer_size = 768K
sort_buffer_size = 1M
table_cache = 4096
thread_cache_size = 1024
tmp_table_size = 32M
wait_timeout = 500
# Debug
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
# InnoDBinnodb_buffer_pool_size = 256Minnodb_additional_mem_pool_size = 10Minnodb_log_file_size = 32Minnodb_flush_method = O_DIRECTinnodb_file_per_table = 1innodb_flush_log_at_trx_commit = 0
[mysqld_safe]
nice = -5
open_files_limit = 8192

[isamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

Most of these parameters were set for my particular usage and with insights from the MySQL Tuning Primer Script.

PHP-FPM

Unlike MySQL, the structure of PHP configuration files on Debian Squeeze doesn’t let us easily add our customizations. We have to modify the default files provided at the package installation.

Here is my setup of the PHP processes pool:

--- /etc/php5/fpm/pool.d/www.conf.orig     2011-06-07 08:14:30.000000000 +0200
+++ /etc/php5/fpm/pool.d/www.conf  2011-08-15 17:34:09.000000000 +0200
@@ -237,3 +237,10 @@
 ;php_admin_value[error_log] = /var/log/fpm-php.www.log
 ;php_admin_flag[log_errors] = on
 ;php_admin_value[memory_limit] = 32M
+
+pm.max_children = 25
+pm.start_servers = 4
+pm.min_spare_servers = 2
+pm.max_spare_servers = 10
+pm.max_requests = 500
+request_terminate_timeout = 30

The second customization I made is not about performances but convenience. It just allow my WordPress’ users to upload larger files:

--- /etc/php5/fpm/php.ini.orig      2011-06-18 13:32:37.000000000 +0200
+++ /etc/php5/fpm/php.ini   2011-06-22 22:50:49.000000000 +0200
@@ -725,7 +725,7 @@

 ; Maximum size of POST data that PHP will accept.
 ; http://php.net/post-max-size
-post_max_size = 8M
+post_max_size = 15M

 ; Magic quotes are a preprocessing feature of PHP where PHP will attempt to
 ; escape any character sequences in GET, POST, COOKIE and ENV data which might
@@ -876,7 +876,7 @@

 ; Maximum allowed size for uploaded files.
 ; http://php.net/upload-max-filesize
-upload_max_filesize = 2M
+upload_max_filesize = 15M

 ; Maximum number of files that can be uploaded via a single request
 max_file_uploads = 20

Nginx

Let’s say my WordPress blog is installed in /var/www/my_wordpress. To let it be served by Nginx, we add a configuration file for this site in /etc/nginx/sites-available/my_wordpress:

server {
  server_name blog.example.com;
  root /var/www/my_wordpress/;
  include /etc/nginx/wordpress.conf;
  location /static {
    autoindex on;
  }
}

server {
  listen 80 default_server;
  server_name .example.com .example.org .example.net;
  rewrite ^ http://blog.example.com$request_uri? permanent;
}

In the configuration above, you can see that I want my blog to be served at http://blog.example.com. I also added some domain redirections in the form of a second server section, and a way to better display my static file repository by letting Nginx generate index pages.

Then don’t forget to activate this site:

$ ln -s /etc/nginx/sites-available/my_wordpress /etc/nginx/sites-enabled/

The file above refer to /etc/nginx/wordpress.conf which is where I place all the configuration directives common to all the WordPress blogs on my server. Here is the content of that file:

# This order might seem weird - this is attempted to match last if rules below fail.
# See: http://wiki.nginx.org/HttpCoreModule
location / {
  try_files $uri $uri/ /index.php?q=$uri&$args;
}

# Add trailing slash to */wp-admin requests.
rewrite /wp-admin$ $scheme://$host$uri/ permanent;

include global.conf;

include php.conf;

Again, this file make a reference to php.conf, which is the same as the one featured in my previous article. I only removed the index directive to place it elsewhere, and added a limit on the number of PHP requests a client can make:

location ~ \.php$ {
  # Throttle requests to prevent abuse
  limit_req zone=antidos burst=5;

  # Zero-day exploit defense.
  # http://forum.nginx.org/read.php?2,88845,page=3
  # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
  # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
  try_files $uri =404;

  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  include /etc/nginx/fastcgi_params;

  # As explained in http://kbeezie.com/view/php-self-path-nginx/ some fastcgi_param are missing from fastcgi_params.
  # Keep these parameters for compatibility with old PHP scripts using them.
  fastcgi_param PATH_INFO       $fastcgi_path_info;
  fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

  # Some default config
  fastcgi_connect_timeout        60;
  fastcgi_send_timeout          180;
  fastcgi_read_timeout          180;
  fastcgi_buffer_size          128k;
  fastcgi_buffers            4 256k;
  fastcgi_busy_buffers_size    256k;
  fastcgi_temp_file_write_size 256k;

  fastcgi_intercept_errors    on;
  fastcgi_ignore_client_abort off;

  fastcgi_pass 127.0.0.1:9000;
}

Here is where the index directive moved: /etc/nginx/conf.d/kev.conf. I also added there some tweaks and the global request throttling configuration:

# Hide Nginx version
server_tokens off;

# Set default index file names
index index.php index.html index.htm;

# Allow uploads up to 15 Mo
client_max_body_size 15m;

# Create a global request accounting pool to prevent DOS
limit_req_zone $binary_remote_addr zone=antidos:10m rate=3r/s;

The global.conf file we saw in /etc/nginx/wordpress.conf refer to /etc/nginx/global.conf, which contain additional measures to remove cruft from log files and enhance security:

# Do not log excessive request on common web content like favicon and robots.txt
location = /favicon.ico {
  log_not_found off;
  access_log off;
}
location = /robots.txt {
  allow all;
  log_not_found off;
  access_log off;
}

# Deny all attempts to access any dotfile (=hidden files) such as .htaccess, .htpasswd, .DS_Store, .directory, .svn, .git, ...
location ~ /\. {
  deny all;
  access_log off;
  log_not_found off;
}

All of default Nginx configuration can’t be overridden by additional files. We have to change /etc/nginx/nginx.conf itself:

--- /etc/nginx/nginx.conf.orig   2011-06-06 00:46:56.000000000 +0200
+++ /etc/nginx/nginx.conf        2011-08-15 17:44:58.000000000 +0200
@@ -3,8 +3,9 @@
 pid /var/run/nginx.pid;

 events {
-       worker_connections 768;
-       # multi_accept on;
+       use epoll;
+       worker_connections 1024;
+       multi_accept on;
 }

 http {
@@ -16,7 +17,7 @@
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
-       keepalive_timeout 65;
+       keepalive_timeout 3;
        types_hash_max_size 2048;
        # server_tokens off;

That’s all for our customizations. We can now restart all our servers:

$ /etc/init.d/mysql restart
$ /etc/init.d/php5-fpm restart
$ /etc/init.d/nginx restart

Conclusion

I’m running my websties under this configuration for about 3 months and I’m really happy with the results. I’m sure I can push optimizations further, but it may require lots of time and effort compared to the marginal gain I’ll get. My websites are responsive enough to me. And if they collapse in the future under the load of the Reddit crowd, I’ll still have the option to move to a bigger virtual server (vertical scaling FTW!).

Everything will be great after that. Until the day this repository is updated, which will break the Flash plugin if you attempt an upgrade with KPackageKit.

The Flash package does not contain the binary plugin itself, but is just an empty shell which download the plugin from the web to your machine. And for this operation to work as expected, the package need to be in a terminal environment.

To fix this, you’ll have to remove all previous Flash package (as a prevention measure), then install the latest with the command line. Here are the commands to do exactly this:

sudo apt-get remove --purge flashplugin-installer flashplugin64-installer
sudo apt-get clean
sudo apt-get update
sudo apt-get install flashplugin64-installer

Commands above are for a 64 bits distribution. If you’re still running a 32 bits Linux, just replace the last reference of flashplugin64-installer by flashplugin-installer.

Now restart your browser and YouTube videos and other Flash stuff should work again.

WebPing is a script I started to work on in 2009 while working at EDF. Back then, I needed a monitoring tool to keep an eye on the 80+ Plone instances that my team managed. For several corporate reasons, I wasn’t allowed to use a proper monitoring tool like Munin or Nagios. So I created a small script to fill this need. That’s how WebPing came to be.

WebPing is just a stupid Python script that is designed to be ticked regularly by a cron job. It try to fetch a list of URLs and store response times in an SQLite database. Then it create a static HTML report you’re free to serve with any HTTP server (an example Apache configuration is provided). The configuration of WebPing and the list of URLs it monitor is stored in a YAML file.

The produced HTML report use the Flot jQuery plugin to render graphs. Here is how the dashboard looks like:

Finally, WebPing is able to send reports and alerts by emails. Here is how a mail alert looks like:

Since I created WebPing, I found several other projects more or less developed around the same idea. See Kong, which is based on Django and Twill, a web-oriented DSL. Another project I spotted after the facts was multi-mechanize. Like Kong, it’s written in Python. But I never played with one or the other.

First, install the required packages:

$ aptitude install lighttpd-mod-webdav

As we want to provide a secure WebDAV access, we need to install OpenSSL:

$ aptitude install openssl

Then we create the file /etc/lighttpd/clear-creds.lst, that will contain credentials required for authentication, under the following form:

user1:password1
user2:password2
user3:password3

Logins and passwords are stored here in clear. This is stupid, but for this project I was looking to setup a quick and dirty server. For temporary tests this setup is OK, but I encourage you to switch to a better credential storage system.

Now I want to serve WebDAV content within a secure channel. A self-signed SSL certificate will be enough. Let’s generate one:

$ cd /etc/lighttpd/
$ openssl req -x509 -nodes -subj '/' -days 3650 -newkey rsa:2048 -keyout server.pem -out server.pem

We’ll configure Lighttpd by loading the default parameters of modules we use:

$ cd /etc/lighttpd/conf-enabled/
$ ln -s ../conf-available/05-auth.log
$ ln -s ../conf-available/10-ssl.conf
$ ln -s ../conf-available/10-webdav.conf

Now I create a custom configuration file:

$ touch /etc/lighttpd/conf-available/99-custom.conf
$ cd /etc/lighttpd/conf-enabled/
$ ln -s ../conf-available/99-custom.conf

Here is the content of that 99-custom.conf configuration file:

# Hide server version
server.tag = "lighttpd"

# Force all request to be in HTTPs
# This also redirects all WebDAV requests to WebDAVs
$HTTP["scheme"] == "http" {
  $HTTP["host"] =~ "(.*)" {
    url.redirect = ( "^/(.*)" => "https://%1/$1" )
  }
}

# Valid credentials are required for any request
auth.backend = "plain"
auth.backend.plain.userfile = "/etc/lighttpd/clear-creds.lst"
auth.require = (
  "/" => (
    "method" => "digest",
    "realm" => "My WebDAV server",
    "require" => "valid-user"
  )
)

# Enable WebDAV in read and write mode
webdav.activate = "enable"
webdav.is-readonly = "disable"

# Customize directory listings a bit
dir-listing.set-footer = "<a href='http://example.com'>Company</a>'s document repository."

And do not forget to restart the server:

$ /etc/init.d/lighttpd restart

As you can see in the screenshot above, you can now:

• Browse the file system in read/write mode with a WebDAV client via a webdavs://12.34.56.78/ URL;
• Access content in read-only mode with a browser by a classic https://12.34.56.78/ URL.

$ aptitude install php5-apc

In my case this package come from the PHP bundle distributed by the Dotdeb repository.

If installing APC is easy, monitoring it with Munin requires some extra manipulations. There is currently no good APC plugin available on Munin Exhange. So we’ll use the external munin-php-apc project instead.

The latter can’t get APC statistics by itself: it need an extra PHP file to be served locally. As you can read in my previous article, my Munin is powered by Nginx. So now we’ll setup Nginx to serve this extra PHP file:

$ mkdir -p /var/www/apc
$ cd /var/www/apc
$ wget http://munin-php-apc.googlecode.com/svn/trunk/php_apc/apc_info.php
$ chown -R www-data:www-data /var/www/apc

Then I need to update my /etc/nginx/sites-available/munin file (see details about this file on my previous article) to have the second server section look like this:

server {
  server_name localhost;
  include /etc/nginx/php.conf;
  root /var/www/apc;
  allow 127.0.0.1;
  deny all;
  location / {
    access_log off;
  }
  location /nginx_status {
    stub_status on;
    access_log off;
  }
}

Here the included /etc/nginx/php.conf file is the one in which I’ve concentrate all the Nginx directives required to activate PHP file parsing. The content and the mechanism behind this file is describe in my article on setting up Nginx with PHP-FPM.

Let’s get back to our Munin monitoring setup. I can restart now Nginx and check that I can access locally to my raw statistics:

$ /etc/init.d/nginx reload
$ wget http://localhost/apc_info.php
$ wget http://localhost/nginx_status

The last step is to install and configure the Munin plugin:

$ aptitude install libwww-perl
$ wget http://munin-php-apc.googlecode.com/svn/trunk/php_apc/php_apc_ --output-document=/usr/share/munin/plugins/php_apc_
$ chmod -R 755 /usr/share/munin/plugins/
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_usage
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_hit_miss
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_purge
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_fragmentation
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_files
$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_rates
$ echo "[php_apc_*]
user root
env.url http://localhost/apc_info.php?auto
" > /etc/munin/plugin-conf.d/php_apc
$ /etc/init.d/munin-node restart

And finally, after a while, you’ll get those beautiful graphs:

This is a special release as it ends the active development cycle. I officially declare this plugin unmaintained. I don’t plan to work on it again.

The 1.4 will be the last version. Unless you send me code contribution, you’ll not see a 1.5 version.

The reason behind this decision is simple and straightforward: I’ve recently migrated my last e107 website to WordPress. Now that e107 is no longer part of my life, I have no interest in spending time and energy working on this plugin.

So if you plan to get rid of your e107 sites in favor of WordPress, do the migration now, as the plugin will self-deteriorate as WordPress API gets updated.

First, we’ll get all our packages from an up-to-date DotDeb repository. If this is not already done, add those repositories to aptitude:

$ echo "deb http://packages.dotdeb.org squeeze all" > /etc/apt/sources.list.d/squeeze-dotdeb.list
$ gpg --keyserver keys.gnupg.net --recv-key 89DF5277
$ gpg -a --export 89DF5277 | apt-key add -
$ aptitude update

Now we can install the whole stack:

$ aptitude install nginx
$ aptitude install php5-fpm php5-mysql php5-gd php5-curl
$ aptitude install mysql-server

FYI, here is the list of versions I installed:

• Nginx 1.0.2
• PHP 5.3.6
• MySQL 5.1.57

As a way to test that our setup is working, we’ll serve a simple PHP file:

$ mkdir -p /var/www/example.com/
$ cd /var/www/example.com/
$ echo "
<?php phpinfo(); ?>
" > ./index.php
$ chown -R www-data:www-data /var/www

Now let’s create a minimal Nginx configuration file for this site:

$ touch /etc/nginx/sites-available/example.com

In this brand new file, put the following directives:

server {
  server_name example.com;
  include /etc/nginx/php.conf;
  location / {
    root /var/www/example.com/;
    access_log on;
  }
}

This will only work if you’ve updated your DNS with an A record having example.com redirecting to the IP address of your Nginx server.

Now it’s time to create the /etc/nginx/php.conf file referenced in the Nginx configuration above. This file is where I put the generic setup making the bridge between Nginx and PHP-FPM. Here is what it should contain:

index index.php index.html index.htm;

location ~ \.php$ {
  # Zero-day exploit defense.
  # http://forum.nginx.org/read.php?2,88845,page=3
  # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
  # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
  try_files $uri =404;

  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  include /etc/nginx/fastcgi_params;

  # As explained in http://kbeezie.com/view/php-self-path-nginx/ some fastcgi_param are missing from fastcgi_params.
  # Keep these parameters for compatibility with old PHP scripts using them.
  fastcgi_param PATH_INFO       $fastcgi_path_info;
  fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

  # Some default config
  fastcgi_connect_timeout        60;
  fastcgi_send_timeout          180;
  fastcgi_read_timeout          180;
  fastcgi_buffer_size          128k;
  fastcgi_buffers            4 256k;
  fastcgi_busy_buffers_size    256k;
  fastcgi_temp_file_write_size 256k;

  fastcgi_intercept_errors    on;
  fastcgi_ignore_client_abort off;

  fastcgi_pass 127.0.0.1:9000;
}

Finally you can activate the site configuration and restart the whole stack:

$ ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/
$ /etc/init.d/mysql restart
$ /etc/init.d/php5-fpm restart
$ /etc/init.d/nginx restart

If everything’s OK on your DNS, pointing your browser to http://example.com will show you the famous page produced by phpinfo():

Note that MySQL doesn’t need any special attention to make it work out of the box. But again, if you plan to use it in production, its configuration needs special care, as for Nginx and PHP.

$ aptitude install fail2ban

Then I simply create a local configuration file where I’ll put all my custom config:

$ touch /etc/fail2ban/jail.local

Here is the content of that file:

[DEFAULT]
# Do not filter connexion from my apartment and from the server itself
ignoreip  = 127.0.0.1 88.123.123.123 91.123.123.123
# Ban for a week
bantime   = 604800
maxretry  = 3
destemail = kevin@deldycke.com
banaction = iptables-allports
action    = %(action_mwl)s

[ssh]
enabled  = true
port     = 22
maxretry = 2

[ssh-ddos]
enabled = true
port     = 22

[apache]
# Apache basic auth
enabled   = true
maxretry  = 3
# Ban for 1 hour
bantime   = 3600

[apache-noscript]
enabled = true

[apache-overflows]
enabled = true

[apache-badbots]
enabled  = true
filter   = apache-badbots
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[apache-nohome]
enabled  = true
filter   = apache-nohome
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

[exim]
enabled  = true
filter   = exim
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

[exim-relay]
enabled  = true
filter   = exim-relay
port     = smtp,ssmtp
action   = iptables-allports
logpath  = /var/log/exim*/rejectlog
maxretry = 1

While adjusting Fail2Ban, I was surprised by how sensitive this software is. It can just refuse to start without any notice in the log or on the command line. Even if its log_level variable is set to 4 (= DEBUG) in /etc/fail2ban/fail2ban.conf.

In such a case, a sure way to find the culprit is to use a brute force debugging method: first set all the enabled variable of your jail.local‘s sections to false. Then activate one section after another until Fail2Ban refuse to restart.

For me, the problem was that I forgot to add my custom exim-relay filter to Fail2Ban. So I fixed my issue by creating an empty file at /etc/fail2ban/filter.d/exim-relay.conf in which I pasted the following content:

# Based on default exim.conf filter by Cyril Jaquier
# Real life exemaple:
# 2009-07-02 08:16:42 H=118-167-129-21.dynamic.hinet.net (91.121.198.84) [118.167.129.21] F=<titieueue@hotmail.com> rejected RCPT <s2288@mail2000.com.tw>: relay not permitted

[Definition]

# Option:  failregex
# Notes.:  regex to match use of my exim mail server as a relay it does not
#          allow.
# Values:  TEXT
#
failregex = \[<HOST>\] .*(?:relay not permitted)

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Speaking of custom filters, here is one to filter DFind scans (file located at /etc/fail2ban/filter.d/apache-w00tw00t.conf):

# Based on http://howflow.com/tricks/block_w00tw00t_scan_hosts_with_fail2ban
# Real life exemaple:
# [Sat Jun 27 16:43:08 2009] [error] [client 94.23.57.77] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Definition]

# Option:  failregex
# Notes.:  regex to match the w00tw00t scan messages in the logfile.
# Values:  TEXT
failregex = ^.*\[client <HOST>\].*w00tw00t\.at\.ISC\.SANS\.DFind.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

And here is the corresponding section from my jail.local file:

[apache-w00tw00t]
enabled  = true
filter   = apache-w00tw00t
action   = iptables-allports
logpath  = /var/log/apache*/*error.log
maxretry = 1