As usual, let’s start by installing the main Munin package:

$ aptitude install munin

FYI, the version that aptitude choose to install was Munin 1.4.5. The default configuration coming along will make it produce graphs and HTML content to /var/cache/munin/www. Now we need to serve these pages via a web server.

As I wanted to play with nginx for a long time, I will use this opportunity to serve Munin’s content. The default version coming with Squeeze is quite old, so we’ll get the latest version from the Dotdeb repository:

$ echo "deb http://packages.dotdeb.org squeeze all" > /etc/apt/sources.list.d/squeeze-dotdeb.list
$ aptitude update
$ aptitude install nginx

And if you don’t want to get those error messages about untrusted packages, don’t forget to add Dotdeb’s keys to your keyring.

We can now test that nginx is working by starting it up then fetch the default served page:

$ /etc/init.d/nginx start
$ wget --output-document=- http://localhost | grep "Welcome to nginx"

Then we’ll disable the default nginx config and create a new one for Munin:

$ rm /etc/nginx/sites-enabled/default
$ touch /etc/nginx/sites-available/munin

In the latter, we put this minimal configuration:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  location / {
    index index.html;
    access_log off;
  }
}

Now we have to activate it before restarting nginx:

$ ln -s  /etc/nginx/sites-available/munin /etc/nginx/sites-enabled/munin
$ /etc/init.d/nginx restart

Now we are free to point our browser to the http://munin.example.com URL to get our graphs.

You’ll see that by default, Munin refer to your machine as localhost.localdomain. It’s time to tweak Munin a little to get nice reports:

$ sed -i 's/\[localhost\.localdomain\]/\[munin\.example\.com\]/g' /etc/munin/munin.conf

By default Munin activate a lot of great graphs. But I always find that some crucial monitoring are missing. Let’s add some more monitoring scripts:

$ aptitude install munin-plugins-extra

Here is a collection of general purpose graphs I automatically add to Munin:

$ ln -s /usr/share/munin/plugins/df_abs  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/netstat /etc/munin/plugins/
$ echo "[netstat]
user root
" > /etc/munin/plugin-conf.d/netstat

It’s also good to have a clue about your connectivity to the rest of the world:

$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_google.com
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_ovh.fr
$ ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_example.com

I also like to have insight about my automated backups:

$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_duplicity
$ ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_sshd

Monitoring temperatures, voltages and other hardware metrics is a must, unless your machine is a virtual server :

$ ln -s /usr/share/munin/plugins/cpuspeed         /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/acpi             /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/hddtemp_smartctl /etc/munin/plugins/
$ aptitude install i2c-tools lm-sensors
$ sensors-detect
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_temp
$ ln -s /usr/share/munin/plugins/sensors_ /etc/munin/plugins/sensors_volt

I sometimes have a Fail2Ban deamon running on a server, so that’s a good thing to monitor it:

$ ln -s /usr/share/munin/plugins/fail2ban /etc/munin/plugins/
$ echo "[fail2ban*]
user root
" > /etc/munin/plugin-conf.d/fail2ban

Having an UPS, it’s good to monitor it too. Here is for the UPS on the local system having the MGE-Ellipse750 ID (as defined in your /etc/nut/ups.conf file):

$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_voltages
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_charge
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_freq
$ ln -s /usr/share/munin/plugins/nutups_   /etc/munin/plugins/nutups_MGE-Ellipse750_current
$ ln -s /usr/share/munin/plugins/nut_misc  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nut_volts /etc/munin/plugins/
$ echo "[nut*]
user root

[nut_*]
env.upsname MGE-Ellipse750@localhost
" > /etc/munin/plugin-conf.d/nut

And if you have a MySQL server running on the machine, that’s a good idea to get stats:

$ ln -s /usr/share/munin/plugins/mysql_threads     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_slowqueries /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_queries     /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/mysql_bytes       /etc/munin/plugins/

I also use some other Munin plugins coming from Munin exchange:

$ wget http://exchange.munin-monitoring.org/plugins/mysql_size_all/version/1/download --output-document=/usr/share/munin/plugins/mysql_size_all
$ ln -s /usr/share/munin/plugins/mysql_size_all /etc/munin/plugins/

An here is how I monitor my RAID array:

$ wget http://exchange.munin-monitoring.org/plugins/raid/version/3/download --output-document=/usr/share/munin/plugins/raid
$ ln -s /usr/share/munin/plugins/raid /etc/munin/plugins/
$ echo "[raid]
user root
" > /etc/munin/plugin-conf.d/raid

Finally, it’s time to monitor nginx itself:

$ ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
$ ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
$ echo "[nginx_*]
env.url http://localhost/nginx_status
" > /etc/munin/plugin-conf.d/nginx

These two scripts above have some Perl module dependencies:

$ aptitude install libio-all-lwp-perl

If you don’t install the libraries above, you’ll get these kind of errors in /var/log/munin/munin-node.log:

2011/05/03-17:50:10 [2009] Error output from nginx_request:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_request line 106.
2011/05/03-17:50:10 [2009] Service 'nginx_request' exited with status 9/0.
2011/05/03-17:50:10 [2009] Error output from nginx_status:
2011/05/03-17:50:10 [2009]      Can't locate object method "new" via package "LWP::UserAgent" at /etc/munin/plugins/nginx_status line 109.
2011/05/03-17:50:10 [2009] Service 'nginx_status' exited with status 2/0.

But for this to work, we have to update the /etc/nginx/sites-enabled/munin file. Now it looks like this:

server {
  server_name munin.example.com;
  root /var/cache/munin/www/;
  # Restrict Munin access
  auth_basic "Restricted";
  auth_basic_user_file /etc/nginx/htpasswd;
  location / {
    index index.html;
    access_log off;
  }
}
server {
  allow 127.0.0.1;
  deny all;
  location /nginx_status {
    stub_status on;
    access_log off;
  }
}

Note that I’ve added a simple HTTP authentication to Munin webpages and restricted access to nginx statistics from the local machine only.

At last, before rebooting Munin and Nginx, make sure all downloaded plugins are executables. This is important and always forgotten:

$ chmod -R 755 /usr/share/munin/plugins/
$ /etc/init.d/nginx restart
$ /etc/init.d/munin-node restart

If you want one, please leave a comment below and don’t forget to fill the appropriate field with your email. I’ll then be able to contact you and send you the coupon’s code.

This article explains how I upgraded the firmware of my Enttec DMX/USB widget under linux thanks to Qemu.

First, plug your device in one of your computer’s USB port. We need to get the hardware UID of the widget. We can do so as root in a linux terminal:

cat /proc/bus/usb/devices

This command output a big mess in which you should find a block of lines separated by two blank lines (one above and one below) corresponding to your USB device. It’s easy to spot, as it contain the ENTTEC string. Mine look like this:

T:  Bus=02 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=12  MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=0403 ProdID=6001 Rev= 6.00
S:  Manufacturer=ENTTEC
S:  Product=DMX USB PRO
S:  SerialNumber=ENQXXXXX
C:* #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=300mA
I:* If#= 0 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=ftdi_sio
E:  Ad=81(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=02(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms

What we are looking for is the vendor’s ID and the product’s ID, that’s all Qemu needs to talk to the device. This is found on the line starting with P:. For me:

• Vendor ID: 0403
• Products ID: 6001

With this information, we can launch Qemu and bind it to the device. Assuming you already have a Qemu image containing a working version of windows XP, the command looks like this:

qemu -m 512 -usb -usbdevice host:0403:6001 -hda ./qemu-win-xp-with-freestyler.qcow

Alternatively, you can “hotplug” the USB device once inside Qemu. This can be done by calling the Qemu interactive shell by pressing Ctrl + Alt + 2 simultaneously. Then, to hotplug the USB device, type:

usb_add host:0403:6001

If you’re as unlucky as I am, you’ll get this error message:

Could not add USB device 'host:0403:6001'

Which is doubled by the following message from your legacy console:

/proc/bus/usb/002/002: Permission denied

The latter point to the restrictive access rights on our device, which can be fixed by:

chmod -R a+rw /proc/bus/usb/002/002

Instead, if you get the following error message:

usb_host: device already grabbed

It probably mean that your linux kernel has already identified the device when you plugged in and has loaded some drivers. To unload them and free the device, I had to do:

lsmod
rmmod dmx_usb
rmmod ftdi_sio

At last, you can check under the emulated Windows that your Enttec widget is recognized by windows:

And finally you’re free to upgrade (at your own risks) your widget’s firmware with the tools available on Enttec’s official website:

FYI, all these operations where performed on a Mandriva 2008.1, Qemu 0.9.0 and linux kernel 2.6.24.

By using a combination of following commands, I was soon aware of the gravity of the situation:

cat /proc/mdstat
mdadm --examine /dev/sda1

My /dev/sda1 disk was kicked out of the array, so I did the right stuff which consisted of reconstructing the array:

mdadm /dev/md0 -a /dev/sda1

Then, in an unlucky combination of cosmic ray bombardment, spooky action at a distance and astrological misalignment, half-way to the end of the rebuilding process (which can take up to 5 hours), another disk failed ! It was late, I was tired and utterly worried about losing 1.5 To of precious data. In such a bad shape, I was afraid to worsen the situation. So I decided to shutdown the server and sleep on the problem.

The next day I tried to boot my server to find it (surprise !) stuck in the middle of the boot process, with the famous message:

hit control-D to continue or give root password to fix manually

This is “normal” as my server tried to mount the ext3 filesystem from the /dev/md0 partition that was just assembled by mdadm. Of course md0, if assembled and available to the system, was not running because only one disk, out of three, was in a clean state.

I skip here the epic substory in which I wasted days in a search of a working keyboard, but I let you imagine how such adventures makes my week…

Eventually, I was able to analyze the situation in details. My first reflex ? Check that disks are not physically dead:

fdisk -l /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sdc

“Linux raid partitions” (type code “fd“) are still there. Good. I assumed here that disks where not physically damaged. Maybe I should have looked at S.M.A.R.T. datas and statistics (via smartmontools). But remember, I’m lazy (and a bit crazy).

The next step was to get informations about the RAID array itself using:

mdadm --detail /dev/md0

which output the status table below (probably inaccurate as I reconstructed it afterwards):

Number   Major   Minor   RaidDevice State
   0       0        0        0      removed
   1       0        0        1      faulty removed
   2       8       33        2      active sync   /dev/sdc1
   3       8       17        3      spare

What this table told us ?

• The array is up, but not running. One of its device (sdc1) was clean and active, but it’s not enough to get a working RAID-5.
• My first attempt to rebuild the array lead to an unexpected result: it added sda1 as a spare device (in slot #3).
• It confirm that sdb1 unexpectedly failed and is now in a bad state (“faulty removed“).

Then I stopped the array and tried to fearlessly (re)assemble it using 3 differents methods:

mdadm -S /dev/md0
mdadm -A /dev/md0
mdadm --assemble /dev/md0 --verbose /dev/sd[abc]1
mdadm --assemble --force --scan /dev/md0 --verbose

It always failed with messages like:

mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: /dev/md0 assembled from 1 drives and 1 spare - not enough to start the array.

So I examined each drive from mdadm‘s point of view:

mdadm -E /dev/sda1
mdadm -E /dev/sdb1
mdadm -E /dev/sdc1
mdadm -E /dev/sd[abc]1 | grep Event

The lastest command compare the “Event” attribute of all devices. It output something like:

Events : 0.53120
Events : 0.53108
Events : 0.53120

which indicate that sda1 and sdc1 are somewhat synced (share the same number) and sdb1 “late” (lower number).

Here I’ve got the idea of recreating the raid array without sdb1, relying only on sda1 and sdc1, by using the “magic” (hence dangerous) --assume-clean option. The latter doesn’t build, erase or initialize a new array. It just try to assemble it “as is”. Here is the command:

mdadm --create /dev/md0 --assume-clean --level=5 --verbose --raid-devices=3 /dev/sda1 missing /dev/sdc1

And it worked !

I mounted the md0 partition and cleaned it up:

fsck.ext3 -v /dev/md0
mount /dev/md0

I updated my mdadm configuration before rebooting my server:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
vi /etc/mdadm/mdadm.conf
reboot

But history repeat itself, and again, the system hang up during boot. Except this time I knew what was happening: the boot process detected the remaining sdb1 device as part of the old array (the one before the regeneration I did above) and tried to run it. Remembering my last year post, I zero-ized the superblock of sdb1:

mdadm -S /dev/md0
mdadm --zero-superblock /dev/sdb1

A server reboot proved I was right and my md0 partition was automagically mounted in altered state:

localhost:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sda1[0] sdc1[2]
      1465143808 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]

unused devices: <none>

I just had to re-add sdb1 to fill the available slot and update the mdadm configuration to get back my array in its initial state:

mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
vi /etc/mdadm/mdadm.conf

I’ve upgraded my TSSTcorp CD/DVDW TS-L632D drive from the SC02 to SC04 firmware revision and my laptop is stable now !

By the way, to get the model name and revision of your optical drive under linux, use:

cdrdao scanbus

Assuming that this situation was about an inconsistent file index, I decided to reset the superblocks of the remaining physical disks:

mdadm --zero-superblock /dev/hdc1
mdadm --zero-superblock /dev/hdb1

I don’t know why I decided to do so, but it was the stupidest idea of the week. After such a violent treatment, my array refused to start:

[root@localhost ~]$ mdadm --assemble /dev/md0 --auto --scan --update=summaries --verbose
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/hdc1
mdadm: /dev/hdc1 has wrong raid level.
mdadm: no RAID superblock on /dev/hdb1
mdadm: /dev/hdb1 has wrong raid level.
mdadm: no devices found for /dev/md0

At this moment I was sure that all my data assets were lost. I was desperate. My only alternative was to ask Google. So I did.

I spend several minutes browsing the web without hope. I finally found someone in the same situation as mine (sorry, in french) on debian-user-french mailing list.

The solution was to recreate the RAID array. This sound counter-intuitive: if we recreate a raid array over an existing one, it will be erased ! Right ? Wrong ! As it is said on debian-user-french, mdadm is smart enough to “see” that HDD of the new array were elements of a previous one. Knowing that, mdadm will try to do its best (i.e. if parameters match the previous array configuration) and rebuild the new array upon the previous one in a non-destructive way, by keeping HDD content.

So, here is how I finally recovered my RAID array:

[root@localhost ~]$ mdadm --create /dev/md0 --verbose --level=5 --raid-devices=3 /dev/hdc1 missing /dev/hdb1
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 312568576K
mdadm: array /dev/md0 started.

Of course this doesn’t solve my initial problem about the /dev/md0 file system: it is still in an altered state. Maybe it’s too late to recover data. But at least I reverted all my today’s mistakes, and the situation will not deteriorate until I power up my RAID !

The hard disk of the machine is supposed to be a 120 GB one, and as usual a hidden partition is reserved for windows backups. I deleted this one and reformatted the whole disk. After this operation, fdisk (fdisk -l /dev/hda) report me a total usable space of 112.4 GiB. Not bad.

Now I’ll tell you what I noticed about the aesthetic and the usability of the machine itself.

The touchpad has a good soft touch, but is located so close of the keyboard that I inadvertently activate it while typing. This is quite bad, especially when you write long texts. I think with some practice I’ll be able to move my fingers to avoid this unintended behavior.

I feared having a glossy screen but I’m finally happy. This one is not as reflexive as I thought, and bright colors are welcomed for watching movies. I still hadn’t the opportunity to test it in a sunny environment. October is not the sunniest month in the north of France !

To keep the machine as compact as possible, some keys of the keyboard were located in unusual places. Page Up / Page Down keys are for example on the right and on the left of the Upper key. Home and End need the function (Fn) key to work. This slow me down because I use them a lot when programming or editing blog posts.

The fan is located on the right side of the machine. I’m right handed so it’s not cool (pun intended). In reality it’s not a problem at all since the fan is not often activated during “normal” use (Internet surfing, mails, etc…).

I don’t like having sounds connectors in front of the machine.

The battery is unpleasant because half of it appear outside the laptop case. But having twice the capacity (6 cells instead of 3) is welcomed.

Finally the laptop is not ultra-mobile. It’s a very compact laptop (only 1.9 Kg), but not “ultra”: it has big margins around the keyboard and the screen, and is as large as usual 14” laptops. It is also as thick as others, because of the optical drive.

My Conclusion: If you need an ultra light laptop or if you are a gamer, this is not a machine for you. But if you need a light-weight notebook with everything heavy laptops have (including the optical drive), it’s a very good deal. I paid mine 1160 € on a French online store. This laptop is not an object of design. But for 100 € more than a 13” MacBook, you have a compact machine with superior hardware: Core 2 Duo, 1GB of RAM, 120GB HDD, 7 hours of max autonomy and 1280×800 screen. This is why I choose this machine, and I’m very happy with it !

I had the project to setup a RAID 5 array using 3*120 Gb hard drives in USB enclosures. Unfortunately my project stalled due to instability in early 2.6.x kernels (I heard that 2.6.12 and upper are now useable for “RAID over USB”).

Because of the urgency of reliable storage (and because I don’t want to waste time compiling and fine-tuning kernels), I decided to do it using traditionnal IDE host. So I plugged two 120Gb HDD on my machine as master device, one on each IDE channel.

Then I made a big XFS partition on each, and update my /etc/fstab:

/dev/sda1 /                auto  noatime   1 1
/dev/hda1 /mnt/hd1         xfs   defaults  1 2
/dev/hdc1 /mnt/hd1_mirror  xfs   defaults  1 2

At that moment I have to explain you that my machine is an OpenBrick NG, with a USB 2.0 512 Mb thumb drive (/dev/sda1 in the fstab) on which all my linux system is installed. That explain why my two IDE channels are free for use.

The idea is now to use /mnt/hd1 to store and manipulate my datas, then rsync that drive with his alter-ego (/mnt/hd1_mirror) every night. To do that, I’ve just added the following command in a cron entry:

rsync -a --delete --delete-excluded --delete-after /mnt/hd1/ /mnt/hd1_mirror/

And voilà !

As you guess, this solution is far from perfect, and has major inconvenients regarding RAID 1:

• No immediate backup : the backuped datas are 1-day old;
• Seek time is not reduce by half;
• Transfer rate is not doubled.

Oh, and by the way, be careful to not write files on /mnt/hd1_mirror/ because they will be deleted each night during the mirroring process.