I’ve just replaced my script with duplicity, a tool written by the same author of rdiff-backup. And Amazon S3 cloud storage replaced my home server. Here is how I did it.

First, we need to create an account on Amazon AWS. This is easy and fast. My account was activated in minutes.

Now that you have access to Amazon’s cloud, let’s create a bucket on S3. I used the reversed domain name of the server, which give me a bucket name like com.example.server.backup. With this naming scheme, I can identify the purpose of the bucket by its label only.

Duplicity can use the cheaper RRS storage, but you need at least version 0.6.09. Having a Debian Squeeze, the only way to get a recent version is to install it from the backports:

$ apt-get -t squeeze-backports install duplicity python-boto

Then I created a simple symmetric key with GPG:

$ gpg --gen-key

You absolutely need to provide a passphrase, else Duplicity will refuse to run.

Now update the script below with the GPG key passphrase and your AWS credentials:

# Do not let this script run more than once
[ `ps axu | grep -v "grep" | grep --count "duplicity"` -gt 0 ] && exit 1

# Set some environment variables required by duplicity
export PASSPHRASE=XXXXXXXXXX
export AWS_ACCESS_KEY_ID=XXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXX

# ~/.cache/duplicity/ should be excluded, as explained in http://comments.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4449
PARAMS='--exclude-device-files --exclude-other-filesystems --exclude **/.cache/** --exclude **/.thumbnails/** --exclude /mnt/ --exclude /tmp/ --exclude /dev/ --exclude /sys/ --exclude /proc/ --exclude /media/ --exclude /var/run/ --volsize 10 --s3-use-rrs --asynchronous-upload -vinfo'
DEST='s3+http://com.example.server.backup'

# Export MySQL databases
mysqldump --user=root --opt --all-databases > /home/kevin/mysql-backup.sql

# Do the backup
duplicity $PARAMS --full-if-older-than 1M / $DEST

# Clean things up
duplicity remove-older-than 1Y --force --extra-clean $PARAMS $DEST

# Remove temporary environment variables
unset PASSPHRASE
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY

Before running duplicity, the script will dump all MySQL databases to a plain-text file. Then the first duplicity call will do the backup itself, and the second call will remove all backup older than a year.

I saved the script above in /home/kevin/s3-backup.sh and cron-ed it:

$ chmod 755 /home/kevin/s3-backup.sh
$ echo "
# Backup everything to an Amazon S3 storage
0 1 * * * root /home/kevin/s3-backup.sh
" > /etc/cron.d/s3-backup

I can now sleep better knowing all the work I do on my server will not be lost in case of a catastrophic event. Amazon S3 is today a no-brainer for server backups: your data will be secured and available. And for small quantity of data (like the 10 Go of my server), it’s incredibly cheap. Especially if you compare it to the cost of maintaining a storage server at home.

This solution is so good and obvious, that I don’t know why I haven’t implemented it earlier…

Before erasing and selling my iPhone, I want to backup and extract all the data I produced with it and that is still trapped inside. This mean photos, SMSs, voice messages, safari bookmarks, etc…

There is a nice OSX app simply called iPhone Backup Extractor which let you get these data. Instead of getting data directly from the iPhone, it reads its backups made by iTunes.

So first thing you have to do is to backup your phone using iTunes:

Then you can download and run the iPhone Backup Extractor app:

Here you just have to click the Read Backups button to get a list of all backups available on your machine. Then choose your latest backup:

You’ll get a list of all installed applications on your iPhone. As we are interested in “core” iPhone apps (SMSs, photos and so on), we’ll choose the “iOS Files” item, then choose a place where to extract:

Then the extraction itself will take place:

You’ve just finished the essential part of the process. You now have a nice folder structure containing all the important informations that was trapped in your phone:

Let’s browse the file structure that was just created. You can see photos are available as is, in the /iOS Files/Media/DCIM/XXXAPPLE/:

Most of other datas are located in the /iOS Files/Library/ folder. For example here are voice messages:

Again, .amr files here are playable as-is, like VLC or mplayer.

And finaly most, if not all, other kind of data and metadata are stored in SQLite databases (.db files). The best GUI I found to manipulate with these files under Mac OSX is SQLite Database Browser. See how I can easily extract to a CSV file all metadatas associated with my voice messages:

If screenshots were taken with a french version, instructions given here are for the english one. This will give you enough clues to perform the export whatever the localisation is. The Lotus Notes version I used was the 7.0.2 release.

So first, let’s start Notes and open your mailbox. You should be on a screen similar to this one:

Then, go to the File › Database › New Copy menu:

And you’ll get an export screen that’ll let you choose where to create a local copy of your database:

This will generate a .nsf file containing all your current mail.

Now that you have a backup, you are free to delete all your mails in Lotus Notes. By following this procedure regulary, you can create yearly or monthly archives of you mails without reaching the mailbox quota ! For example, this is how my local archive folder looks like:

The trick is to know that cron need percents to be escaped by a backslash in the command zone. For example, here is my crontab entry to create every 10 minutes a local backup of an important project file I currently work on:

*/10 * * * * kevin cp "/home/kevin/Desktop/Projects/Very Important Project/project.file" "/home/kevin/Desktop/Projects/Very Important Project/project.file-backup-`date +\%s`"

Quick and dirty, but may saves you precious time on unstable machines !

By using a combination of following commands, I was soon aware of the gravity of the situation:

cat /proc/mdstat
mdadm --examine /dev/sda1

My /dev/sda1 disk was kicked out of the array, so I did the right stuff which consisted of reconstructing the array:

mdadm /dev/md0 -a /dev/sda1

Then, in an unlucky combination of cosmic ray bombardment, spooky action at a distance and astrological misalignment, half-way to the end of the rebuilding process (which can take up to 5 hours), another disk failed ! It was late, I was tired and utterly worried about losing 1.5 To of precious data. In such a bad shape, I was afraid to worsen the situation. So I decided to shutdown the server and sleep on the problem.

The next day I tried to boot my server to find it (surprise !) stuck in the middle of the boot process, with the famous message:

hit control-D to continue or give root password to fix manually

This is “normal” as my server tried to mount the ext3 filesystem from the /dev/md0 partition that was just assembled by mdadm. Of course md0, if assembled and available to the system, was not running because only one disk, out of three, was in a clean state.

I skip here the epic substory in which I wasted days in a search of a working keyboard, but I let you imagine how such adventures makes my week…

Eventually, I was able to analyze the situation in details. My first reflex ? Check that disks are not physically dead:

fdisk -l /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sdc

“Linux raid partitions” (type code “fd“) are still there. Good. I assumed here that disks where not physically damaged. Maybe I should have looked at S.M.A.R.T. datas and statistics (via smartmontools). But remember, I’m lazy (and a bit crazy).

The next step was to get informations about the RAID array itself using:

mdadm --detail /dev/md0

which output the status table below (probably inaccurate as I reconstructed it afterwards):

Number   Major   Minor   RaidDevice State
   0       0        0        0      removed
   1       0        0        1      faulty removed
   2       8       33        2      active sync   /dev/sdc1
   3       8       17        3      spare

What this table told us ?

• The array is up, but not running. One of its device (sdc1) was clean and active, but it’s not enough to get a working RAID-5.
• My first attempt to rebuild the array lead to an unexpected result: it added sda1 as a spare device (in slot #3).
• It confirm that sdb1 unexpectedly failed and is now in a bad state (“faulty removed“).

Then I stopped the array and tried to fearlessly (re)assemble it using 3 differents methods:

mdadm -S /dev/md0
mdadm -A /dev/md0
mdadm --assemble /dev/md0 --verbose /dev/sd[abc]1
mdadm --assemble --force --scan /dev/md0 --verbose

It always failed with messages like:

mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: /dev/md0 assembled from 1 drives and 1 spare - not enough to start the array.

So I examined each drive from mdadm‘s point of view:

mdadm -E /dev/sda1
mdadm -E /dev/sdb1
mdadm -E /dev/sdc1
mdadm -E /dev/sd[abc]1 | grep Event

The lastest command compare the “Event” attribute of all devices. It output something like:

Events : 0.53120
Events : 0.53108
Events : 0.53120

which indicate that sda1 and sdc1 are somewhat synced (share the same number) and sdb1 “late” (lower number).

Here I’ve got the idea of recreating the raid array without sdb1, relying only on sda1 and sdc1, by using the “magic” (hence dangerous) --assume-clean option. The latter doesn’t build, erase or initialize a new array. It just try to assemble it “as is”. Here is the command:

mdadm --create /dev/md0 --assume-clean --level=5 --verbose --raid-devices=3 /dev/sda1 missing /dev/sdc1

And it worked !

I mounted the md0 partition and cleaned it up:

fsck.ext3 -v /dev/md0
mount /dev/md0

I updated my mdadm configuration before rebooting my server:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
vi /etc/mdadm/mdadm.conf
reboot

But history repeat itself, and again, the system hang up during boot. Except this time I knew what was happening: the boot process detected the remaining sdb1 device as part of the old array (the one before the regeneration I did above) and tried to run it. Remembering my last year post, I zero-ized the superblock of sdb1:

mdadm -S /dev/md0
mdadm --zero-superblock /dev/sdb1

A server reboot proved I was right and my md0 partition was automagically mounted in altered state:

localhost:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sda1[0] sdc1[2]
      1465143808 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]

unused devices: <none>

I just had to re-add sdb1 to fill the available slot and update the mdadm configuration to get back my array in its initial state:

mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
vi /etc/mdadm/mdadm.conf

Changelog:

• Check version of Python (at least v2.4 is required)
• Rename --debug option to --verbose
• Add a --dry-run option for testing
• Remove use of deprecated pexpect methods
• Add and update some error messages

Instinctively, I assumed that my folder was of the Maildir format, and Kmail local mails too. So I tried to copy my ~/Maildir folder from the mail server to my local machine (~/.kde/share/apps/kmail/mail/). And that was the result in Kmail:

It looks good but it’s not: there is no sub-folders !

After some googling, I found what was wrong: my ~/Maildir folder is not a Maildir, but a Maildir++ folder. This kind of folder is handle by popular IMAP MTA like qmail, Dovecot and courier-imap (which was used on the mail server where my ~/Maildir come from). There is some advantages of using the “++” flavor of Maildir over the classic one, like quotas and sub-folders. Unfortunately Kmail is not able to read the Maildir++ folder structure.

To fix this, I’ve created a tiny python script to migrate a Maildir++ folder to Kmail.

How-to use it ? Simply:

1. Download it to your disk,
2. Edit it and change the MAILDIR_SOURCE and KMAILDIR_DEST variables to match your local configuration,
3. Give it execution privileges,
4. Run it !

I advise you to try it first in a safe environment (like under a temporary user account). And don’t forget to backup everything before playing with it: because this script work for me doesn’t mean that it will work for you !

The main update is about the lock file, which I implemented in the last version to keep the script to run twice (or more) in parallel. This is a nice feature to avoid overlapping processes that fight each other to use the same ressources. But in some extreme cases (reboot or power failure during backup, …), the lock file will remain and so will prevent the script to start (until you notice the problem and remove the lock file manually). This new version take care of this problem and is now able to remove the lock automatically if a timeout is reached. It also kill all remaining child processes.

Here is the detailed changelog:

• Auto-kill the script if the backup process take to much time. Timeout can be defined via a constant.
• Clean kill: track all child processes to kill them safely before removing the lock file.
• Require newer versions of python (>= v2.4), rsync (>= v2.6.7) and rdiff-backup (>= v1.1.0).
• Use --preserve-numerical-ids option when adding rdiff-backup increment.
• Keep 15 increments by default instead of 20. This value can be easily changed thanks to a defined constant.
• Remove deleted file first during mirroring and delete outdated increments before adding a new one to gain space. This strategy is safer for target disk with low remaining free space.
• Tell rsync to print human-readable values.

I’ve also added a locking mechanism to prevent the script to be run twice on the same machine. I’ve added this feature because I start my script every day thanks to cron and some backups can take more than one day.

Finally, all rsync commands will now be run first to reduce the time-window during which all external machines are reached and, as mentionned above, because rdiff-backup can take lots of time to finish its job.

Here is a direct link to the new version of the script. You can also find it in my page dedicated to various linux scripts.

First, create a local repository:

svk mirror //local https://username@svn.kde.org/home/kde

SVK will ask you to create a new repository. Tell it you want so:

Repository /home/user/.svk/local does not exist, create? (y/n) y

Then, sync the remote repository with the local one:

svk sync //local

That’s all !

To update your locale repository with the latest set of changes from the remote one, just run the previous command from time to time.

Now you can play with your local repository (/home/user/.svk/local in this exemple) as if it was a normal SVN repository !

Update: If you want to generate a vanilla SVN dump out of your SVK local mirror, as suggest by Thomas Mølhave in his “Remote Backup Of A Subversion (svn) Repository” blog post, use svnadmin:

svnadmin dump -r2:HEAD ~/.svk/local > my-repository-dump