For years I was backing up my server with
website-backup.py, a custom script I wrote to manage data mirroring, do incremental backups and monthly snapshots based on
bzip2. All these data were pushed to a storage server hosted at home.
First, we need to create an account on Amazon AWS. This is easy and fast. My account was activated in minutes.
Now that you have access to Amazon’s cloud, let’s create a bucket on S3. I used the reversed domain name of the server, which give me a bucket name like
com.example.server.backup. With this naming scheme, I can identify the purpose of the bucket by its label only.
Duplicity can use the cheaper RRS storage, but you need at least version 0.6.09. Having a Debian Squeeze, the only way to get a recent version is to install it from the backports:
$ apt-get -t squeeze-backports install duplicity python-boto
Then I created a simple symmetric key with GPG:
$ gpg --gen-key
You absolutely need to provide a passphrase, else Duplicity will refuse to run.
Now update the script below with the GPG key passphrase and your AWS credentials:
# Do not let this script run more than once [ `ps axu | grep -v "grep" | grep --count "duplicity"` -gt 0 ] && exit 1 # Set some environment variables required by duplicity export PASSPHRASE=XXXXXXXXXX export AWS_ACCESS_KEY_ID=XXXXXXXXXX export AWS_SECRET_ACCESS_KEY=XXXXXXXXXX # ~/.cache/duplicity/ should be excluded, as explained in http://comments.gmane.org/gmane.comp.sysutils.backup.duplicity.general/4449 PARAMS='--exclude-device-files --exclude-other-filesystems --exclude **/.cache/** --exclude **/.thumbnails/** --exclude /mnt/ --exclude /tmp/ --exclude /dev/ --exclude /sys/ --exclude /proc/ --exclude /media/ --exclude /var/run/ --volsize 10 --s3-use-rrs --asynchronous-upload -vinfo' DEST='s3+http://com.example.server.backup' # Export MySQL databases mysqldump --user=root --opt --all-databases > /home/kevin/mysql-backup.sql # Do the backup duplicity $PARAMS --full-if-older-than 1M / $DEST # Clean things up duplicity remove-older-than 1Y --force --extra-clean $PARAMS $DEST duplicity cleanup --force $PARAMS $DEST # Remove temporary environment variables unset PASSPHRASE unset AWS_ACCESS_KEY_ID unset AWS_SECRET_ACCESS_KEY
Before running duplicity, the script will dump all MySQL databases to a plain-text file. Then the first duplicity call will do the backup itself, and the second call will remove all backup older than a year.
I saved the script above in
$ chmod 755 /home/kevin/s3-backup.sh $ echo " # Backup everything to an Amazon S3 storage 0 1 * * * root /home/kevin/s3-backup.sh " > /etc/cron.d/s3-backup
I can now sleep better knowing all the work I do on my server will not be lost in case of a catastrophic event. Amazon S3 is today a no-brainer for server backups: your data will be secured and available. And for small quantity of data (like the 10 Go of my server), it’s incredibly cheap. Especially if you compare it to the cost of maintaining a storage server at home.
This solution is so good and obvious, that I don’t know why I haven’t implemented it earlier… :)