Backup your server to Memstore using Duplicity

Posted on May 28, 2012 by Juan J. Martínez (in Dev).

We are frequently asked about using  rsync with Memstore, our cloud storage solution. Although we have been working on that, at the time of writing we can't fully support rsync because Memstore is an object storage solution and rsync expects to work on a real file system.

Instead of rsync, I recommend you use  Duplicity for backups. It's a very efficient backup tool that uses the rsync algorithm and provides some interesting extras such as encryption, signing packages and Cloud Files API support (which is compatible with the Memstore API).

Let's say we want to backup our sever running Debian. We don't need to create a special user because Duplicity won't work with a container user (it needs to create the container), so we're going to use the administrator account for our backup.

Now we need to install Duplicity using the apt-get package manager:

# apt-get install duplicity

Note: Duplicity version 0.5.18 or later is required in order to work with Memstore.

If we're running Wheezy or Sid, we can install Cloud Files support with:

# apt-get install python-cloudfiles

In Squeeze we may need to install it with the pip tool:

# apt-get install -y python-pip && pip install python-cloudfiles

Once we have the required packages installed, we can create a small script to backup our files regularly with cron. We can put that file in /root/backup2memstore.sh:

#!/bin/bash

# duplicity will create that container for you
UPLOAD_TO_CONTAINER=backup

# what do we want to back up
BAKPATH=/home
# activity log
LOG=/root/backup.log

export CLOUDFILES_AUTHURL=https://auth.storage.memset.com/v1.0
export CLOUDFILES_USERNAME=mymemstore.admin
export CLOUDFILES_APIKEY=your-memstore-password
export PASSPHRASE=passphrase-for-encryption
 
duplicity --log-file=$LOG $BAKPATH cf+http://$UPLOAD_TO_CONTAINER > /dev/null
exit $?

Set the right permissions for the file with:

# chmod +x /root/backup2memstore.sh

Now you can use this script to backup your server daily setting a line in your crontab such as:

MAILTO=mymail@mydomain.dom
0 5 * * * /root/backup2memstore.sh

If there is any problem with the backup, you'll get an email at  mymail@mydomain.dom and you can check the logs in the configured log file (/root/backup.log in the example).

Using this post as an example, a quick introduction to Duplicity commands would be:

Verify a backup

# duplicity verify cf+http://backup /home
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 28 12:34:15 2012
Verify complete: 1425 files compared, 0 differences found.

Delete backups older than 15 days

# duplicity remove-older-than 15D cf+http://backup 
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 28 12:34:15 2012
No old backup sets found, nothing deleted.

Restore a backup

# duplicity restore cf+http://backup /restored-home
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 28 12:34:15 2012

I encourage you to read  Duplicity's documentation on line for further details.