Encrypted Remote Backups

I've been toying around with my offsite backup system for Quite Some Time Now (probably more than 5 years). I've settled into something that really does seem to work. My old method had some problems. In particular, backups kept taking longer and longer (presumably because of block fragmentation in the loop-aes FS) and deleting lots of stuff increased backup time. Ick. It also had the annoying side effect of the backups being too large for using rsync.net to be viable; I'm using them with my new method.

Much of the text and layout in this doc is stolen from my old method.

Introduction

Introduction

I have a method of personal data backup that seems to be fairly novel, in that it has all of the following properties:

• I back up multiple OSes on multiple machines, with incremental backups and a CGI interface and all sorts of good stuff like that
• I synchronize it offsite (daily)
• The offsite backup is encrypted, and only I have the key
• I don't have to synch everything every time I backup
• It deals with OS special files, like symlinks
• It's reasonably fast; I'm at about 7GiB of data, and the daily synch takes about an hour.
• It's free (except for the offsite account)

None of these things are especially exceptional, but the combination of all of them is rare enough that in searching for a good solution I've found a number of people also looking for this feature set (and largely failing to find it, by the way).

My method is based around BackupPC as the main backup system. The problem with BackupPC, is that it likes hard links. A lot. No non-UNIX file system understands these and, in particular, it means Amazon's S3 is right out. I'm not aware of any Linux multi-OS backup system that doesn't suck that also doesn't have this problem; that is, such systems make few assumptions about the file system they're backing up, but a lot of assumptions about the file system they're backing up to. The big advantage to BackupPC's hard link system is that it minimizes the size of each new backup, whilst still retaining the enterprise-level tiered backup system we've all come to know and, umm, tolerate.

The Machines

• Various and sundry home machines (a Windows XP box, a couple of Linux boxes).
• The BackupPC server; we'll call it "LOCAL".
• The remote server, which we'll call "REMOTE". I recommend rsync.net for this purpose.

The Software

• rsync, and a REMOTE that fully supports the rsync algorithm when operating at the block-level. You really want rsync 3.0 or better here if possible; it speeds the transfer of large file trees immensely.
• rsyncrypto, which performs rsync-friendly file encryption. It's a bit rough around the edges, but our method is such that we avoid the encrypted file name stuff, which is the stuff that (as of March 2008 anyways) still needs some work
• BackupPC or a similar multi-OS backup system that runs on Linux as the LOCAL server

The Process

Note that this document doesn't discuss setting up BackupPC at all.

Set Up rsyncrypto

You can just follow the man page tutorial to set up a key. In addition, I gpg symmetrically encrypt both the public and private key and send them with the backups. I use a very long key for this (over a hundred characters). You can do that with just "gpg -c [file]".

The password to the key is used in my scripts, but if someone compromises my machine to the extent required to see it, they won't need to compromise my backups, since they can just copy all the live data.

Extra Bits You Should Transfer

If you want to be able to decrypt fully without too much trouble, you should probably copy the following as part of your backups. Encrypt as appropriate.

• BackupPC_zcat (for uncompressing)
• FileZIO.pm (BackupPC_zcat needs it)
• a restore script (make sure it doesn't have any passwords in it!)
• the backuppc configs (encrypted!); without them you can't use backuppc for restoration and will have to do it by hand
• the rsyncrypto keys (as I said, I gpg encrypt them; otherwise you need to keep secure copies elsewhere)

Set Up Data For Encryption

I have a few bits of miscellaneous data and so on that I won't bother describing here, but some setup definately needs to be done before we try to encrypt.

I actually use two forms of encryption: gpg symmetric as described above and rsyncrypto. This is largely for historical reasons.

1. Stop backuppc
2. Create a tar of the backuppc "pc" directory; this is the directory with all the hardlinks in it. BackupPC now has a utility to generate a tar of this directory without too much trouble. Between this and the cpool directory, you'll have everything you need. BackupPC_tarPCCopy is the utility

The Real Work

1. Encrypt the BackupPC_tarPCCopy output with rsyncrypto
2. Encrypt the cpool directory with rsyncrypto
3. rsync it all to your remote site

The Code

Here's my backup script (with passwords cropped and such):

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

# After 12 hours, assume the other rsync is dead or something.
lockfile -l 43200 -r 1 /var/tmp/lock.backup_rc_full

if [ $? -ne 0 ]
then
        echo "Could not obtain lock; exiting."
        exit
fi

log() {
    echo "
=======================================================================
$(date): $1
=======================================================================
"
}

rm -f /tmp/backup_rc_full.log

exec 1>/tmp/backup_rc_full.log
exec 2>&1

log "Stopping BackupPC."
# Stop backuppc
/etc/init.d/backuppc stop

log "BackupPC stopped."

log "Doing basic setup."

# Basic setup
chmod og-rwx /backups/rcb/
chown root /backups/rcb/
cp /usr/share/backuppc/bin/BackupPC_zcat /backups/rcb/BackupPC_zcat
mkdir -p /backups/rcb/BackupPC
cp /usr/share/backuppc/lib/BackupPC/FileZIO.pm /backups/rcb/BackupPC
cp /home/zroot/bin/restore_rc_full /backups/rcb/

log "Saving backuppc configs."

# Put the backuppc configs in there.
rm /backups/rcb/etc_backuppc.tgz.gpg
tar -zc /etc/backuppc/ -f /backups/rcb/etc_backuppc.tgz
echo **ERASED*PASSWORD** | gpg -q --batch --passphrase-fd 0 --symmetric /backups/rcb/etc_backuppc.tgz
rm /backups/rcb/etc_backuppc.tgz

log "Saving keys."

# Put the backup keys in there
cp /home/zroot/backup.key.gpg /backups/rcb/
cp /home/zroot/backup.crt.gpg /backups/rcb/

chmod og-rwx /backups/rcb/*
chown root /backups/rcb/*

CERT=/backups/rcb_plain/backup.crt

# Make unencrypted crt for rsyncrypto to work with
echo **ERASED*PASSWORD** | \
    gpg --batch -q -d --passphrase-fd 0 /home/zroot/backup.crt.gpg >$CERT

log "Creating tar of the pc dir"
cd /backups/rcb_plain
sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarPCCopy /var/lib/backuppc/pc >pc.tar

log "Running rsyncrypto against the pc tarball"
/usr/local/bin/rsyncrypto -c -vv /backups/rcb_plain/pc.tar /backups/rcb/backuppc/pc.tar /backups/rcb_plain/pc.tar.key $CERT

log "Done with rsyncrypto"

log "Running rsyncrypto against the pool and such"
find /var/lib/backuppc/* -type d -prune | egrep -v '/(\.ssh|\.gnupg|ball|chain|pc|trash)$' | sed 's;$;/;' | /usr/local/bin/rsyncrypto --trim=2 --delete -c -vv --filelist - /backups/rcb/ /backups/rcb_plain/keys/ $CERT

log "Done with rsyncrypto"

# Fix ownership
chown root /backups/rcb/* /backups/rcb/*/*

log "Starting BackupPC."

# Start backuppc
/etc/init.d/backuppc start

log "BackupPC started."

log "Syncing to REMOTE"

# Note that everything is already compressed
/usr/bin/rsync --no-compress --delete -S --stats -av /backups/rcb/ REMOTE:/backups/rcb/

log "Done syncing to REMOTE"

rm $CERT

# Remove the lockfile
rm -f /var/tmp/lock.backup_rc_full

Here's the restore script:

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

log() {
    echo "
=======================================================================
$(date): $1
=======================================================================
"
}

log "Doing basic setup."

mkdir -p /backups/rcb_restore/backuppc

CERT=/backups/rcb_restore/backup.key

# Make unencrypted crt for rsyncrypto to work with; asks for password
gpg -d /backups/rcb/backup.key.gpg >$CERT

log "Running rsyncrypto decrypt"
/usr/local/bin/rsyncrypto --trim=2 --delete -d -vv -r /backups/rcb/backuppc/ /backups/rcb_restore/ /backups/rcb_restore/keys/ $CERT

log "Done with rsyncrypto"

log "Expanding tar of the pc dir"
# the P is important; see the backuppc man page
mkdir -p /backups/rcb_restore/backuppc/pc
cd /backups/rcb_restore/backuppc/pc
tar -xvPf ../pc.tar

log "Done with rsyncrypto"

# Fix ownership
chown root /backups/rcb_restore/* /backups/rcb_restore/*/*

rm $CERT

---