Backups, Options, and Effort

Followup Backups Revisited 10FEB2017 Followup Display

Backups

blah blah blah. We all know, or should know, that backups are important.

Options

There are a Lot of Options out there. A lot of them don't do what I want. There are a few criteria that I desire, deltas, encrypted transfer, encrypted storage, space conservation, remote backup. There are few programs that did these, and did them in a way I liked. rsnapshot and obnam are the two I put most of my attention into. I did make an exception for rsnapshot in that it does not do encryption itself, but luks+dmcrypt deals with that.

rsnapshot

I started off playing with rsnapshot as it sounded very tasty, with its rsync use, and log like rotation of backups, and hard linking files between backups. To start with, I found pigmonkey's post about their use of rsnapshot (I've been subscribed to their blog for years, and hey! A post about what I'm trying to use!). Their script (cryptshot) worked great locally, though the first thing I did was to tweak it to gpg decrypt my luks keyfile.

gpg2 --no-permission-warning --no-tty -qd $KEYFILE | cryptsetup luksOpen --key-file=- $volume $name

^{NOTE: --no-permission-warning is there due to the use of sudo -E.}

This was all well and good, but I wanted backups to happen automatically. Since I use a laptop, hooking in the external drive would be a thing that prevents this from being automagic. So, I start tweaking the script, and make a 'backups' user on my pi, giving them NOPASSWD permissions in /etc/sudoers with exact augments provided, so that the account can't be missused. I figure out that I can stream the gpg decrypted luks key via ssh as I only want it to live on my laptop, and have the process controlled by ssh-agent and gpg-agent being cached.

gpg2 --no-permission-warning --no-tty -qd $KEYFILE | ssh $re_addr "sudo cryptsetup luksOpen --key-file=- $volume $name"

Then I actually started playing with /etc/sudoers.d as it has some benefits, and adjusting permissions. I'm going to get to this section later, as I went through a number of revisions.

At this point I made a user on the backup server, and locked them down. I'll go into specifics later.

So, this where I ran into an annoying wall. rsnapshot only works on a 'pull' model. This is fine if everything is local. And also means you can pull an external server's content. But not the other way around...

Which is exactly what I wanted to do, as my GPG key is staying on my computer and is only unlocked for a few hours. I periodically reopen the key when I refresh mutt/pianobar, which would mean if-i-am-around -> do-the-backup.

So I set my rsnapshot plans down, and look at my options again.

obnam

For the next two days I poked obnam instead. It seems nice, works on the push philosophy, does deltas, can use gpg encryption.

While trying it out I run into a few issues though. obnam uses gpg1, or at the very least /usr/bin/gpg. A future (soon) version of debian sid may change this, as I hear news of gpg2 being /usr/bin/gpg in the works. ^{(has happened now, on debian sid 17AUG2016)}

Anyway, this is an 'issue', as my low-importance key (for things like pass, local files, etc) is a gpg2 key, and gpg1 just doesn't see it.

*sigh*

Next up: It takes longer to backup than rsnapshot. I didn't do scientific testing here, but it really seemed to take 10-15 minutes to do a delta backup with obnam, where rsnapshot knocked it out in about 5-7 minutes. (context: local backups with external plugged straight into laptop, total hard drive usage <300gb out of 500gb, external drive is 1tb)

The gpg issue isn't obnam's fault, and the backup speed is just a little annoying.

The third issue was a deal breaker... obnam is tar based, and offers two ways of accessing the backups, fuse and extraction. The added layer of fuse makes unfortunately really slows down your interaction speed. If I run ncdu on my laptop, it takes ~5 minutes to the scan. If I have one backup via rsnapshot it takes ~5 minutes to scan (or if I just point ncdu at one hourly backup). If I have one backup in obnam it takes >3 hours to ncdu... ffs!

So, back to the drawing board.

Back to rsnapshot

At this point I look at rsnapshot again. I mostly put it down, as thinking about the threat model of letting the backup server have access to my laptop was going to take effort.

I decide I will have to give the backup user limited access to my laptop, if I want backups to work and be automatic. There will be a backup user on both the pi and the laptop, passwords will be disabled with passwd -l which still allows key based ssh login. The ssh key will have restricted IPs, and point at a script that checks the incoming commands (even though sudo checks too).

End Setup

Here is a summary of how I tied everything together.

Users

Both the laptop and the server have a backup user, with their password disabled

# Add the user
sudo adduser backups
# Become the user
sudo su - backups
# I'm lazy, this makes ~/.ssh with the right premissions.
ssh localhost
exit
# make a non standard key
ssh-keygen -b BIG_NUMBER_HERE
exit
# Disable the user password
sudo passwd -l backups

^{NOTE: Non standard key sizes (and primes) are a good thing!}

SSH

Installed to /usr/local/sbin/val-back-cmd.sh for permission security.

These accounts have an authorized_keys file. The keys are restricted to set IPs, and run a script that checks the incoming commands.

#~/.ssh/authorized_keys
from="10.0.0.2,10.0.0.3",command="/usr/local/sbin/val-back-cmd.sh" SSH_KEY_HERE

Paired with:

#! /bin/bash

laptop_host="moving-computer-of-doom"
remote_host="tiny-server-of-doom"

if [ $HOSTNAME = $laptop_host ]; then
    case "$SSH_ORIGINAL_COMMAND" in
        *\&*|*\|*|*\;*|*\>*|*\<*|*\!*)
        exit 1
        ;;

        ## This needs to be set in /etc/rsnapshot.conf rsync_long_args
        /usr/bin/rsync\ --server\ --sender*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        *)
        exit 1
        ;;
    esac
elif [ $HOSTNAME = $remote_host ]; then
    case "$SSH_ORIGINAL_COMMAND" in
        *\&*|*\|*|*\;*|*\>*|*\<*|*\!*)
        exit 1
        ;;

        mount*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        umount*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        cryptsetup\ luksOpen*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        cryptsetup\ luksClose*)
        sudo $SSH_ORIGINAL_COMMAND

        rmdir*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        mkdir*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        rsnapshot*)
        sudo $SSH_ORIGINAL_COMMAND
        ;;

        *)
        exit 1
        ;;
    esac
fi

It is worth noting that you do need to manually connect from laptop-to-server, and server-to-laptop at least once, so that the host key is accepted.

Example val-back-cmd.sh

Sudoers

Eariler I talked about setting up /etc/sudoers.

But thinking about it, lets set this up better, and use /etc/sudoers.d/. Now we don't have to worry about the order, and not backups will not be effected by future sudo package updates. To make a new entry, sudo visudo -f /etc/sudoers.d/backup_sudo.

Cmnd_Alias MOUNT = /bin/mount /dev/mapper/crypt-YOUR_UUID_HERE /mnt/YOUR_UUID_HERE, /bin/umount /mnt/YOUR_UUID_HERE

Cmnd_Alias LUKS = /sbin/cryptsetup luksOpen --key-file\=- /dev/disk/by-uuid/YOUR_UUID_HERE crypt-YOUR_UUID_HERE, /sbin/cryptsetup luksClose crypt-YOUR_UUID_HERE

Cmnd_Alias DIR = /bin/rmdir /mnt/YOUR_UUID_HERE, /bin/mkdir /mnt/YOUR_UUID_HERE

Cmnd_Alias RSYNC = /usr/bin/rsync

Cmnd_Alias RSNAPSHOT = /usr/bin/rsnapshot hourly, /usr/bin/rsnapshot daily, /usr/bin/rsnapshot weekly, /usr/bin/rsnapshot monthly, /usr/bin/rsnapshot yearly

Cmnd_Alias CRYPTSHOTR = /usr/local/sbin/cryptshotr -q hourly, /usr/local/sbin/cryptshotr -q daily, /usr/local/sbin/cryptshotr -q weekly, /usr/local/sbin/cryptshotr -q monthly, /usr/local/sbin/cryptshotr -q yearly

## On laptop, when remote is accessing
## Laptop needs RSYNC and CRYPTSHOTR, as 'sudo cryptshotr' will handle the decrypting/mounting/mkdir. laptop backup user needs RSYNC. 'cryptshotr-cron' needs CRYPTSHOTR.
## Remote server needs MOUNT, LUKS, DIR, RSYNC, and RSNAPSHOT.
%backups ALL=(ALL) NOPASSWD: RSYNC, CRYPTSHOTR

With the commands broken up by type, it makes it much more readable, and I don't need to give the backup server CRYPTSHOTR access, etc.

Example backup_sudoers

rsnapshot

Installed to /etc/rsnapshot.conf

TABS, you must use tabs. Really, options and augments have to be tab separated. I've warned you.

There are three big settings, as well as the interval and exclude section. The follow is just to highlight those settings, it is incomplete.

snapshot_root   /mnt/YOUR_UUID_HERE

#...

### Intervals ### {{{
## Must be unique and in ascending order
retain          hourly          24
retain          daily           7
retain          weekly          4
retain          monthly         12
retain          yearly          5
### End Intercals ### }}}

#...

rsync_long_args         -ev --rsync-path=/usr/bin/rsync

#...

## For local cryptshotr
backup  /               doom/
## For remote cryptshotr
#backup backups@moving-computer-of-doom.local:/          doom/

Example rsnapshot

cryptshotr

Installed to /usr/local/sbin/cryptshotr for permission security.

For cryptshotr, my modification of cryptshot by pigmonkey, I ended up doing a few things. I already talked about using a GPG encrypted LUKs keyfile. The rest of my changes are going to be summarized here.

☑ Uses GPG with LUKs key, so it is secure at rest.
- GPG keys are cached with gpg-agent, which I periodically refresh for mutt/pianobar.
☑ Can backup over the network.
- Checks if remote feature is enabled in settings.
- Checks if connected to home network, via MAC address.
- Checks if remote server is pingable.
- Checks if remote volume is connected.
- Will then pipe gpg decrypted LUKs key over ssh, to open volume; then mount container.
- This works via my Pi at home
☑ Added a 'quiet' mode for less crontab emails
- I expect a number of backups to fail:
  - My laptop is moble
  - My gpg-agent will time out
  - My server might not recover from the router rebooting _{(working on that)}

cryptshotr code

cryptshotr-cron

Installed to /usr/local/sbin/cryptshotr-cron for permission security.

I wrote a wrapper to deal with schduling run times, and running periodicities in order. crontab would have ran multiple periodicities in parallel if they were scheduled at the same time. This would cause all besides the first to fail, as the container would already be open. While I considered teaching the script to not care, I decided that leaving it as is would be more tamper-evident.

☑ log file
- Will log periodicities' pass/fail
- On pass, will purge previous entries for periodicity
☑ Will check periodicity's last run epoch against current time
- If time not met, skips periodicity.
- If missing runs periodicity.
- If last run failed, run.

cryptshotr-cron code

crontab

To finish it all up, I call cryptshotr-cron in crontab every half hour.

*/30 * * * * /usr/local/sbin/cryptshotr-cron

My git repo

I have set up a git repo with the files in one place, if people wish to use my work (built off of pigmonkey's).

TODO

Make script for server to detect orphaned volume mount (backup failed/hung/laptop closed), and shut the container.

_{Edited 18MAY2016: Improved val script's cryptsetup slightly, suggested by pigmonkey.}

Backups, Options, and Effort

Fri 06 May 2016 backups backups / CLI / rsnapshot / rsync / network / obnam / ncdu / gpg / luks / dmcrypt / awk

Backups

Options

rsnapshot

obnam

Back to rsnapshot

End Setup

Users

SSH

Sudoers

rsnapshot

cryptshotr

cryptshotr-cron

crontab

My git repo

TODO

Links