Backups -- Revisited

Backups

blah blah blah. We all know, or should know, that backups are important.

As I mentioned in my first post about backups, they are important. I could link to so many statistics about lost data here... but then I would need to vet their legitimacy. *wink*

Options

Again, as I mentioned last time:

There are a Lot of Options out there. A lot of them don't do what I want. There are a few criteria that I desire, deltas, encrypted transfer, encrypted storage, space conservation, remote backup. There are few programs that did these, and did them in a way I liked.

Last time I talked about rsnapshot and obnam. After close to a year of using my cryptshotr (a fork of cryptshot), a rsnapshot wrapper, I have learned more about what I want. I have also gotten myself a shiny NAS recently.

NAS

I got a QNAP TS-453A recently, and put four 4TB WesternDigital Red Drives in it. Sadly, one drive was dead on arrival, and I had to go through warranty.

After getting the replacement drive, I now have a RAID6 with ~7.1TB of usable space! ^{(About two terabytes were immediately filled with content)}

Having gotten this nice, big, directly networked, high network speed/disk speed beast, I decided it was time to relook at my backups.

rsnapshot

My rsnapshot based backups have worked, and were reliable, but had two 'issues'. They where slower than I would like. An incremental backup would take about 15-20 minutes. And they use an encrypted LUKS+dmcrypt, which isn't going to fly with my NAS.

Now, I'm sure that a big part of the time sink was needing to decrypt/mount the USB 3 external for every backup. ^{(And the pi3 only has so much CPU...)}

But to be honest, cryptshotr is also a bit Rube Goldberg-esk. cryptshotr diagram

Attic

After yet another round of backup option research, Attic was looking like a decent option.

works over the network
has built in encryption
does deltas
does de-duplication

However when I went to install in on debian sid, I found it was no longer in the repo. With a bit more digging, I notices it was looking a bit stale. While looking at the git issues, I noticed ThomasWaldmann referencing issues to BorgBackup, so I gave it a peek.

BorgBackup (aka borg)

What are the differences between Attic and Borg?

Borg is a fork of Attic and maintained by “The Borg collective”.

Here’s a (incomplete) list of some major changes:

more open, faster paced development (see issue #1)

lots of attic issues fixed (see issue #5)

less chunk management overhead (less memory and disk usage for chunks index)

faster remote cache resync (useful when backing up multiple machines into same repo)

compression: no, lz4, zlib or lzma compression, adjustable compression levels

repokey replaces problematic passphrase mode (you can’t change the passphrase nor the pbkdf2 iteration count in “passphrase” mode)

simple sparse file support, great for virtual machine disk files

can read special files (e.g. block devices) or from stdin, write to stdout

mkdir-based locking is more compatible than attic’s posix locking

uses fadvise to not spoil / blow up the fs cache

better error messages / exception handling

better logging, screen output, progress indication

tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X

Usage

Usage is pretty simple. To initialize a backup location borg init /path/to/repo and to make a backup borg create /path/to/repo::archive_name_x ~/one ~/two. Then you can start dropping options on top of that.

AutoFS

Initially I was pointing my backups at an AutoFS local CIFS mount, which worked quite well. *skims past* After getting everything working and automated ^{(more on that later)}, I decided to switch over to the ssh connection type so I could start backing up my VPS too.^{(Since... uh... I'm not going to be doing a CIFS mount over the Internet)}

I should also note that incremental updates are only taking one to three minutes now!!! (with ~45gb of space that borg is paying attention to).

Setting up SSH

This proved to be a big of a chore. With my QNAP I have a third party package called Entware-ng install, which provides the opkg package manager from OpenWRT (and I have used on my router). This is a handy way to get various linux packages onto the NAS, like my deluged. Unfortunately borg wasn't an offered package. The precompiled binaries also have issues.

QNAP Container

This seemed pretty easy. Install the app called 'Container Station', and wait for it to install itself and its dependency apps. Then make a container of your chosen linux distro ^{(please don't say 'ubuntu'...)}. I choose a debian lxc ^{(which I promptly upgraded to debian sid)}. During the setup process, select 'Advanced Settings>>>', set up the Port Forwarding, and the Shared Folders. ^{(I didn't do this the first time round...)}

Fixing QNAP Container Port Forwarding

So... about Port Forwarding... The UI only lets you edit it during container creation...

Bug anyone?

At this point I start using my GoogleFu, and see that a few people have posted about this over the past couple of years... with no solutions known.

Oooo the bug gets funner! After restarting my container, the Port Forward vanished!

Well, I decide it can't hurt to start digging though the files on the filesystem, and I luck out. Behold the qnap.json, found at /share/ROOT_OF_RAID/SHARE_NAME/container-station-data/lib/lxc/NAME_OF_CONTAINER. In my case: /share/CACHEDEV1_DATA/containers/container-station-data/lib/lxc/debian.

Reading the now 'bad' file, I find:

{"volume": {"new": [], "host": {}, "container": []}, "version": "8", "resource": {"device": [], "limit": {}}, "name": "bad-container", "autostart": true, "arch": "amd64", "image": "debian-jessie", "type": "lxc", "network": {"hostname": "bad-container", "port": [], "mode": "nat"}}

After making a through away container, I find a good one should look like:

{"volume": {"new": [], "host": {}, "container": []}, "version": "8", "resource": {"device": [], "limit": {}}, "name": "throw-away", "autostart": true, "arch": "amd64", "image": "debian-jessie", "type": "lxc", "network": {"hostname": "throw-away", "port": [[1111, 22, "TCP"]], "mode": "nat"}}

Where [[1111, 22, "TCP"]] is [[Exterior_port, container_port, "TCP_or_UDP"]]. While I did not test, I suspect multiple ports would look like [[1111, 22, "TCP"], [2222, 8080, "TCP"], [ETC...]]

Which is now posted on the QNAP forum as well.

End Setup

Here is a summary of how I tied everything together.

borg_wrap.sh

I wrote up a nice little script to handle my backups.

It features:

Error handling
Config items, and config checking
Tests that I am on home network
pass (passwordstore) is used to store the password
- Relies on gpg key being cached to access password

borg_wrap.sh code

Users

Users? We don't need this section! Well, aside from a normal account on the destination with access to the backup destination.

SSH

ssh-ed25519 XXX... laptop_key
command="borg serve --restrict-to-path /mnt/borg/laptop",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 XXX...
command="borg serve --restrict-to-path /mnt/borg/server",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 XXX...

Notable parts:

The script calls a passphraseless key, so that it doesn't need to source ssh-agent.
- Trying to source ssh-agent with crontab is possible, but sudo just makes that a painful thought.
The borg serve --restirct-to-path both limits borg serve's access, and prevents non-matching commands from working -- thus preventing unauthorized access.
Due to QNAP's... Interesting Port Forwarding set up, I can't limit the keys with from=, as it isn't seen correctly by the container.

Sudoers.d

Here is the content of my /etc/sudoers.d/borg_sudoers:

demure ALL=(ALL) NOPASSWD: /usr/local/sbin/borg_wrap.sh -q, /usr/local/sbin/borg_wrap.sh

Compared to my old cryptshotr it is very simple and concise.

Notable parts:

borg_wrap.sh is moved to /usr/local/sbin to prevent edits for user escalation.
/etc/sudoers.d is used to prevent issues with bad edits breaking sudo completely, and on some OS's to survive sudo upgrades.
NOPASSWD is used so that it can run automated.

crontab

I have borg_wrap.sh called every hour, and unlike my old cryptshotr I don't bother with fancy time checks as I don't need to balance/track multiple periodicities.

## borgbackup wrapper
0 */1 * * * sudo /usr/local/sbin/borg_wrap.sh -q

Note: Aside from how often you run backups, periodicities are partially a function of how you prune.

Pruning

While I have not accumulated enough backups to use this yet, this is close to what I will be using:

borg prune -v --list --dry-run -d=7 -w=4 -m=12 -y=-1 --keep-within=1d /path/to/repo

Which will keep:

all archives <= 24hours old
one archive per day <= 7days old
one archive per week <= 4weeks old
one archive per month <= 12months old
unlimited yearly archives

conky

As a finishing touch, I updated my back_log_backups.conkyrc to use my borg_wrap.sh log, stored at $HOME/.config/borg/log. This will show the last six entries, and if none six are a 'passed' it will show a seventh line at the top, showing the last 'passed' entry. borg conky

back_log_backups.conkyr code

borg_wrap.sh Diagram

So, as you can see, my setup is much more simplified now: borg_wrap.sh diagram

^{2017JUL09: Updated picture as I noticed an error.}

Afterword

Having done all the leg work for my laptop, it was easy to add another repo/authorized_key for my VPS and just tweak the script (and disable the wifi MAC address lookup). So now I have both my laptop and my VPS backed up on my NAS.

Backups -- Revisited

Thu 09 February 2017 backups backups / CLI / borg / borgbackup / rsnapshot / sudoers.d / network / ncdu / gpg / awk / conky