Duplicacy with Wasabi Nov 22, 2017
No, this is not a recipe for a meal…
I’d like to describe the backup structure now in place here at JeeLabs, which has been running for a few months. It’s working out nicely for a personal + family setup.
There are several moving parts involved:
- a central Core i3 server, running 24⁄7 behind a FritzBox 7490 router
- Caddy is the public HTTPS server w/ automatic Let’s Encrypt certificates
- Nextcloud is used as DropBox-like sync server, as well as Calendar + Contacts server - i.e. WebDAV, CalDAV, and CardDAV
- Btrfs (“butter-fs”) adds RAID1-like redundancy on two HDs via USB3, with support for CoW snapshots
- Duplicacy provides periodic off-site backup and cleanup
- Wasabi is the cloud storage provider I’ve been using lately
- and a few more tools, such as Hugo, Gogs, PiHole, and good ol’ Redmine
So that also explains the title of this post.
There are many trade-offs & considerations involved. First of all, everything is now combined into a single server machine, and hence a single point of failure:
- I’m not a fan of “the cloud” (to put it delicately), hence this private setup for file sync, calendars, and contacts
- with an 80 Mb/s FTTH internet setup, local servers are plenty fast
- the server is std h/w, easily replaced
- it serves mostly (90%?) static pages
- it’s running stock Ubuntu 16.04 LTS
- there’s NO private data in plaintext
If either the ISP link or server break down:
- the JeeLabs servers will be off-line
- we can still read and send email from our ISP via cellular, i.e. GSM
- all Nextcloud data is cached locally: file syncs, calendar, and contacts
- setting up a new server somewhere won’t take more than a few hours
There are exactly four services out there which this all depends upon:
- the local Dutch XS4ALL ISP (#1 in customer service, year after year)
- 1Password, by AgileBits in Canada, to guard my (encrypted) passwords
- GitHub for all code and issues, with Gogs tracking local copies of the code
- Wasabi for low-cost encrypted data storage (≈ $4 per TB per month)
Note that all data storage resides on at least 3 disks: 2 local HDs, managed with Btrfs, plus the Wasabi backup, which is in fact a historical archive to allow retrieving old data (Nextcloud also keeps old versions of the last 30 days for its file sync areas).
Every once in a while, several months apart, I also make some backups on BluRay disks, just to have stuff on physical media I can hold in my hands. Just in case…
The beauty of Nextcloud (and Dropbox), is that you can have fast local folder copies without ever worrying about the copying and sync side of it. But there’s much more to it than that: it makes for an absolutely effortless way to manage file exchanges between your own devices as well as with “shared links”, i.e. URLs, handed out to others. Nextcloud is well integrated across macOS, iOS, and Linux (and yes: also Windows and Android).
Another very useful feature is that you can turn on or off the syncing per (sub-) folder. This makes it easy to turn the Nextcloud server into a large-scale / long-term archive and permanent repository, especially when not all of it fits on your local machine.
Duplicacy has some interesting properties in this context. First of all, it can back up to a wide range of targets:
Furthermore, it deduplicates all the data it backs up. So you can move files around and rename them at will, without leading to duplicates getting stored on the backup.
Of course the crucial feature is that the data in the backup gets encrypted before being sent to a remote system. With Duplicacy, the encryption keys never leave the system on which you perform a backup or restore.
Best of all, the core engine of Duplicacy is open source (it’s on GitHub) and written in Go (effortless command-line tool deploys).
The Windows & MacOS GUI versions (i.e. wrappers) are available for a small yearly fee - making it more probable that it won’t end up as abandonware any day soon.
The name is a bit confusing, since there’s also Duplicity, the default backup tool used in Debian-/Ubuntu-based desktop installs.
All in all, this setup is definitely not for everyone. You have to go through a lot of little details to get all the pieces right, and to be honest, I’m still tweaking it at times.
But this new setup has been delightfully effective. Nextcloud delivers all the key benefits of cloud services with a private server, inside as well as outside the home. Backups are fully managed, with proper provisions for redundancy and archival (i.e. “out-of-sight”) storage for everything that tends to end up in digital form these days.
The proof is perhaps that I’m starting to move more files from my
dev machine(s) to
~/nextcloud/ - i.e. the auto-synced folder.
And because of that, a future laptop is very likely to need a much smaller