Laying the Foundation
With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.
When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.
I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.
In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.
The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.
Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.
The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.
For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.
Based on the type of access, I use two types of encryption:
... continue reading