4 ms·
Can someone suggest an approach to backup container environment? E.g. running inside Kubernetes. As I see it: I write some kind of configuration. someproject-
by vbezhenar 4y ago
Can someone suggest an approach to backup container environment? E.g. running inside Kubernetes.
As I see it: I write some kind of configuration.
someproject-db is a deployment which runs a postgres db. Tool should connect to this DB, issue some kind of pg_backup command, capture output, retrieve some metadata about previous backup from S3, compute difference with previous run, compress that difference and store it to S3.
anotherproject is a deployment which runs an sqlite db. Tool should do the same but with sqlite-specific commands.
yetanotherproject-data is a pvc which has attached pv. Tool should find pod which mounted this volume, exec into that pod and retrieve pv data, again find different and store it to S3.
Of course things should be configurable. Like store difference every 15 minutes, store complete backup every week and so on.
I'm fine with manual recover and with manual configuration (I just don't want to write and test all the scripts myself).
What I don't want is some kind of magic tool which will backup the entire cluster, etcd and my grandparents automatically in some magic way only for $50k/cpu core.
- aborsy 4y agoDatabase and VM snapshot and backup can be tricky. My suggestion is using ZFS.
- mekster 4y agoDoesn't zfs kind of solve all the backup problems alone? Technically, no other backup tools can beat it as being the filesystem itself, it knows more than any external tools can know, like instantly know what file got changed over time without scanning the entire tree. I use Borg as backup of backup (zfs snapshots), so I'll be having multiple implementations of backups (also both are on different remote location) just to be on the safe side. I don't use any other fancier ones as I don't like risking data on less reliable tools.
- m3nu 4y agoRemote ZFS replication kinda does. But the offsite backup wouldn't be encrypted and not everyone is using ZFS. So it's not for all situations.
- mekster 4y agoYou can send zfs encrypted volume as encrypted. How does it matter if anyone else is using zfs? You either use a service that supports zfs target or run your own Linux instance which is just installing a single package for Ubuntu.
- m3nu 4y agoWhat I meant was that there are people who don't run ZFS, but still need backups. So it won't work for everyone. Even for my own use cases, not every server and system I maintain could use ZFS right away. Still good to know about the encrypted volume feature. Will be sure to test this next year.
- dpedu 4y agoI'm using Velero to do this in my toy kubernetes clusters. It uses Restic under the hood and can store things into S3. By default it will take a filesystem-level copy of whatever is on a pv. It looks like it supports hooks, e.g. to run pg_backup like you mentioned, but I haven't used them. https://github.com/vmware-tanzu/velero https://github.com/vmware-tanzu/velero
- seymon 4y agoIs there something to backup helm releases? With including all k8s manifests, configmaps, secrets and also persistent volumes. Preferably FOSS?
- onedr0p 4y agoUse Git and furthermore GitOps for your kubernetes manifests. Look into Flux2 and Argo.
- m3nu 4y agoI recently wrote up my strategy for backing up local containers with Borg & Borgmatic here: https://docs.borgbase.com/setup/borg/containers/ https://docs.borgbase.com/setup/borg/containers/ Borgmatic will beautifully deal with DB dumps and there is a popular container image to run it. As for the cache ("retrieve some metadata about previous backup from S3"), you don't need to keep it locally. It can be restored from the backup repository. Hope some of this applies to your K8s setup.