4 ms·
Ask HN: Versioning Backup
Hello HN!
I currently have a backup/archival setup which works like this:
Local Machine --> Syncthing --> cluster of boxes regionally separated --> zfs shares --> restic --> Amazon S3 (soon to be replaced)
As it stands, I have restic running on two servers, however they operate in a sort of primary/secondary mode (of my own design), where restic will only run on the secondary server if it cannot reach the primary server on a designated port and receive a correct message.
So far, this set up has worked well, I have hourly/daily/weekly snapshots on zfs, and restic runs daily archiving all the information.
I've been pretty happy with this setup, it's mostly been designed to protect against loss of data on my main machine (via PEBAK or ransomware type issues).
One noticeable gap I've found is the gap of versioning backup. Similar to something like Dropbox where it tracks the versions of files (up to a certain number), it would be nice if the first layer (i.e. the Syncthing replacement) could handle file versions. Obviously, this wouldn't be useful for items for which versioning isn't required (e.g. video/audio files) or where versioning is implicit (e.g. source code repos), but I found I like having versioning for things like my documents which do not fall into that category. The ideal state would be a program which could be configured for multiple profiles, so I could have video/audio backed up in one way, source code/artifacts backed up another, and finally documents backed up yet another way. I would then be able to view the revisions and restore.
I'm not looking for it to understand the structure of file and the changes (a la Office365 document changes) but at least show me the change times and the file size etc, and I could restore accordingly.
Some solutions I've tried:
- https://bvckup2.com/ (windows-only) -- this one came really close in that it had almost all the features, however it lacked a robust versioning feature. As it stands, it supports a file based feature which moves the file to an "archive" location. There isn't an in-app way to do this, although reviewing the archive folder is helpful.
- git-annex -- this one looked like it would work, but as I understand it, it does not actually store the differences of file versions. If you manually create tags, it will store those, but it won't by default (or atleast I couldn't find it in the git-annex-assistant stuff).
- Syncthing -- while there is a versioning capability as part of Syncthing, it is for inbound changes only, it won't version an outbound change.
- VSC/Time Machine -- these are close, but ultimately seem to be similar versions of creating snapshot at time intervals.
I'll be honest, I haven't spent too much time with git annex, it feels like it could work with a special remote of bup, but I just couldn't figure it out and I also wanted to avoid using bup until it was a bit more stable.
I also tried using NILFS2 (https://www.kernel.org/doc/html/latest/filesystems/nilfs2.html) on one of my boxes to see if I could use that in a similar fashion. While it worked it a fairly interesting manner, I didn't see a lot of broad support and didn't want to go down that route.
Does anyone have an ideas?
HN threads I've read:
- https://news.ycombinator.com/item?id=18692844
- https://news.ycombinator.com/item?id=28758415
- https://news.ycombinator.com/item?id=25758675
- https://news.ycombinator.com/item?id=18689269
- https://news.ycombinator.com/item?id=21211859
- https://news.ycombinator.com/item?id=22855755
- https://news.ycombinator.com/item?id=23281880
- https://news.ycombinator.com/item?id=29863822
- https://news.ycombinator.com/item?id=22253981
- https://news.ycombinator.com/item?id=30481935
- bradknowles 5y agoFirst off, I think you want to be running a filesystem that supports versioning and snapshots. If those features aren't available in the source operating system, then I think nothing else really matters.
- backup_help 5y agoI'd happy to consider something like that if it would be available! Do you have any suggestions?
- bradknowles 5y agoIf you want a filesystem that can support snapshots, my preference would be ZFS. I'm not sure if ZFS can also do versioning outside of snapshots. The recent macOS file systems can do versioning, and Time Machine is how you would access earlier versions. But they can't do snapshots, so far as I know. But snapshots are less critical for building a backup system -- you can live without those, although it can be painful. Versioning would be a bigger win for desktop users.
- backup_help 5y agoThanks for following up! I have snapshots (hourly/daily/weekly) on my ZFS shares, but that doesn't fully replace the versioning that I was looking for. NILFS (https://nilfs.sourceforge.io/en/index.html https://nilfs.sourceforge.io/en/index.html) is something that was a lot closer to what I wanted (and is accepted into the kernel) but it wasn't as feature complete as I wanted it to be. I'll keep an eye out for products in this space!
- apankrat 5y agoRe: bvckup2 - the UI support for archiving is a part of the next release, looks like this - https://bvckup2.com/wip/r82-backup-settings-archiving.png https://bvckup2.com/wip/r82-backup-settings-archiving.png - and it's available as a part of preview builds - https://bvckup2.com/support/preview https://bvckup2.com/support/preview (I'm the author)