4 ms·

I used to have a "poor man's time machine" system based on rsync + hard links to files that didn't change with new backups. Essentially it was the same concept

I used to have a "poor man's time machine" system based on rsync + hard links to files that didn't change with new backups. Essentially it was the same concept than time machine. Of course you couldn't upload a single "snapshot" because tar wouldn't know what's a hard link. One advantage of using rsync is that you can also keep track of things you delete. Today I'm using zfs with real snapshots. For systems with no zfs support (my wife's iMac for instance), I have a zfs fs that those systems rsync to, after the rsync is done I create a snapshot. All scripted. The snapshots can be stored in another server for an additional layer of backup, or incrementally send them to s3 if you want.

svara 9y ago
Cool setup. But if you're incrementally sending snapshots to a non-ZFS host (S3), your S3 storage will keep growing indefinitely, because you can't delete any snapshots from there ever, since they reference each other. Correct?
- reificator 9y ago
  Seems like you could probably use a keyframe like approach to get a nice middle ground. Every month, or biweekly or so, take a full backup and base the snapshots on that.
azag0 9y ago
I was surprised how easy it is to setup a time-machine-like backup with rsync. I'm backing up every hour to my server, and a cron job prunes the backups every night. Disarmingly simple.
Rotareti 9y ago
> I used to have a "poor man's time machine" system based on rsync + hard links to files that didn't change with new backups. I build something similar, that runs on a Raspberry Pi and creates backups for the machines in my home network [0]. The Pi pings each machine every hour, if a machine is online and a backup is due, it starts a backup process. My Pi uses a USB battery as a UPS (unlimited power supply) [1]. I put all the hardware in a little medicine cabinet on the wall [1]. It's been running stable for month now, without a single reboot. It needs about 15 minutes to backup my dev machine over WiFi. It's a little independent backup module, I'm really happy with it. :) [0] https://github.com/feluxe/very-hungry-pi https://github.com/feluxe/very-hungry-pi [1] https://www.amazon.de/gp/product/B00FAU7ZB2/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 https://www.amazon.de/gp/product/B00FAU7ZB2/ref=oh_aui_searc... [2] https://www.amazon.de/HMF-Medizinschrank-Arzneischrank-Hausapotheke-Original/dp/B002CVKEWY https://www.amazon.de/HMF-Medizinschrank-Arzneischrank-Hausa...
- pmalynin 9y ago
  Surely you mean Uninterruptible Power Supply. I would indeed be interested in an unlimited one myself.
  - Rotareti 9y ago
    This makes sense now that you say it. Whatever it is, it powers my Pi very well :)
- anotherevan 9y ago
  I similarly have a Pi in a cupboard doing the backups. In my case though, it uses wake-on-lan to wake the computers up in the middle of the night to run the backups, then lets them go back to sleep after. I use rsnapshot to a encrypted external usb disk.
- icebraining 9y ago
  I used to have a Pi for that, but the poor IO was annoying when you needed to backup large files. I got a small Atom board instead, which runs circles around it. And since it's a regular x86 board, I can use rtcwake to make it sleep between each backup, so the power consumption is quite good.
extra88 9y ago
I still use rsnapshot to manage such backups. I like that I can use an old Mac to run it. Doesn't ZFS require more from the system, RAM particularly?
derekp7 9y ago
I wrote a tool called Snebu (www.snebu.com), which is intended to give you the same thing as rsync + hard links. Uses find and tar to grab file metadata and retrieve files, on the back end it stores file data in .lzo compressed files with all metadata for each snapshot (including symlinks, hardlinks, etc) stored in a sqlite3 DB file. The design goals were to make something that didn't store data in a proprietary format (you can analyse the backups using straight sql commands, and access the data using lzop), and to be able to back up systems without installing a client agent on them, and support compression, and avoid filesystem issues you run into with a large number of hard links (such as https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283). So far I've been using it to back up a few dozen RHEL 4 - RHEL 7 systems over the last couple years without issues.

svara 9y ago

Cool setup. But if you're incrementally sending snapshots to a non-ZFS host (S3), your S3 storage will keep growing indefinitely, because you can't delete any snapshots from there ever, since they reference each other. Correct?

reificator 9y ago
Seems like you could probably use a keyframe like approach to get a nice middle ground. Every month, or biweekly or so, take a full backup and base the snapshots on that.

azag0 9y ago

I was surprised how easy it is to setup a time-machine-like backup with rsync. I'm backing up every hour to my server, and a cron job prunes the backups every night. Disarmingly simple.

Rotareti 9y ago

> I used to have a "poor man's time machine" system based on rsync + hard links to files that didn't change with new backups. I build something similar, that runs on a Raspberry Pi and creates backups for the machines in my home network [0]. The Pi pings each machine every hour, if a machine is online and a backup is due, it starts a backup process. My Pi uses a USB battery as a UPS (unlimited power supply) [1]. I put all the hardware in a little medicine cabinet on the wall [1]. It's been running stable for month now, without a single reboot. It needs about 15 minutes to backup my dev machine over WiFi. It's a little independent backup module, I'm really happy with it. :) [0] https://github.com/feluxe/very-hungry-pi https://github.com/feluxe/very-hungry-pi [1] https://www.amazon.de/gp/product/B00FAU7ZB2/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 https://www.amazon.de/gp/product/B00FAU7ZB2/ref=oh_aui_searc... [2] https://www.amazon.de/HMF-Medizinschrank-Arzneischrank-Hausapotheke-Original/dp/B002CVKEWY https://www.amazon.de/HMF-Medizinschrank-Arzneischrank-Hausa...

pmalynin 9y ago
Surely you mean Uninterruptible Power Supply. I would indeed be interested in an unlimited one myself.
- Rotareti 9y ago
  This makes sense now that you say it. Whatever it is, it powers my Pi very well :)
anotherevan 9y ago
I similarly have a Pi in a cupboard doing the backups. In my case though, it uses wake-on-lan to wake the computers up in the middle of the night to run the backups, then lets them go back to sleep after. I use rsnapshot to a encrypted external usb disk.
icebraining 9y ago
I used to have a Pi for that, but the poor IO was annoying when you needed to backup large files. I got a small Atom board instead, which runs circles around it. And since it's a regular x86 board, I can use rtcwake to make it sleep between each backup, so the power consumption is quite good.

extra88 9y ago

I still use rsnapshot to manage such backups. I like that I can use an old Mac to run it. Doesn't ZFS require more from the system, RAM particularly?

derekp7 9y ago

I wrote a tool called Snebu (www.snebu.com), which is intended to give you the same thing as rsync + hard links. Uses find and tar to grab file metadata and retrieve files, on the back end it stores file data in .lzo compressed files with all metadata for each snapshot (including symlinks, hardlinks, etc) stored in a sqlite3 DB file. The design goals were to make something that didn't store data in a proprietary format (you can analyse the backups using straight sql commands, and access the data using lzop), and to be able to back up systems without installing a client agent on them, and support compression, and avoid filesystem issues you run into with a large number of hard links (such as https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283). So far I've been using it to back up a few dozen RHEL 4 - RHEL 7 systems over the last couple years without issues.