5 ms·

You should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, chan

by rsync 7y ago

You should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... your cloud provider has no access to the data. The borg website is here: https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/ and a good description of how it works and why you should use it is here: https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/

metafunctor 7y ago
I already use Duplicacy for backing up family personal macs to the cloud (in addition to local Time Machine backups). For linux servers, borg is great, and me and my companies are happy users.
cmcd 7y ago
Are there any advantages of using borg over rclone?
- rovr138 7y ago
  Yes. Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy. This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted. Borg is a backup tool. Versioning is at its core. It does that efficiently by deduplicating file (chunks really) even if they’re not in the same location. So with Borg, if you create a backup 1 of `~/Documents` today and a backup 2 tomorrow of `~/Documents` you can see both backups and work with each snapshot. The size it takes should be close to the amount of data changed in the whole source. If you move directories or files inside, rclone has to reupload them. Borg detects but doesn’t have to store it again. With rclone some remotes have versioning (Google Drive, Dropbox). This could help in this case, but it depends on the remote. With Borg this is built in and you can change the underlying storage and migrate without loosing any data. Using versioning with crypt would probably be a pain too due to the file names. Not sure if rclone has commands/flags to help with this that I simply don’t know about.
  - rsync 7y ago
    "Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy." ... "This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted." This is correct. It is widely advised to not consider a "sync" like this a proper backup. However, for what it's worth, rsync.net does support rclone[1] and because of the ZFS snapshots that are created and maintained[2] in your account, you can just do a dumb sync because the retention is handled by the snapshots. I am not sure if rclone is really the right tool for plain old cloud backups - I think rclone distinguishes itself for the ability to transfer data between cloud providers.[3] [1] https://rsync.net/products/rclone.html https://rsync.net/products/rclone.html [2] ZFS snapshots reside in the .zfs/snapshot directory inside your rsync.net account. [3] ssh user@rsync.net rclone file/in/rsync/net/account s3:/some/bucket
6ak74rfy 7y ago
After looking at a few alternatives (Borg, Duplicacy etc.), I setup Arq on my Mac yesterday. One thing that irks me about these solutions is that they seem to scan my folders each time they want to backup. Are there tools that are smarter about this? For e.g., while running, they could keep a log of what's changing and only scan those while backing up.
- novok 7y ago
  You would probably need the kernel / filesystem to keep track of it to keep it efficient. And even then that is iffy if you implement it as a kernel extension vs. something integrated into filesystem code. I've found arq backup overall slow once you start hitting 0.5TB overall. It's a design issue.
  - kalleboo 7y ago
    > You would probably need the kernel / filesystem to keep track of it to keep it efficient I believe most operating systems have a mechanism for this. E.g. on macOS it's called fsevents, Linux has inotify
    - 72deluxe 7y ago
      Yes, these utilities would probably need something in them such that fswatch implements (https://emcrisostomo.github.io/fswatch/ https://emcrisostomo.github.io/fswatch/) - very useful.
- tzs 7y ago
  > After looking at a few alternatives (Borg, Duplicacy etc.) I may have looked at Duplicacy. I'm sure that I looked at one of Duplicacy [1], Duplicati [2], and Duplicity [3]. Whichever one that was, I kept getting it mixed up with the other two when looking for information online, and finally said "screw this" and bought Arq. [1] https://duplicacy.com/ https://duplicacy.com/ [2] https://www.duplicati.com/ https://www.duplicati.com/ [3] http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
- Abundnce10 7y ago
movedx 7y ago
How does this help with the context we're dealing with here: the iPhone (and other iDevices) being heavily encrypted/secure and iCloud not being secure? Quickly looking at Borg's website (thanks for the heads up - great tool/option) I see it doesn't support iOS or backing up an iOS device. I assume you're just suggesting it as a general purpose option for general backups on the desktop?
- rovr138 7y ago
  Maybe backup to desktop and then Borg. I wonder how efficient Borg would be with this setup.

by rsync 7y ago

metafunctor 7y ago

I already use Duplicacy for backing up family personal macs to the cloud (in addition to local Time Machine backups). For linux servers, borg is great, and me and my companies are happy users.

cmcd 7y ago

Are there any advantages of using borg over rclone?

rovr138 7y ago
Yes. Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy. This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted. Borg is a backup tool. Versioning is at its core. It does that efficiently by deduplicating file (chunks really) even if they’re not in the same location. So with Borg, if you create a backup 1 of `~/Documents` today and a backup 2 tomorrow of `~/Documents` you can see both backups and work with each snapshot. The size it takes should be close to the amount of data changed in the whole source. If you move directories or files inside, rclone has to reupload them. Borg detects but doesn’t have to store it again. With rclone some remotes have versioning (Google Drive, Dropbox). This could help in this case, but it depends on the remote. With Borg this is built in and you can change the underlying storage and migrate without loosing any data. Using versioning with crypt would probably be a pain too due to the file names. Not sure if rclone has commands/flags to help with this that I simply don’t know about.
- rsync 7y ago
  "Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy." ... "This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted." This is correct. It is widely advised to not consider a "sync" like this a proper backup. However, for what it's worth, rsync.net does support rclone[1] and because of the ZFS snapshots that are created and maintained[2] in your account, you can just do a dumb sync because the retention is handled by the snapshots. I am not sure if rclone is really the right tool for plain old cloud backups - I think rclone distinguishes itself for the ability to transfer data between cloud providers.[3] [1] https://rsync.net/products/rclone.html https://rsync.net/products/rclone.html [2] ZFS snapshots reside in the .zfs/snapshot directory inside your rsync.net account. [3] ssh user@rsync.net rclone file/in/rsync/net/account s3:/some/bucket

6ak74rfy 7y ago

After looking at a few alternatives (Borg, Duplicacy etc.), I setup Arq on my Mac yesterday. One thing that irks me about these solutions is that they seem to scan my folders each time they want to backup. Are there tools that are smarter about this? For e.g., while running, they could keep a log of what's changing and only scan those while backing up.

novok 7y ago
You would probably need the kernel / filesystem to keep track of it to keep it efficient. And even then that is iffy if you implement it as a kernel extension vs. something integrated into filesystem code. I've found arq backup overall slow once you start hitting 0.5TB overall. It's a design issue.
- kalleboo 7y ago
  > You would probably need the kernel / filesystem to keep track of it to keep it efficient I believe most operating systems have a mechanism for this. E.g. on macOS it's called fsevents, Linux has inotify
  - 72deluxe 7y ago
    Yes, these utilities would probably need something in them such that fswatch implements (https://emcrisostomo.github.io/fswatch/ https://emcrisostomo.github.io/fswatch/) - very useful.
tzs 7y ago
> After looking at a few alternatives (Borg, Duplicacy etc.) I may have looked at Duplicacy. I'm sure that I looked at one of Duplicacy [1], Duplicati [2], and Duplicity [3]. Whichever one that was, I kept getting it mixed up with the other two when looking for information online, and finally said "screw this" and bought Arq. [1] https://duplicacy.com/ https://duplicacy.com/ [2] https://www.duplicati.com/ https://www.duplicati.com/ [3] http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
Abundnce10 7y ago

movedx 7y ago

How does this help with the context we're dealing with here: the iPhone (and other iDevices) being heavily encrypted/secure and iCloud not being secure? Quickly looking at Borg's website (thanks for the heads up - great tool/option) I see it doesn't support iOS or backing up an iOS device. I assume you're just suggesting it as a general purpose option for general backups on the desktop?

rovr138 7y ago
Maybe backup to desktop and then Borg. I wonder how efficient Borg would be with this setup.