14 ms·

Backing up data like the adult I supposedly am

207 points by miked85 6y ago

rubatuga 6y ago
If you want to find out how to backup your iPhone on Linux, I've also made a guide! It's actually kind of complicated, but it can be fully automated. I connect my iPhone to my Linux server in my room to do an incremental backup every night at 5 AM (it also fast charges at 1.8 amps over USB C). I then create ZFS snapshots every week, since the iPhone backup is an overwrite type. https://www.naut.ca/blog/2020/03/20/self-hosting-series-part-4-backup/#compilingidevicebackup2 https://www.naut.ca/blog/2020/03/20/self-hosting-series-part...
- walterbell 6y ago
  Thanks for your self-hosting tutorials for iOS services! The next question is how to extract individual files from a backup, without needed an iDevice for a full restore. There are several commercial products sold for this purpose, but I've not yet seen OSS tools to parse iOS backups.
graton 6y ago
Like the author I have also been very happy with Borg backup software ( https://www.borgbackup.org/ https://www.borgbackup.org/ ). The compression and de-duplication is very useful. A little bit of a learning curve to get everything up and running, but not too bad.
- aborsy 6y ago
  I also like Borg back up. I wonder if I am missing something compared to restic?
  - more-coffee 6y ago
    Borg is supposedly a bit faster, restic supports a ton of backends https://github.com/restic/restic/issues/1875 https://github.com/restic/restic/issues/1875 Also restic does not allow to backup unencrypted. I haven't used borg. Only done some maintenance on restic backup jobs at work. Restic's command design is intuitive and the documentation is good. But Borg looks just fine in that regard as well.
    - aborsy 6y ago
      Thank you ! I posted another question on Borg, in case you know the answer!
  - magicalhippo 6y ago
    I found restic did not scale well with a large amount of files when I tried it a few years ago. Has this changed? Is Borg better? I used to run Crashplan with near-continuous backup for my important files, and I'm still missing this.
    - stevesimmons 6y ago
      I too used to use Crashplan until they discontinued their personal subscription and with it the option to backup to local drives. Around the same time I tried Restic. That ran into difficulties (don't recall what anymore) so I switched to Borg. Borg has been 100% reliable, including a full restore of /home after my laptop was stolen.
    - fidelramos 6y ago
      I don't know what you call large, but my personal restic backup runs weekly, ~250000 files, 2 TB, finishes in 7 hours saturating a 600 Mbps connection over S3 protocol. I'm very happy with restic.
      - magicalhippo 6y ago
        Well I have about 2-3x as many files, but only ~100GB. The main concern was finding out which files to take backup of, it would take quite a long time. Like I said I was hoping to get the 10-15 minute intervals I had with Crashplan.
  - rsync 6y ago
    "I wonder if I am missing something compared to restic?" The biggest difference for us is that borg really requires a server side 'borg' binary to talk to, which we have built into rsync.net. restic, on the other hand, can just connect to any old SFTP endpoint. This means we need to preserve some amount of backwards-compat and so we maintain borg0.x and borg1.x binaries in our environment (and eventually, borg2.x).
- aborsy 6y ago
  I have a question about Borg’s encryption. It’s stated [0] that if multiple clients update same repository, the server might be able to decrypt data. Why is that the case and wouldn’t that make the encryption very weak? Simultaneous updates happen quite often. Would restic have the same problem? ——————- Update: The issue happens because Borg uses AES in the CTR mode (not AES GCM) and two clients could provide the same nonce. The server could then recover the plaintext from two cipher texts. This is the famous nonce reuse problem. So Borg developers are not using established primitives for this use case. Also, I am not comfortable with the OpenSSL even though it’s got better since 2015. The libssl code base is a mess and buggy. On the other hand using the low level libcrypto library would expose developers to the crypto primitives with possibilities for errors for people not expert in cryptography. Borg should consider ChaCha-Poly135 as in rclone (or at least AES-GCM). [0]https://borgbackup.readthedocs.io/en/stable/internals/security.html#attack-model https://borgbackup.readthedocs.io/en/stable/internals/securi...
  - TimWolla 6y ago
    > Why is that the case [...]? This is explained in the "Encryption" section: https://borgbackup.readthedocs.io/en/stable/internals/security.html#encryption https://borgbackup.readthedocs.io/en/stable/internals/securi... The important part is the part about avoiding re-use of the AES CTR value. > Simultaneous updates happen quite often. Personally I created a dedicated borg repository per machine I want to backup, because that avoids sharing passphrases across machines. This comes with the drawback that I cannot deduplicate across machines, but that is acceptable to me, because the data is mostly unique-ish anyway. I only backup the user data, not everything (e.g. /bin/).
    - aborsy 6y ago
      Yes, thanks! I meanwhile read about it and updated my comment. Would it be practical to rclone the output of the Borg into a cloud service using an rclone crypt remote? In my experience, rclone’s crypt remote is sluggish, even locally. I am not sure how the mount would work. It’s unfortunate that we have to get the dedup from Borg and the encryption from rclone!
- linsomniac 6y ago
  Of course, as with all backups, you should make sure to run a test recovery. I ran into a situation with Borg where the encoding of one of my filenames caused that file and anything else behind it in the backup to be unrecoverable. Or at least I was never able to find a way to recover it. I gave it the old college try to recover, using the different tools to try to access it (the fuse mount, the CLI), I tried all sorts of different settings for my locale. At the time I had at least 2 other backups of that so eventually I recovered from my primary backup. I was testing out Borg at the time. I've ended up using Restic more recently, and it seems to be fine. Uses kind of a lot of memory in some situations though. Small AWS instances have issues. My primary backups still go via rsync though.
- m3nu 6y ago
  Since we are talking about Borg, I should plug my https://BorgBase.com https://BorgBase.com service, which is offsite backup hosting, specialized for Borg only and offers additional feature, like append-only mode, quick setup of isolated repos and monitoring for stale backups.
toast0 6y ago
What software do people like for backing up Windows desktops? I really want something that ends with a full disk image that's easy to restore to a new device, runs backups on a schedule (and will run a while after the next boot if the computer is off at the scheduled time), writes the images to a unix system on the LAN (either directly, or by writing to SMB), and doesn't cost an arm and a leg.
- theshrike79 6y ago
  I just went with Backblaze Personal. It's pretty much fire and forget. Doesn't provide a perfect full disk image, but it does store everything I need. I've done one full restore from them (fried motherboard from a power surge) and it went as smoothly as I could expect.
  - aborsy 6y ago
    Unencrypted?
    - ls612 6y ago
      There's the option to set a password to encrypt it.
      - aborsy 6y ago
        Well I should perhaps elaborate: does it offer end to end authenticated encryption with keys that never leave user’s device in an open source program? Another point, I suppose that backblaze comes with dedup and compression?
        jszymborski 6y ago
        Re: encryption, long-story short, the keys used to decrypt your data are stored in their data centers, but you can also encrypt those keys with a symmetric key which only you know. [0] Re dedup/compression, it's a bit irrelevant because their plans are unmetered. [0] https://help.backblaze.com/hc/en-us/articles/217664688-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/hc/en-us/articles/217664688-Can-y...
RealStickman_ 6y ago
I should maybe have a closer look at borg. Just to learn what alternatives there are to my current restic + Backblaze B2 setup.
walterbell 6y ago
With many people using mobile devices, a plug for PhotoSync enabling seamless photo backup and sync between iOS, Android, Windows, Mac, Linux, local NAS, iXpand local flash and cloud services. They have both subscription/rental and lifetime/ownership licenses, https://www.photosync-app.com https://www.photosync-app.com iOS storage management has improved with a user-visible "filesystem" and storage providers allowing edit-in-place, but there's still virtually no support for backup or rsync. The native iOS Files app is not a robust client for NAS storage. So far, the best option has been GoodReader (Russian devs) which implements robust sync (SMB, SFTP & more) within the app, along with optional in-app strong encryption that goes beyond iOS data protection. Unencrypted files are visible to other apps, https://www.goodreader.com/ https://www.goodreader.com/ Samsung's iXpand has built an ecosystem of iOS apps that support their custom protocol for iXpand flash drives via Lightning. Now that iPad enables access to local storage via USB-C, we need a similar ability to mount a ZFS drive, even if Apple won't provide this natively in iOS. With a low-cost x86 SBC like Odroid H2+, an entry-level NAS can be constructed with Ubuntu ZFS and dual 3.5" drives.
- DelightOne 6y ago
  > So far, the best option has been GoodReader Dropbox?
  - walterbell 6y ago
    It's about flexibility. GoodReader supports both public clouds (Dropbox, OneDrive, Google Drive, SugarSync, Box) and open protocols (WebDAV, FTP, SFTP, AFP, SMB) for private storage. Then, after remote files have been synced to/from the iDevice, editing them in-place from other apps.
    - cosmie 6y ago
      I don't use GoodReader, so can't say if it's exactly the same. But as of iOS 13, the Files app now supports directly connecting to external servers. SMB for sure, but can't find a definitive list of supported protocols. You can also use Files to directly browse and access files from third party cloud services, although it requires you to download the service's app first[1]. [1] https://support.apple.com/guide/iphone/connect-external-devices-or-servers-iph8d8f0f1a0/ios https://support.apple.com/guide/iphone/connect-external-devi...
jng 6y ago
I use TineMachine to three separate idéntica disks: one at home, another one of the office, and and a third one at my parents place 1000 km away. Pretty anythingproof. I also have three other disks with three identical copies of my old archived stuff, in the same locations. Also all code repos are online (svn, git and hg), and I have most non-code stuff on Dropbox too. Restored entire machine from TimeMachine once when I upgraded the laptop, ideal experience. I’m not happy that Covid made me have the office disks at home now too, but otherwise, I feel pretty safe.
- remote_phone 6y ago
  Is there a way to test if TimeMachine backups are uncorrupted? I backup using TimeMachine as well but as far as I can tell there is no way to verify a backup. I’m concerned that at some point my backup will get corrupted and I won’t know why. This happened to my iPhone backup to iTunes, luckily I had a iCloud backup.
  - saagarjha 6y ago
    Restore it?
    - remote_phone 6y ago
      Why on earth would I restore an image back into my MacBook if I don’t know if it’s corrupt?
      - saagarjha 6y ago
        Use another computer or partition, of course.
      - fuzxi 6y ago
        Restore it in a VM.
        remote_phone 6y ago
TimWolla 6y ago
The author uses a systemd timer to schedule their backups. For backups going to a remote host I prefer adding a little bit of variance to the execution time to avoid consistently hitting some hotspot. From the timer I use to backup my server using Borg to rsync.net: [Timer] OnUnitActiveSec=24h RandomizedDelaySec=1h This will run the backup script every 24 hours with a random delay of up to 1 hour, so every 24.5 hours on average. This causes the job to nicely rotate around the day.
- corytheboyd 6y ago
  That’s really such a nice solution to the problem, nice. Can you imagine not reading the docs to discover those options. So you spin up a database to save state about runs to implement the delay. And you need a dashboard to monitor the various parts of the system for debugging. Or you read the docs
  - coldtea 6y ago
    Or you prefix your command / script with sleep and a randomlly generated 1h value!
    - TimWolla 6y ago
      This works, but compared to using systemd it has the drawback that the range of possible times is anchored to the configured time in cron. The systemd timer example I gave causes the next cycle to start when the previous job finished. So if it initially runs the script between 0:00 and 1:00 and the script takes 1.5 hours to finish then the next run will be between 1:30 and 2:30 the next day, instead of 0:00 and 1:00.
      - MayeulC 6y ago
        > So if it initially runs the script between 0:00 and 1:00 and the script takes 1.5 hours to finish then the next run will be between 1:30 and 2:30 the next day Shouldn't it be between 1:30 ans 3:30? I'm just nitpicking, of course, that's a nice solution.
kneckebrot 6y ago
> half-assed rsync and shell script abomination I don’t understand the author’s difficulties with a minimalist bash-wrapped rsync-based backup. You can even hardlink to unchanged files from a previous backup to save space. This is how I wrap rsync: https://github.com/kaumanns/snapshot https://github.com/kaumanns/snapshot And regarding file permissions: why not simply use an EXT4 backup drive instead of an FAT32 one? Non-rhetorical question. My home network Raspberry has an HDD attached which gets fired up every couple days for a fresh snapshot of $HOME. The only thing I am missing is redundancy. And possibly encryption.
- linsomniac 6y ago
  Getting an rsync wrapper to be robust takes some work. The wrapper script I use evolved over things I found while running it across ~200 hosts nightly for a couple years. It started as one of those hardlink scripts, but evolved into using zfs snapshots. My goal was to have it be the ultimate in reliability though, I wanted it to just work as much as possible, but be quiet unless the backup failed, at which point it should let me know. 15 years later, nightly backups across maybe 300 machines, this is what I have: https://github.com/tummy-dot-com/tummy-backup/blob/master/sbin/zfsharness https://github.com/tummy-dot-com/tummy-backup/blob/master/sb...
  - tomrod 6y ago
    This is very elegant! Thank you for sharing. I look forward to studying it a bit deeper.
knorker 6y ago
Does borg prevent attacker verifying the presence of a file? Dedup content based things may have this problem, and I can't find in the docs that this is addressed. File or chunk.
beervirus 6y ago
Just use tarsnap.
- shric 6y ago
  rsync.net: 100 GB for $18/year (no ingress/egress charge) tarsnap: 100 GB would be $300/year (excluding ingress/egress) Edit: I believe both support deduplication (borg and tarsnap), but no idea if one is superior to the other?
colordrops 6y ago
I use syncthing to get all my data to a single device, and restic to back every thing up to backblaze. Restic also encrypts and reduplicates, though I don't know if it's at the block level.
grok22 6y ago
If like me you didn't click all the links on the article and were wondering where the 100GB for 18$ pricing was (it wasn't on the main pricing page), it was at: https://www.rsync.net/products/attic.html https://www.rsync.net/products/attic.html
m0zg 6y ago
I wish Backblaze supported Linux. On Mac/Windows it's impossible to beat Backblaze IMO. It saved my ass against disk failure twice already.
- Terr_ 6y ago
  I'm planning to go to Backblaze from Carbonite (which was in turn preceded by a now-defunct Crashplan offering.) Many times I've cursed at Carbonite's app for doing nothing when I want to back up, and popping up annoying when I don't.
- jimmyjack 6y ago
  I use restic pushing to backblaze from Linux Ubuntu. Works perfect.
  - m0zg 6y ago
    You're pushing to B2 though. That'll cost a pretty penny for how much data I need to store. Come to think of it though maybe I could push to a disk attached to a Windows machine, which would then back up to Backblaze. Awkward, but seems like it'd work.
    - wjdp 6y ago
      That's very likely the reason they've not bothered with a Linux client yet. I'd bet the typical Linux user has a hell of a lot more data than the typical Windows one. I use B2/restic and only selectively backup things that are either irreplaceable or I'm willing to pay for. It costs me around 1 USD/month, which is only ~200GB. I have a lot more data than that on my NAS!
GiantSully 6y ago
Currently I use node.js + rsync + duplicity to create two backups, and it works pretty fine. Firstly I create a backup to a local server via node.js and rsync, then upload them to the remote server via duplicity, which supports both encryption and compression. Both tasks run periodically and automatically, and backup files incrementally.
yoz-y 6y ago
Interesting. Personally I went the other way round and stopped doing dedicated periodic backups altogether. Photos are on my computer, phone and the cloud. Code is in repos and the few documents I need in Dropbox. I use a local time machine disk that I plug in from time to time (which I reformat if I ever need to reinstall the machine.) My main reason for ditching periodic backups (backblaze) was that even in situations where restoring a backup would be useful, I found it easier to just reinstall the OS and pull a few repos. Nice thing is that this forced me to automate the machine "setup" so I just have one script that installs my cli tools of choice and links the correct config files.
alkonaut 6y ago
The problem isn't setting up backups, the problem is that you have backups because something can happen to your data which is 1) you are clumsy or 2) someone is malicious like a hacker or burglar and 3) physical damage like harddrive corruption, cosmic rays or fire. Of these, I find 1) is by far the most common, and 2/3 isn't even close. The problem with any backup scheme created by myself is: if I couldnt' be trusted to maintain my data without deleting it, a sure as hell can't be trusted to set up my own backup scheme without screwing it up. Example: I have have had regular backups to a NAS, which were then uploaded to an offsite server were I rotated the data in. But I screwed up a raid config thing on the NAS after a harddrive faliure, and didn't notice I lost a lot of data, which after a couple of years had also been removed on the offsite server. Basically: to be a good backup solution for me it has to be idiot proof. Zero manual configuration (If there is a config file or command line anywhere, it's out). I want a gui tool that gets out of my way and has good defaults, and has such a huge disk area that it can have effectively write-only semantics. I want retention for deleted files. Currently I use iDrive which is pretty good and lets me back up parents computers and so on, in the same 2TB.
- dannyw 6y ago
  I solve this problem by buying a Synology (a popular, idiot proof NAS). Other trusted solution is QNAP. Yes, it’s cool to set up a FOSS FreeNAS server with ZFS pools. However, one look at the forum posts of people losing data due to misconfiguration tells me it should be considered a toy project. All of my important files live on the NAS as a source of truth. This means I passively make sure the data is there, every time I access something. It’s backed up using Hyper Backup to the cloud, in encrypted format. I verify restoration once a quarter. I also keep a HDD around that I manually backup to once a quarter.
  - alkonaut 6y ago
    > I solve this problem by buying a Synology My NAS was a Synology too (The one where I lost a lot of my data, despite Raid1 + offsite sync). No dry-run restoration from the cloud was my mistake.
ezconnect 6y ago
The older I get I got lazier about backups. Now I just use onedrive and icloud. I know I acn lose them all with a mistake but its alot of convenience and is cheap.
cl3misch 6y ago
I have been using borg for a while and it has been a joy. But I was under the impression that you had to trust the server if you aren't mounting the remote repo, but are running borg on the server too (like the author). Before commenting that here, I had a quick read of the docs. Turns out, I was wrong! [1] > If you use encryption, all data is encrypted on the client before being written to the repository. This means that an attacker who manages to compromise the host containing an encrypted repository will not be able to access any of the data, even while the backup is being made. borg is even more amazing than I thought. [1] https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption https://borgbackup.readthedocs.io/en/stable/quickstart.html#...
- StavrosK 6y ago
  I used to use Borg but I switched to restic, they're very similar but restic doesn't require a server (you can use BackBlaze B2).
vanpythonista 6y ago
I'm personally using restic[0] to create encrypted/de-duplicated backups. I use a local external drive and Backblaze B2 to push the snapshots to. There's no server to maintain. The best thing about restic in my opinion is the ability to mount[1] the snapshots using FUSE to my machine without actually explicitly extracting the backup to a local directory. [0] https://restic.readthedocs.io/en/latest/index.html https://restic.readthedocs.io/en/latest/index.html [1] https://restic.readthedocs.io/en/latest/050_restore.html#restore-using-mount https://restic.readthedocs.io/en/latest/050_restore.html#res... edit: formatting
- nix23 6y ago
  Jup, restic is like git for backups, really happy with it.
- baobabKoodaa 6y ago
  Restic has some failure cases where it claims that backup was successful, when in fact some of your data was not backed up. This is just about the most horrifying failure case imaginable. For example, if you take a backup of a bunch of files which includes TrueCrypt containers, and then you modify the containers and take a new backup, it will not back up the new data. Instead, it will look at file metadata to erroneously conclude that the container has not changed. Now, some people argue that this is not an issue, because you can use non default configuration of TrueCrypt and/or Restic to prevent this problem. But how would a Restic user know that they need to do this? I don't want to become an expert in the internals of the backup software I'm using. I just want it to work -- or at least fail in predictable ways.
nix23 6y ago
I used borg for a long time but shifted over to Restic (mostly because of single binary and win/mac/linux/bsd compatibility)

207 points by miked85 6y ago

If you want to find out how to backup your iPhone on Linux, I've also made a guide! It's actually kind of complicated, but it can be fully automated. I connect my iPhone to my Linux server in my room to do an incremental backup every night at 5 AM (it also fast charges at 1.8 amps over USB C). I then create ZFS snapshots every week, since the iPhone backup is an overwrite type. https://www.naut.ca/blog/2020/03/20/self-hosting-series-part-4-backup/#compilingidevicebackup2 https://www.naut.ca/blog/2020/03/20/self-hosting-series-part...

walterbell 6y ago
Thanks for your self-hosting tutorials for iOS services! The next question is how to extract individual files from a backup, without needed an iDevice for a full restore. There are several commercial products sold for this purpose, but I've not yet seen OSS tools to parse iOS backups.

graton 6y ago

Like the author I have also been very happy with Borg backup software ( https://www.borgbackup.org/ https://www.borgbackup.org/ ). The compression and de-duplication is very useful. A little bit of a learning curve to get everything up and running, but not too bad.

aborsy 6y ago
I also like Borg back up. I wonder if I am missing something compared to restic?
- more-coffee 6y ago
  Borg is supposedly a bit faster, restic supports a ton of backends https://github.com/restic/restic/issues/1875 https://github.com/restic/restic/issues/1875 Also restic does not allow to backup unencrypted. I haven't used borg. Only done some maintenance on restic backup jobs at work. Restic's command design is intuitive and the documentation is good. But Borg looks just fine in that regard as well.
  - aborsy 6y ago
    Thank you ! I posted another question on Borg, in case you know the answer!
- magicalhippo 6y ago
  I found restic did not scale well with a large amount of files when I tried it a few years ago. Has this changed? Is Borg better? I used to run Crashplan with near-continuous backup for my important files, and I'm still missing this.
  - stevesimmons 6y ago
    I too used to use Crashplan until they discontinued their personal subscription and with it the option to backup to local drives. Around the same time I tried Restic. That ran into difficulties (don't recall what anymore) so I switched to Borg. Borg has been 100% reliable, including a full restore of /home after my laptop was stolen.
  - fidelramos 6y ago
    I don't know what you call large, but my personal restic backup runs weekly, ~250000 files, 2 TB, finishes in 7 hours saturating a 600 Mbps connection over S3 protocol. I'm very happy with restic.
    - magicalhippo 6y ago
      Well I have about 2-3x as many files, but only ~100GB. The main concern was finding out which files to take backup of, it would take quite a long time. Like I said I was hoping to get the 10-15 minute intervals I had with Crashplan.
- rsync 6y ago
  "I wonder if I am missing something compared to restic?" The biggest difference for us is that borg really requires a server side 'borg' binary to talk to, which we have built into rsync.net. restic, on the other hand, can just connect to any old SFTP endpoint. This means we need to preserve some amount of backwards-compat and so we maintain borg0.x and borg1.x binaries in our environment (and eventually, borg2.x).
aborsy 6y ago
I have a question about Borg’s encryption. It’s stated [0] that if multiple clients update same repository, the server might be able to decrypt data. Why is that the case and wouldn’t that make the encryption very weak? Simultaneous updates happen quite often. Would restic have the same problem? ——————- Update: The issue happens because Borg uses AES in the CTR mode (not AES GCM) and two clients could provide the same nonce. The server could then recover the plaintext from two cipher texts. This is the famous nonce reuse problem. So Borg developers are not using established primitives for this use case. Also, I am not comfortable with the OpenSSL even though it’s got better since 2015. The libssl code base is a mess and buggy. On the other hand using the low level libcrypto library would expose developers to the crypto primitives with possibilities for errors for people not expert in cryptography. Borg should consider ChaCha-Poly135 as in rclone (or at least AES-GCM). [0]https://borgbackup.readthedocs.io/en/stable/internals/security.html#attack-model https://borgbackup.readthedocs.io/en/stable/internals/securi...
- TimWolla 6y ago
  > Why is that the case [...]? This is explained in the "Encryption" section: https://borgbackup.readthedocs.io/en/stable/internals/security.html#encryption https://borgbackup.readthedocs.io/en/stable/internals/securi... The important part is the part about avoiding re-use of the AES CTR value. > Simultaneous updates happen quite often. Personally I created a dedicated borg repository per machine I want to backup, because that avoids sharing passphrases across machines. This comes with the drawback that I cannot deduplicate across machines, but that is acceptable to me, because the data is mostly unique-ish anyway. I only backup the user data, not everything (e.g. /bin/).
  - aborsy 6y ago
    Yes, thanks! I meanwhile read about it and updated my comment. Would it be practical to rclone the output of the Borg into a cloud service using an rclone crypt remote? In my experience, rclone’s crypt remote is sluggish, even locally. I am not sure how the mount would work. It’s unfortunate that we have to get the dedup from Borg and the encryption from rclone!
linsomniac 6y ago
Of course, as with all backups, you should make sure to run a test recovery. I ran into a situation with Borg where the encoding of one of my filenames caused that file and anything else behind it in the backup to be unrecoverable. Or at least I was never able to find a way to recover it. I gave it the old college try to recover, using the different tools to try to access it (the fuse mount, the CLI), I tried all sorts of different settings for my locale. At the time I had at least 2 other backups of that so eventually I recovered from my primary backup. I was testing out Borg at the time. I've ended up using Restic more recently, and it seems to be fine. Uses kind of a lot of memory in some situations though. Small AWS instances have issues. My primary backups still go via rsync though.
m3nu 6y ago
Since we are talking about Borg, I should plug my https://BorgBase.com https://BorgBase.com service, which is offsite backup hosting, specialized for Borg only and offers additional feature, like append-only mode, quick setup of isolated repos and monitoring for stale backups.

toast0 6y ago

What software do people like for backing up Windows desktops? I really want something that ends with a full disk image that's easy to restore to a new device, runs backups on a schedule (and will run a while after the next boot if the computer is off at the scheduled time), writes the images to a unix system on the LAN (either directly, or by writing to SMB), and doesn't cost an arm and a leg.

theshrike79 6y ago
I just went with Backblaze Personal. It's pretty much fire and forget. Doesn't provide a perfect full disk image, but it does store everything I need. I've done one full restore from them (fried motherboard from a power surge) and it went as smoothly as I could expect.
- aborsy 6y ago
  Unencrypted?
  - ls612 6y ago
    There's the option to set a password to encrypt it.
    - aborsy 6y ago
      Well I should perhaps elaborate: does it offer end to end authenticated encryption with keys that never leave user’s device in an open source program? Another point, I suppose that backblaze comes with dedup and compression?
      - jszymborski 6y ago
        Re: encryption, long-story short, the keys used to decrypt your data are stored in their data centers, but you can also encrypt those keys with a symmetric key which only you know. [0] Re dedup/compression, it's a bit irrelevant because their plans are unmetered. [0] https://help.backblaze.com/hc/en-us/articles/217664688-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/hc/en-us/articles/217664688-Can-y...

RealStickman_ 6y ago

I should maybe have a closer look at borg. Just to learn what alternatives there are to my current restic + Backblaze B2 setup.

walterbell 6y ago

With many people using mobile devices, a plug for PhotoSync enabling seamless photo backup and sync between iOS, Android, Windows, Mac, Linux, local NAS, iXpand local flash and cloud services. They have both subscription/rental and lifetime/ownership licenses, https://www.photosync-app.com https://www.photosync-app.com iOS storage management has improved with a user-visible "filesystem" and storage providers allowing edit-in-place, but there's still virtually no support for backup or rsync. The native iOS Files app is not a robust client for NAS storage. So far, the best option has been GoodReader (Russian devs) which implements robust sync (SMB, SFTP & more) within the app, along with optional in-app strong encryption that goes beyond iOS data protection. Unencrypted files are visible to other apps, https://www.goodreader.com/ https://www.goodreader.com/ Samsung's iXpand has built an ecosystem of iOS apps that support their custom protocol for iXpand flash drives via Lightning. Now that iPad enables access to local storage via USB-C, we need a similar ability to mount a ZFS drive, even if Apple won't provide this natively in iOS. With a low-cost x86 SBC like Odroid H2+, an entry-level NAS can be constructed with Ubuntu ZFS and dual 3.5" drives.

DelightOne 6y ago
> So far, the best option has been GoodReader Dropbox?
- walterbell 6y ago
  It's about flexibility. GoodReader supports both public clouds (Dropbox, OneDrive, Google Drive, SugarSync, Box) and open protocols (WebDAV, FTP, SFTP, AFP, SMB) for private storage. Then, after remote files have been synced to/from the iDevice, editing them in-place from other apps.
  - cosmie 6y ago
    I don't use GoodReader, so can't say if it's exactly the same. But as of iOS 13, the Files app now supports directly connecting to external servers. SMB for sure, but can't find a definitive list of supported protocols. You can also use Files to directly browse and access files from third party cloud services, although it requires you to download the service's app first[1]. [1] https://support.apple.com/guide/iphone/connect-external-devices-or-servers-iph8d8f0f1a0/ios https://support.apple.com/guide/iphone/connect-external-devi...

jng 6y ago

I use TineMachine to three separate idéntica disks: one at home, another one of the office, and and a third one at my parents place 1000 km away. Pretty anythingproof. I also have three other disks with three identical copies of my old archived stuff, in the same locations. Also all code repos are online (svn, git and hg), and I have most non-code stuff on Dropbox too. Restored entire machine from TimeMachine once when I upgraded the laptop, ideal experience. I’m not happy that Covid made me have the office disks at home now too, but otherwise, I feel pretty safe.

remote_phone 6y ago
Is there a way to test if TimeMachine backups are uncorrupted? I backup using TimeMachine as well but as far as I can tell there is no way to verify a backup. I’m concerned that at some point my backup will get corrupted and I won’t know why. This happened to my iPhone backup to iTunes, luckily I had a iCloud backup.
- saagarjha 6y ago
  Restore it?
  - remote_phone 6y ago
    Why on earth would I restore an image back into my MacBook if I don’t know if it’s corrupt?
    - saagarjha 6y ago
      Use another computer or partition, of course.
    - fuzxi 6y ago
      Restore it in a VM.
      - remote_phone 6y ago

TimWolla 6y ago

The author uses a systemd timer to schedule their backups. For backups going to a remote host I prefer adding a little bit of variance to the execution time to avoid consistently hitting some hotspot. From the timer I use to backup my server using Borg to rsync.net: [Timer] OnUnitActiveSec=24h RandomizedDelaySec=1h This will run the backup script every 24 hours with a random delay of up to 1 hour, so every 24.5 hours on average. This causes the job to nicely rotate around the day.

corytheboyd 6y ago
That’s really such a nice solution to the problem, nice. Can you imagine not reading the docs to discover those options. So you spin up a database to save state about runs to implement the delay. And you need a dashboard to monitor the various parts of the system for debugging. Or you read the docs
- coldtea 6y ago
  Or you prefix your command / script with sleep and a randomlly generated 1h value!
  - TimWolla 6y ago
    This works, but compared to using systemd it has the drawback that the range of possible times is anchored to the configured time in cron. The systemd timer example I gave causes the next cycle to start when the previous job finished. So if it initially runs the script between 0:00 and 1:00 and the script takes 1.5 hours to finish then the next run will be between 1:30 and 2:30 the next day, instead of 0:00 and 1:00.
    - MayeulC 6y ago
      > So if it initially runs the script between 0:00 and 1:00 and the script takes 1.5 hours to finish then the next run will be between 1:30 and 2:30 the next day Shouldn't it be between 1:30 ans 3:30? I'm just nitpicking, of course, that's a nice solution.

kneckebrot 6y ago

> half-assed rsync and shell script abomination I don’t understand the author’s difficulties with a minimalist bash-wrapped rsync-based backup. You can even hardlink to unchanged files from a previous backup to save space. This is how I wrap rsync: https://github.com/kaumanns/snapshot https://github.com/kaumanns/snapshot And regarding file permissions: why not simply use an EXT4 backup drive instead of an FAT32 one? Non-rhetorical question. My home network Raspberry has an HDD attached which gets fired up every couple days for a fresh snapshot of $HOME. The only thing I am missing is redundancy. And possibly encryption.

linsomniac 6y ago
Getting an rsync wrapper to be robust takes some work. The wrapper script I use evolved over things I found while running it across ~200 hosts nightly for a couple years. It started as one of those hardlink scripts, but evolved into using zfs snapshots. My goal was to have it be the ultimate in reliability though, I wanted it to just work as much as possible, but be quiet unless the backup failed, at which point it should let me know. 15 years later, nightly backups across maybe 300 machines, this is what I have: https://github.com/tummy-dot-com/tummy-backup/blob/master/sbin/zfsharness https://github.com/tummy-dot-com/tummy-backup/blob/master/sb...
- tomrod 6y ago
  This is very elegant! Thank you for sharing. I look forward to studying it a bit deeper.

knorker 6y ago

Does borg prevent attacker verifying the presence of a file? Dedup content based things may have this problem, and I can't find in the docs that this is addressed. File or chunk.

beervirus 6y ago

Just use tarsnap.

shric 6y ago
rsync.net: 100 GB for $18/year (no ingress/egress charge) tarsnap: 100 GB would be $300/year (excluding ingress/egress) Edit: I believe both support deduplication (borg and tarsnap), but no idea if one is superior to the other?

colordrops 6y ago

I use syncthing to get all my data to a single device, and restic to back every thing up to backblaze. Restic also encrypts and reduplicates, though I don't know if it's at the block level.

grok22 6y ago

If like me you didn't click all the links on the article and were wondering where the 100GB for 18$ pricing was (it wasn't on the main pricing page), it was at: https://www.rsync.net/products/attic.html https://www.rsync.net/products/attic.html

m0zg 6y ago

I wish Backblaze supported Linux. On Mac/Windows it's impossible to beat Backblaze IMO. It saved my ass against disk failure twice already.

Terr_ 6y ago
I'm planning to go to Backblaze from Carbonite (which was in turn preceded by a now-defunct Crashplan offering.) Many times I've cursed at Carbonite's app for doing nothing when I want to back up, and popping up annoying when I don't.
jimmyjack 6y ago
I use restic pushing to backblaze from Linux Ubuntu. Works perfect.
- m0zg 6y ago
  You're pushing to B2 though. That'll cost a pretty penny for how much data I need to store. Come to think of it though maybe I could push to a disk attached to a Windows machine, which would then back up to Backblaze. Awkward, but seems like it'd work.
  - wjdp 6y ago
    That's very likely the reason they've not bothered with a Linux client yet. I'd bet the typical Linux user has a hell of a lot more data than the typical Windows one. I use B2/restic and only selectively backup things that are either irreplaceable or I'm willing to pay for. It costs me around 1 USD/month, which is only ~200GB. I have a lot more data than that on my NAS!

GiantSully 6y ago

Currently I use node.js + rsync + duplicity to create two backups, and it works pretty fine. Firstly I create a backup to a local server via node.js and rsync, then upload them to the remote server via duplicity, which supports both encryption and compression. Both tasks run periodically and automatically, and backup files incrementally.

yoz-y 6y ago

Interesting. Personally I went the other way round and stopped doing dedicated periodic backups altogether. Photos are on my computer, phone and the cloud. Code is in repos and the few documents I need in Dropbox. I use a local time machine disk that I plug in from time to time (which I reformat if I ever need to reinstall the machine.) My main reason for ditching periodic backups (backblaze) was that even in situations where restoring a backup would be useful, I found it easier to just reinstall the OS and pull a few repos. Nice thing is that this forced me to automate the machine "setup" so I just have one script that installs my cli tools of choice and links the correct config files.

alkonaut 6y ago

The problem isn't setting up backups, the problem is that you have backups because something can happen to your data which is 1) you are clumsy or 2) someone is malicious like a hacker or burglar and 3) physical damage like harddrive corruption, cosmic rays or fire. Of these, I find 1) is by far the most common, and 2/3 isn't even close. The problem with any backup scheme created by myself is: if I couldnt' be trusted to maintain my data without deleting it, a sure as hell can't be trusted to set up my own backup scheme without screwing it up. Example: I have have had regular backups to a NAS, which were then uploaded to an offsite server were I rotated the data in. But I screwed up a raid config thing on the NAS after a harddrive faliure, and didn't notice I lost a lot of data, which after a couple of years had also been removed on the offsite server. Basically: to be a good backup solution for me it has to be idiot proof. Zero manual configuration (If there is a config file or command line anywhere, it's out). I want a gui tool that gets out of my way and has good defaults, and has such a huge disk area that it can have effectively write-only semantics. I want retention for deleted files. Currently I use iDrive which is pretty good and lets me back up parents computers and so on, in the same 2TB.

dannyw 6y ago
I solve this problem by buying a Synology (a popular, idiot proof NAS). Other trusted solution is QNAP. Yes, it’s cool to set up a FOSS FreeNAS server with ZFS pools. However, one look at the forum posts of people losing data due to misconfiguration tells me it should be considered a toy project. All of my important files live on the NAS as a source of truth. This means I passively make sure the data is there, every time I access something. It’s backed up using Hyper Backup to the cloud, in encrypted format. I verify restoration once a quarter. I also keep a HDD around that I manually backup to once a quarter.
- alkonaut 6y ago
  > I solve this problem by buying a Synology My NAS was a Synology too (The one where I lost a lot of my data, despite Raid1 + offsite sync). No dry-run restoration from the cloud was my mistake.

ezconnect 6y ago

The older I get I got lazier about backups. Now I just use onedrive and icloud. I know I acn lose them all with a mistake but its alot of convenience and is cheap.

cl3misch 6y ago

I have been using borg for a while and it has been a joy. But I was under the impression that you had to trust the server if you aren't mounting the remote repo, but are running borg on the server too (like the author). Before commenting that here, I had a quick read of the docs. Turns out, I was wrong! [1] > If you use encryption, all data is encrypted on the client before being written to the repository. This means that an attacker who manages to compromise the host containing an encrypted repository will not be able to access any of the data, even while the backup is being made. borg is even more amazing than I thought. [1] https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption https://borgbackup.readthedocs.io/en/stable/quickstart.html#...

StavrosK 6y ago
I used to use Borg but I switched to restic, they're very similar but restic doesn't require a server (you can use BackBlaze B2).

vanpythonista 6y ago

I'm personally using restic[0] to create encrypted/de-duplicated backups. I use a local external drive and Backblaze B2 to push the snapshots to. There's no server to maintain. The best thing about restic in my opinion is the ability to mount[1] the snapshots using FUSE to my machine without actually explicitly extracting the backup to a local directory. [0] https://restic.readthedocs.io/en/latest/index.html https://restic.readthedocs.io/en/latest/index.html [1] https://restic.readthedocs.io/en/latest/050_restore.html#restore-using-mount https://restic.readthedocs.io/en/latest/050_restore.html#res... edit: formatting

nix23 6y ago
Jup, restic is like git for backups, really happy with it.
baobabKoodaa 6y ago
Restic has some failure cases where it claims that backup was successful, when in fact some of your data was not backed up. This is just about the most horrifying failure case imaginable. For example, if you take a backup of a bunch of files which includes TrueCrypt containers, and then you modify the containers and take a new backup, it will not back up the new data. Instead, it will look at file metadata to erroneously conclude that the container has not changed. Now, some people argue that this is not an issue, because you can use non default configuration of TrueCrypt and/or Restic to prevent this problem. But how would a Restic user know that they need to do this? I don't want to become an expert in the internals of the backup software I'm using. I just want it to work -- or at least fail in predictable ways.

nix23 6y ago

I used borg for a long time but shifted over to Restic (mostly because of single binary and win/mac/linux/bsd compatibility)