15 ms·

How I store my files

247 points by iio7 5y ago

1MachineElf 5y ago
I'm near a very similar setup. Syncthing for file replication, storage/backup to ZFS TrueNAS, then a secondary backup to another ZFS system that's usually offline. I wonder how OP does snapshot rotation.
tmslnz 5y ago
My approach these days: 1. Hot live data on my laptop. 2. Archive dir on NAS. It’s either 1 or 2, no overlap. Then: 1. Backup whole laptop to USB bootable drive “when I fancy”. 2. Backup home dir to NAS using duplicacy. This is accessible via SFTP, so it works no matter where I am. 3. Backup the Archive dir of the NAS to cheap cloud storage. Ideally I should add an external drive for the NAS to backup #3 locally, too.
renonce 5y ago
I put everything in a Resilio Sync folder, and keep a full sync on at least two devices (a home NAS and a cloud seedbox). Resilio Sync handles pretty much everything. You instantly get hot backup, have files immediately available on every device you have, and if you have a phone you can download any file on demand, etc. Unlike other file synchronization methods such as `rsync --delete`, it keeps a version for every file modification and moves the file to Archive when it is deleted, so you can't lose data. Also, you get encryption without the headache by using the "encrypted folder key". This syncs an encrypted copy so other devices can sync from that device.
- wereHamster 5y ago
  I use resilio sync together with a NAS (which is still large enough to hold all my files, I wonder how long that'll stay given how much 6k BRAW footage I'm shooting), and I also sync to a Google Cloud Storage bucket from my NAS (moving items to Archive storage class if they haven't been touched in a while, significantly saving cost).
- deleted 5y ago
  [deleted]
faeyanpiraat 5y ago
Excellent article. The method is simple but strong. The most important part is that it is actually implementable, examples and reasoning are provided for all steps.
jarenmf 5y ago
I use borg backup to sync all our files (from ~5 machines) to an old PC which synchronizes everything to the cloud. Borg has deduplication, compression, encryption, and saves several versions of the files which I think is crucial. Also all text-based data are on Git like software, papers, etc.
- unilynx 5y ago
  Is your PC synchronising the source files or the borg files to the cloud ? Otherwise it's still a SPOF for your backups (the borg files might corrupt, and you might be syncing that corruption to the cloud)
  - jarenmf 5y ago
    Good point. I am indeed synchronizing the borg file. I have however my script run borg check before it synchronizes to the cloud. It would be better to synchronize the source files but the encryption and deduplication are what's keeping me from doing that.
ubermonkey 5y ago
It's a good discussion. I've taken a different path but with the same ideals. 1. All my live, working data is in Dropbox. I've been using my (paid) Dropbox account for over a decade. This step isn't really intended to be a backup per se, but you get the backup behavior "for free" because this is how I keep my secondary machine in sync with my primary. My Dropbox folder is also replicated to a 3rd computer in my home as another level of redundancy. 2. Everything in my home dir, especially photography, is ALSO covered by Backblaze. Having some backups elsewhere is mandatory; not enough people really understand how important this is. Should my house burn down, I still have data. 3. My primary system is a Mac, so I use Time Machine. TM is the only backup on this list I've ever actually used as a backup. When our home was robbed a number of years ago in a quickie smash-window-and-grab affair, they got my laptop. I went out and bought a new computer, plugged it into the TM drive, and in an hour or so I was right back where I left off. Hard to beat that. Even my app windows were in the same place. 4. Periodically, I take a full clone of my main machine's drive using a drive imaging tool (Mac specific; I use SuperDuper) and **store that drive at a friend's house**. This probably only happens a few times a year a this point. I should do it more often. That tertiary computer I mentioned in step 1 is also the home NAS server / home media server. It holds the photo archive in a large outboard disk. Backblaze covers that disk, and Time Machine on that computer keeps the outboard disk backed up as well. This data is mostly static, so the images I've taken of it and stored elsewhere don't need to be updated all that often (ie, just when I migrate prior year photo data to that drive).
- malshe 5y ago
  My set up is strikingly similar to yours! I keep a backup in my office instead of a friend’s home but my logic is the same as yours.
leotaku 5y ago
I currently trust Restic with basically all of my long-term backups, which, according to the author, really isn't a thing I should do. However, I'm still somewhat confident in my strategy as I backup my all of my data to two entirely different repositories, one of them backed by Google Cloud, and another by the server sitting in my pantry. So one of these repositories could get irrecoverably corrupted and I still wouldn't lose any of my data. With cloud storage becoming so cheap I've also thought about adding a third repo. Of course this would not protect me from a hypothetical bug in Restic that corrupts all my repositories before I notice, so maybe I should also add another auto-backup solution into the mix. Doing manual things like moving data to external storage seems like a robust strategy, but I really don't trust myself to do something like that nearly often enough for it to be useful.
- wedowhatwedo 5y ago
  I have used restic for quite a while. Once in a while, I test that I can restore my backups. That's an important step that lots of people miss. I had a client that asked me to setup their system. I setup the system, they got a tape drive and I had them rotate tapes daily. There was a cronjob to tar everything to tapes. It was great until their hard drive failed and I found I had a typo where it was only backing up the current folder, not the whole drive. Needless to say, they found a new IT provider and I learned an important lesson. If you haven't tested your backups, you have no backups.
ChrisMarshallNY 5y ago
Being an Apple chap, I have found that Time Machine works really well. I use a Synology 5-disk NAS. It has a ton of capacity, and allows me to increase capacity, if needed. My important stuff is in the cloud. I have a nightly "hot backup," that is a CC clone of my main and dev drives. It used to be every 4 hours, but I found that to be overkill. I almost never need anything more than the "hot backup."
visviva 5y ago
Darn, I was hoping this would be an article about organizing one's files. I was really in the mood for reading about that, then spending the rest of the morning reworking my own system.
- kawfey 5y ago
  Same. I get lost in researching everyone else's file organization and other workflow methods. I always feel like mine is a dog's breakfast, scattered across many incongruent, un-coordinated drives, clouds, shares, etc.
- pricecomstock 5y ago
  I have used this system in certain areas before. It's decent. But it's interesting to read about if you're in the mood! https://johnnydecimal.com/ https://johnnydecimal.com/
  - visviva 5y ago
    That's what I'm talking about... and a wasted Friday afternoon is even better than a wasted Friday morning!
  - tailspin2019 5y ago
    Oh man, that's a cool approach. Down the rabbit hole I go...
- quacked 5y ago
  This one is great too: https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
  - olivertaylor 5y ago
    One of my favorites.
Gurdt 5y ago
Can someone point me to the flaws of my method? I simply use Google Drive. Me and my partner have multiple devices (tablet, iPhone, Android phone, multiple Mac and Windows laptops) and we just sync folders to our desktop. We just store everything we don't want to lose in Drive. We share folders we both need. Photo's we take on our phones are automagically backed up in the cloud, music and movies are streamed. My house can burn down overnight and we won't lose any valuable data.
- CatAtHeart 5y ago
  Although I'm not sure of the rate, people have reported getting banned from google services without notice for seemingly no apparent reason. And there is little recourse to unlocking their account. It's something I personally don't worry about, but something to consider when relying on their services for backing up data.
- leephillips 5y ago
  The flaw is that you depend on Google. You can not trust them at all, for anything. You can use Drive as a redundant fifth backup either for data that you’ve encrypted yourself or for things that you don’t mind being public, but not as the only place that has your data. Things that could very well happen: you get locked out with no recourse; people at Google browse your data; your data gets mined; your data gets lost; Google messes up and other people get access to your data; Google gets a subpoena and those photos that you thought were perfectly normal are now evidence of a manufactured crime.
- mbiondi 5y ago
  I do the same with Dropbox. I've been doing this for years and it's saved me a number of times. I don't know if there are flaws with this but it's worked better than anything I've cobbled together myself.
- foobarian 5y ago
  It should work fine until you get into the territory of having to store multiple terabytes of data, at which point cloud services can get expensive.
- 5y ago
mongol 5y ago
Something tricky is how to bootstrap restoration of backup. If you have lost "everything", how do you get it back? For example, if you use borg to backup remotely via ssh, you will need ssh keys as well as a passphrase for the encrypted backup. Where do you store those to make sure you have them if your computer is gone? What I did was create a self-extracting restoration script, which embedded everything needed. This is also encrypted, and synced to many places. The idea is, as long as I have the passphrase for that, it takes care of the rest.
- beervirus 5y ago
  For small stuff like this, print it out and put it in a safe deposit box.
  - codesections 5y ago
    This would be the only reason I'd need to pay for a safe deposit box. It might be worth it, just for that, but I haven't made that leap yet.
    - reidjs 5y ago
      If you're on good terms with your family, it might make sense to leave it with them (if you trust them to keep it safe).
    - reaperducer 5y ago
      Check with your bank. If you have enough accounts, or the right kind of accounts, they may give you one for free. It's usually listed included in the long list of benefits that almost nobody ever reads. Each year I get a bill from a major American bank in the amount of $0 for a safe deposit box I don't use because I have a certain amount of money in one account.
  - rovr138 5y ago
    I print my important keys to paper but also as a QR code. Easier to restore. Have't needed to use them yet, but testing it was great.
jl6 5y ago
Every good backup practitioner has an origin story. That one time they lost data to a clicking HDD, or to a scratched CD, or to a typo. The pain of never getting that data back. Galvanizing pain that catalyzed a resolution never to lose data again. Never trust someone (including yourself) to truly look after your data unless they know this pain first hand.
- mirodin 5y ago
  So poetic and so true!
deleted 5y ago
[deleted]
PragmaticPulp 5y ago
I also have a ZFS NAS and a separate backup server for the NAS that only gets turned on once a week. I disagree about his recommendation to avoid encryption. I keep the NAS and backup server drives encrypted. It’s not a big deal to unlock the drives at boot with a password and I’ve never had any issues with it. If someone breaks in and steals one of the servers, at least they won’t have access to years of all of my documents backed up. This author’s setup looks good, but I think most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved. A complex solution isn’t necessarily better if you find yourself not investing the time needed to maintain it. I think most people would be better off trusting one of the online backup services rather than spending thousands of dollars building and maintaining a pair of backup servers. If you’re worried about losing data, signing up for two separate backup services could still be cheaper than building and running two separate NAS servers. Start with easy backup solutions first to get something going. Then consider the more complicated options later.
- sgarland 5y ago
  > most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved. I didn't see the author's stated OS they use for their NAS, but depending on what it is, yes. {Free, True}NAS are dead easy to use as long as you stay inside the confines of the GUI, and hugely reliable. Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool! I have a setup very similar to the author's, with a bunch of Debian VMs on top of Proxmox. Can confirm that even with Packer and Ansible handling base images and configuration, weird edge cases pop up that take quite a while to track down.
  - mbiondi 5y ago
    I thought the same about Free/True NAS until I lost a ton of stuff. Be careful with these. Also, for big backups the performance has been terrible. I carefully followed their guide and the write speed is just dismal.
optimalsolver 5y ago
>Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data Uh, trust me, that's a good thing.
- teekert 5y ago
  For your browser history, sure. But I for one make sure I rsync stuff to a Raspberry Pi4 with an external disk, at my parents place every now and then. In the event of my demise they can unplug it and have all our family pictures and videos. I'm really afraid that I make things to complicated for the people that I leave behind.
  - adkadskhj 5y ago
    I dabble in "life storage" and your comment made me think that some sort of executable shipped alongside backup locations to read the data, if in some deduplicated backup form, seems valuable. Eg Camlistore/Perkeep had the premise of using JSON to store data. However some random person isn't going to write code to parse all that data, pull files out, etc. A lifeboat .exe might be interesting. Though doing it in the most simple, least configurable, least breakable way seems.. necessary. Yea some baked in UI would be cool, but more moving parts means less likely to work.
  - mytdi 5y ago
    I have the same setup for the same reason as you. Only difference, I have my setup at home. As you say, in the event of my demise, I want my relatives to be able to grab the drive and have easy access to the pictures and videos.
- scohesc 5y ago
  If I can't delete my browsing history before I pass on, you bet your bottom dollar I'm not letting my family have my keys!
  - avh02 5y ago
csabakissi 5y ago
My approach is 6 years old and I use Time Capsule. The best ever solution for me. You set it up and you don't case anymore. It's Apple specific unfortunately. Don't know if there is a similar solution for Windows.
- diffeomorphism 5y ago
  Just fyi those seem to be experiencing a data loss issue: https://appleinsider.com/articles/21/07/08/design-failure-in-apples-time-capsule-leads-to-data-loss https://appleinsider.com/articles/21/07/08/design-failure-in...
beervirus 5y ago
Does anybody use tape? I currently have a Synology RAID6 and I use a cloud backup service. But I’ve been thinking about adding another layer in the form of off-site tapes. Any recommendations welcome.
- tomatocracy 5y ago
  I have tapes as my last line of defence (ZFS snapshots, backups to NAS, online are my earlier lines of defence). The hardware itself is relatively simple but (at least in my experience - I tried quite a few options but they were all missing at least one key feature) there's no good "set and forget" software solution out there in the same way as there is for online backups so you're going to spend a long time scripting it all and tweaking it properly. You can build a solution on top of whichever other tools fit your needs that way. I have a set of scripts which create my backups in single large files on disk ready to be spooled out, then I use mbuffer to actually put the data onto tape. In terms of hardware, 1 or 2 generation old LTO drives are often available for a decent price in good condition on eBay. You'll also need either an SAS or Fibre Channel interface depending on what drive you buy. As far as I can tell they're pretty much all supported by Linux these days. Given the complexity of scripting and potential for user errors, testing that you can restore properly is pretty well essential, of course.
armoredkitten 5y ago
I've been using btrfs and btrbk for a few months now and really enjoy it. I love that snapshots are easy to make, and btrbk is a fairly simple way to schedule taking snapshots and transferring them to another machine, as well as set up a retention strategy. I agree with the author that simpler is better -- if I lose data/delete a file I want back, it's as simple as mounting the snapshot and grabbing what I need.
allenbina 5y ago
Regarding the issue he mentions with unmounting of drives on raspberry pis, I've found the issue that fixed it for me was a reliable power supply.
mdale 5y ago
I thought most people just get fast internet and let "the cloud" handle it now days ? Seems like a lot of work to ultimately not do things not as well as army of engineers focused on this problem full time.
- kawfey 5y ago
  Most people do (myself included, I just put everything on Backblaze or Google Drive), it's just that HN isn't most people.
- allenbina 5y ago
  I took the deep dive into tech trying to fix the 2x cd drive someone gave me for my x86 pc. Part of the passion I have for tech is learning and digging deeper, and reading about topics I haven't learned yet to the point where I get a bit of malaise when I'm not learning something new. I say this as I'm repurposing an old pc to a full time nas and learning about zfs pools. There's always a simpler, faster, more expensive solution out there that takes the complexity out of the equation, but part of the fun is the complexity for me.
anjuan 5y ago
For the past few years, I've created a photo book of the best family pictures from the past year. I store these photobooks in a fireproof safe. I think that's the easiest way to pass down pictures since it's paper doesn't require decryption or passwords.
- codetrotter 5y ago
  This summer I forgot the PIN of my iPad, because I didn’t use it during all summer. And it’s only 6 digits. I had, however, made note of the PIN elsewhere so I was able to unlock it. I have similarly forgotten PINs and passwords in the past that I didn’t use for a while. I recommend unlocking your safe at a regular interval, so that you remember the combination, if it uses a code lock. And even if it’s a key lock it’s probably a good idea to do it too, so you know where the key is.
  - benhurmarcel 5y ago
    This was the reason I started using a password manager in the beginning, I didn’t trust my memory to remember them.
ga3h 5y ago
Thumb drives. Financial stuff, copies of legal documents, large libraries of PDF books, reports, etc., etc. And then back the main drive up to FOUR thumb drive backups. Reasons: (1) USB drives are huge now. (2) Convenient - can move from one PC to another easily. (3) Convenient - I do NOT have time to mess with other solutions. (4) Safety - valuable financial stuff almost always OFFLINE reducing hacker vulnerability. (5) All PCs eventually fail - I'm old and have a collection in my basement - plus I have two fairly new ones awaiting repair in my home office right now. Only disadvantage - I do have multiple legal copies of Microsoft Office on multiple PC to make moving thumb drives around practical.
- shreddit 5y ago
  Doesn’t thumbdrive data decay over time? Like all flash based storage types do
  - gen3 5y ago
    That is my understanding. I've heard numbers around 10yr +-5, but I've never seen a study on it. I wonder if storing them zip-locked in a freezer would help (but I guess you could just use a magnetic hardrive :D )
    - mceachen 5y ago
      That the number I've personally experienced (I also replied to GP). It's an interesting idea to deep-freeze your drives, but the thermal stress from modern frost-free freezers that automatically dethaw periodically may negate the benefit of chilly bits... If you don't have that much to store, and are worried about data durability, I'd suggest looking into M-disc instead.
  - dangerface 5y ago
    Yes they do. I used pen drives for backups after I found one that was like 10 years old and still had my old cv on it. I stored my backups on two identical pen drives I stopped doing this when I went to backup and discovered one of the pen drives was corrupt and lost all its data. They are super cheap and fairly reliable but not quite reliable enough for backups.
rsync 5y ago
"While I know of some really good cloud providers, such as rsync.net and Tarsnap, I recommend that you never trust cloud providers blindly." This is very good advice. That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard. ... "There exist some really cool open source backup solutions such as Borg, Restic and duplicity, but you should never rely solely on these "complex" solutions. These tools work really great, until they don't! In the past I have lost data to duplicity and other tools." I think this is very good advice as well - and that is coming from someone who has whole-heartedly endorsed 'borg' as a backup tool and regularly recommends it. It is the "holy grail of backups"[1] after all ... It's also a magic black box for most users and would be difficult to work out failures. The safest and (in my opinion) most useful workflow is to back up your data locally to some kind of NAS or fileserver using plain old rsync and then back up that fileserver to rsync.net (or whomever) using the fancy borg tool. Now you have quick and simple local restores but still have a backup in the cloud that requires zero trust in the provider. [1] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/
- BrS96bVxXBLzf5B 5y ago
  > That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard. This is implied by the 'blindly' part. Searching "cloud storage provider", seeing rsync.net listed and picking it with a thrown dart would be blind. A quick search to see that it's been around for a while and doesn't have any crazy horror stories attached is part of becoming informed.
  - newscracker 5y ago
    In case you didn't notice or realize or recognize, GP is the founder of rsync.net (I'm not saying this to assign any ulterior motive for that comment).
mnemnc 5y ago
I feel like I have a fairly solid backup system for all of my important files. They are protected against elementary disasters and cyber attacks alike. However the only thing I struggle with are my phone photos. I recently caved in and started using iCloud Photos due to the convenience of having all my phone photos alwas available, searchable and tagged, even if the library size exceeds my phone‘s capacity. Does anyone know a reliable and automatable way to back up this iCloud photo library on a self hosted server?
- rolobio 5y ago
  icloud-photo-downloader: https://github.com/icloud-photos-downloader/icloud_photos_downloader https://github.com/icloud-photos-downloader/icloud_photos_do... I've used this to great success! As far as I could tell, it grabbed every photo and video at the highest quality. Very happy.
  - Bellyache5 5y ago
    Brilliant! Thank you.
- Bellyache5 5y ago
  I'm also interested in this. I have iCloud for Windows installed on a VM with lots of attached storage, but it doesn't seem to persistently/reliably download photos unattended. I suppose another way to do it would be with a Mac and then periodically backing up the local Photo Library, but that still leaves the photos tied up in Apple's proprietary library format. Plus you need a spare Mac just laying around and always on.
- wazoox 5y ago
  I don't know how it works on iPhones, but on Android Nextcloud does this just fine.
- poisonborz 5y ago
  I do this with Syncthing (not just photos, but all phone files), a very customizeable file sync tool, it can do trash bin and versioning similar to cloud storage providers. As I heard sadly it doesn't work so well in iOS due to its restrictions though.
dangerface 5y ago
> These tools work really great, until they don't! Huh? Whats the problem? How is ZFS any different because its lower level? > ZFS without ECC memory is no worse than any other file system without ECC memory. You really do need ECC memory unless you are ok with your pool becoming corrupt every 6 months or so. Im just saying this from my experience of running a ZFS server without ECC, the data wasn't critical so I left it like that for 3 years before replacing the memory with ECC now its fixed.
- 90minuteAPI 5y ago
  I was so tempted to jump on this with my anecdata of successfully running ZFS without ECC for a time, but that was in the context of a dirt cheap home tinker box. If you care about data integrity enough to use ZFS, just bite the bullet and use ECC. If you care about data integrity at all, use ECC. If you don’t, what are we even doing here?
- alpaca128 5y ago
  The bit about ZFS was a really flawed argument, saying it's just as complex but in an unavoidable way so that's fine for some reason: > One could argue that ZFS is complex as well, but that is on the filesystem level, a level on which you cannot avoid complexity no matter what you do.
- soupbowl 5y ago
  I stored 2tb of actively used data on zfs without ecc ram from freebsd 8 to 12. I had no noticeable corruption. I only post this because there is a weird assumption that not using ecc ram is a death sentence for your zfs data. I use ecc ram now and I think it's the proper way to do zfs, but let's not pretend you forfeit your data by not using ecc.
  - vermaden 5y ago
    ECC RAM is useful for ALL FILESYSTEMS - not just only for ZFS. But from all filesystems ZFS works best of all when you do not have ECC RAM. The 'requirement' that ZFS needs ECC RAM to work properly is one of the biggest myths of the Internet ...
bachmeier 5y ago
> Everything can look really nice "on paper" but you don't know what goes on behind the scenes. I have worked with a lot of different people and I have seen too much crazy shit to fully trust anyone with my important data. A cloud provider may have the best of intentions, but sometimes all it takes is a single grumpy employee or even a minor mistake to do a lot of damage. OneDrive and Google Drive are both pretty cheap. Is there anything wrong with keeping a backup of your important data in one of them? At a certain point you have to live your life and take a chance. Sure, I never made it to Italy, but I had a 100% safe backup system for my files said noone ever. > Free Git hosting such as GitHub, GitLab and others can also be utilized for data that you don't mind storing in public. GitLab and other providers does provide free private repositories, just don't rely fully on that. At this point, it's clear the author is looking for arguments to make. Of course you're not going to dump all your stuff into a GitLab repo in the cloud. You're going to clone it on multiple machines! My important work stuff is under version control and cloned on multiple machines in multiple locations. If that's not good enough, I'll live with the consequences.
- TchoBeer 5y ago
  Alternatively, "sure all my sensitive data was stolen and leaked, but at least I made it to Italy" said no one ever.
  - helmholtz 5y ago
    How common an occurence is that? How often is an unimportant, middle class person's data at risk, really? Enough that you'd want to spin up your ZFS storage? Hypothetically, let's say I had my entire life on Google. I have a unique password for it, backed up by 2FA, without the SMS/Authenticator fallback. What's the long term consequence? Google knows everything about me? They already do anyway. Someone can steal my printout of the backup codes? I don't ask this to stir shit. I genuinely have these sorts of discussions with friends and family when I try to tell them that privacy is important, and I fail absolutely at convincing them of it.
    - gurchik 5y ago
dmje 5y ago
I have a Synology NAS. All cloud files sync to it (Dropbox and Gdrive, the latter using "real" files, not just their .gdoc format), all backed up nightly to Synology C2 in the cloud, encrypted. Photos, which are in the "absolutely irreplaceable" category: all of the above but also have Mylio running so have external drives and 2x desktops with this data on them. Feels pretty safe, I think...?
- dheera 5y ago
  Having copies in at least 2 physical locations is important, because RAID/on-site backups won't help you if there's a fire/flood/theft that destroys everything. Back in school I used to have 2 Synology NAS, one in my dorm, one in my office, each running two drives with RAID 1. I had a gigabit link between my dorm and office (in 2008-2014!) thanks to MIT's awesome network. Now I live in Silicon Valley where even in 2021 I can only have a shitty 15 megabit Comcrap uplink so that setup doesn't work anymore. What I do now is a 4-bay Synology NAS. - The first 2 bays are RAID 1, they are the "live" volume - The 3rd bay is the "nightly backup" volume and syncs to the "live" volume every day late at night. That way if I accidentally delete something or mess up something I can look at yesterday's version at any time, very conveniently. - The 4th bay is usually empty, but once every 2-4 months I bring a drive from a storage closet in a different town, plug it in, sync it, and put it back in storage. Car is still faster than the internet in Silicon Valley. And that also allows me to have an airgapped backup, just in case there is ever a ransomware attack on Synology that destroys all the online drives.
  - dmje 5y ago
    Good advice, thanks. I actually have an unused Synology NAS which I moved from because it was slow. I wonder if I can use HyperBackup or similar and put this one down in my garden office. That might add another layer of "distance security"...
    - dheera 5y ago
      HyperBackup does this weird nonstandard "hbk" bullshit and I don't understand it, and I'm afraid of anything brand-specific. Between two Synology NAS I use Shared Folder Sync instead. Within the same NAS, sadly Shared Folder Sync doesn't work (even though there's no reason it shouldn't work) but you can set up an rsync task in the Task scheduler e.g. /bin/rsync -ap --delete --verbose /volume1/FOLDER/ /volume2/oldfiles/FOLDER/ >> /tmp/rsync.FOLDER.log and have it email the logs to you.
kumarsw 5y ago
I'll agree with the author partly on software such as rustic/duplicacy - they are not a good solution for long-term archiving, nor are they marketed as such. I keep a network share that I mirror the home folder on my computers to daily, but I take periodic snapshots of the share with duplicacy and copy the snapshots to Backblaze B2. The comments seem to be conflating Google Drive type cloud storage with more general S3/B2 object storage. I wouldn't rely on Google Drive and friends for backups - typically those have pretty short version histories as they are intended more for file synchonization with backup being an afterthought.
- deleted 5y ago
  [deleted]
- gurchik 5y ago
  > I'll agree with the author partly on software such as rustic/duplicacy - they are not a good solution for long-term archiving, nor are they marketed as such. I self host a Nextcloud server and I run duplicity each night to back this up to an offsite location. I use incremental backups because I have a data cap. From there each month I duplicate the most recent versions of the Duplicity directory onto a cold hard drive at the same location. I was surprised to see a couple people in this thread say to not rely on Duplicity for this. What could I be doing better?
  - kumarsw 5y ago
    My understanding is that there are two issues. First, modern deduplicating backup software like borg/restic/duplicacy store data in a repository in unique chunks. This avoids the issue that incremental backup software like duplicity have where they can create long chains of incremental changes which is slow to restore and increases the likelihood of errors on restore. Second, both deduplicating and incremental backup solutions aren't suggested for long-term archiving as they chop your files into lots of little pieces and the chances of not being able to read the repositories 10 years down the road are high. For that reason it's good to have a local backup in a simple, standard format like tar/zip or just a folder. As an example, see criticism of the Perkeep software [1] which is marketed as long term storage, but uses chunking deduplication for no particularly good reason. [1] https://perkeep.org https://perkeep.org
2OEH8eoCRo0 5y ago
Is it just me or is there quite a lot of fanaticism surrounding ZFS?
synergy20 5y ago
What's the best way to encrypt files, GunPG, openssl, ccrypt, some AES256 command line utility, or some backup can do it automatically? I do want to encrypt part of my backup before upload them to the cloud.
- dandanua 5y ago
  I don't know what's the best, but rclone looks like the simplest option https://rclone.org/crypt/ https://rclone.org/crypt/
Johnny555 5y ago
I do pretty much what he does, but use rsync - I rsync up my laptop twice daily to a ZFS fileserver, that fileserver is backed up to the cloud and rsync'ed to a networked hard drive located on the opposite side of the house from the fileserver. Every once in a while I swap out that hard drive with another one, so I have 2 copies of most of the data, with the most up to date data in the cloud. I used to use rsync to make incremental backups, but now I count on the cloud backup to keep versions. I use encfs to encrypt the backups (despite its security weaknesses, it's "good enough" to keep my data safe if a hard drive is lost/stolen). I don't worry about this: Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data. If I die, the only thing my family will care about is the pictures that are on a google photo album, no one is going to care about the RAW source images on the hard drive, no one is going to want to look at the mail archive from my job 10 years ago. My wife is certainly not going to maintain a DLNA server to watch videos that I ripped from disks, she's just going to load the original blu-ray/DVD. And not even I ever play any of the CD's that I spent hours ripping to MP3's - I just stream music from Spotify.
yosito 5y ago
I'm a digital nomad, and I decided I wanted to store all of my data as securely as possible without relying on Google/Amazon/Microsoft/Apple. Here's my setup, in case anyone finds it useful. I store all of my data in Nextcloud hosted on a VPS. It's virtually impossible to guarantee the security of data on a running VPS, so my sensitive data is also encrypted with Cryptomator, so I don't have to trust my VPS host (but I also chose my VPS host carefully with data privacy and security as my main criteria). My host makes daily backups of the VPS off-site, and I also backup my Nextcloud data directory daily to Tardigrade/Storj DCS via their s3 API using Restic. An advantage of Storj DCS is that it's geographically distributed, so you're insulated from natural disasters. I also sync all of my Nextcloud data to both of my laptops, and use Restic to make another backup snapshot on an external USB SSD once a month. My Docker configs for Nextcloud are stored on GitHub, but I wouldn't have trouble recreating them if I somehow lost them. For my own personal risk model, I think my sensitive data is pretty well protected from third parties, and I think all of my data is reliably backed up accounting for multiple types of failure. My biggest vulnerability is probably my password manager, where all of my encryption keys and passwords are stored. If anyone has any suggestions on how I can improve my setup, or any potential problems you see, I'd love it if you share!
- aborsy 5y ago
  I have heard of Cryptomator, but haven’t looked into its code. Has anyone looked into cryptography in Cryptomator? It’s a Java app, and uses Java Cryptography Extension JCE. But I found limited information about JCE. It was audited, but audits don’t mention some important aspects.
- tbronchain 5y ago
  Wondering something, if you are using cryptomator and a VPS - so you don't own your data and use a higher level encryption - why not using the cloud storage services (i.e Dropbox, icloud etc.) ?
  - onefuncman 5y ago
    I put my 1Password vault on Dropbox specifically because I can install both apps via iOS store on a new phone and bootstrap my setup with just 2 passwords. iCloud is actually too hard to recover in case of hardware loss, I don't recommend it to anyone.
buffalobuffalo 5y ago
I use dropbox, and a rsync script to a portable hard drive around once a month. I travel a lot, so i keep hard drives at several different places, which i think spreads the risk fairly well. The only reason I'm willing to sync my data to dropbox though is that it's all encrypted with cryfs. Works really well normally, although with an m1 chip it's been a bit of a pain.
yepthatsreality 5y ago
> Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data. Terrible advice regarding encryption. Really if you have important data that your family needs at the time of your death you should have a plan for that as well. Not avoid encryption because it’s “too hard”.
- hunter-gatherer 5y ago
  Exactly. Encrypt with a simple utility, and write down the password and store in a safe place and let someone know. In my case, I have a friend who is very technical and I can trust to assist my spouse (who is not technical) with some items if need be. Part of my hand-written instructions are his phone number/email, and to call him if there are issues decrypting or restoring data.
thom 5y ago
You can already build such a system yourself quite trivially by installing Dropbox.
tedcrilly 5y ago
I'm in the self-hosted camp as well with two Synology NAS boxes in two different countries that all of the family's computers synchronize to using Syncthing. Each of the boxes runs a local backup to an external drive nightly and one of them also runs a backup of a subset of (really important) folders to Backblaze. This uses Synology's proprietary backup tool (HyperBackup) which I intend to replace with an open source solution (most likely rclone but I'd be interested in suggestions). As an additional measure I rotate the external drives in one location weekly. So far it proved quite reliable when switching machines (in combination with git-managed dotfiles and stow) and accessing data on demand. I also make a full image of my laptop on an external drive more or less once a month to enable quick restore in case my OS gets hosed. One problem I still only semi-solved is synchronizing iPhones. Right now we just synchronize photos using Synology's DS File and I do an iTunes backup using a Windows VM which is clunky. High hopes for libimobiledevice here but I've had no time to properly research it yet. All of the above requires some work but it's fun more than a nuisance. Probably not that great of a solution when I'm no longer around though.
einpoklum 5y ago
I really like this guy's web page source. Do a source view, it's all very neat and tidy, you could just read that and not feel you're losing much in terms of experience.
theandrewbailey 5y ago
> While you might consider doing a full encryption for both your personal laptop and/or desktop, in case one of these gets stolen, you should avoid encryption on backup and storage when it really isn't needed because encryption adds yet another layer of complexity. Wrong. Not encrypting your backups when your daily use systems are encrypted makes no sense. Seriously. Why go through the hassle of figuring out a password, when one can steal the unencrypted off site backup? You want simple encryption? Me too! Use dm-crypt/LUKS. It's been in every modern Linux distro for at least 10 years. If you can plug your encrypted external drive into a live/freshly installed Linux desktop and a encryption prompt comes up, you're in luck!
- baobabKoodaa 5y ago
  I agree with your general sentiment, but I'll provide one redeeming counterpoint anyway: your laptop (that you carry around to places) is at higher risk of theft than an external hard drive that you keep in a storage locker.
simonblack 5y ago
Back up everything of yours at least once daily. Use a cronjob so that it won't matter if you can remember or not. I don't trust the 'Cloud' after several disasters involving DropBox when travelling out of the country. I treat my data like I treat my gold: "If it isn't under your own personal control, you don't own it at all". Store your files such that you are not backing up unnecessary stuff - such as the system itself because that's what system re-installs are for.(Apart from your various config files which are included in the daily backups) I divide my stuff into four main categories: config files and installed-package lists, databases, personal stuff, and "write once, keep forever" archives. The config-files and installed-package lists allow you to reinstall your system itself from scratch in less than an hour, in most cases. Databases are self explanatory. Dump the databases at least once daily, reload it when your system crashes. Personal /home directory stuff is restricted to personal documents and stuff that I am currently programming. That is it. Nothing else. What else would you need? That small amount is generally held down to less than 10 gigs. For all else, (ebooks, videos, photos, music, etc, etc, etc) there is the archive directory. That is stuff that needs to be kept, but isn't actually private and personal. The first three categories are included in my main daily backup as a full snapshot. The last category, the archive directory, is backed up as rsync changes. This goes to several separate backup media. On many days, there will be no changes at all and the rsync backup takes less than a couple of minutes.
bobbydreamer 5y ago
I got two 1TB harddrives Western digital and iOmega. WD scarred me, I had all my family photos in it, childhood, sister'marriage. WD usb pin socket is sort of weak, one time when i pulled the wire, the holder came with it. It was a scary situation. Finally repaired it. Moved it to cloud. That's also scary. Anybody knowing your email can lock you.

247 points by iio7 5y ago

I'm near a very similar setup. Syncthing for file replication, storage/backup to ZFS TrueNAS, then a secondary backup to another ZFS system that's usually offline. I wonder how OP does snapshot rotation.

tmslnz 5y ago

My approach these days: 1. Hot live data on my laptop. 2. Archive dir on NAS. It’s either 1 or 2, no overlap. Then: 1. Backup whole laptop to USB bootable drive “when I fancy”. 2. Backup home dir to NAS using duplicacy. This is accessible via SFTP, so it works no matter where I am. 3. Backup the Archive dir of the NAS to cheap cloud storage. Ideally I should add an external drive for the NAS to backup #3 locally, too.

renonce 5y ago

I put everything in a Resilio Sync folder, and keep a full sync on at least two devices (a home NAS and a cloud seedbox). Resilio Sync handles pretty much everything. You instantly get hot backup, have files immediately available on every device you have, and if you have a phone you can download any file on demand, etc. Unlike other file synchronization methods such as `rsync --delete`, it keeps a version for every file modification and moves the file to Archive when it is deleted, so you can't lose data. Also, you get encryption without the headache by using the "encrypted folder key". This syncs an encrypted copy so other devices can sync from that device.

wereHamster 5y ago
I use resilio sync together with a NAS (which is still large enough to hold all my files, I wonder how long that'll stay given how much 6k BRAW footage I'm shooting), and I also sync to a Google Cloud Storage bucket from my NAS (moving items to Archive storage class if they haven't been touched in a while, significantly saving cost).
deleted 5y ago
[deleted]

faeyanpiraat 5y ago

Excellent article. The method is simple but strong. The most important part is that it is actually implementable, examples and reasoning are provided for all steps.

jarenmf 5y ago

I use borg backup to sync all our files (from ~5 machines) to an old PC which synchronizes everything to the cloud. Borg has deduplication, compression, encryption, and saves several versions of the files which I think is crucial. Also all text-based data are on Git like software, papers, etc.

unilynx 5y ago
Is your PC synchronising the source files or the borg files to the cloud ? Otherwise it's still a SPOF for your backups (the borg files might corrupt, and you might be syncing that corruption to the cloud)
- jarenmf 5y ago
  Good point. I am indeed synchronizing the borg file. I have however my script run borg check before it synchronizes to the cloud. It would be better to synchronize the source files but the encryption and deduplication are what's keeping me from doing that.

ubermonkey 5y ago

It's a good discussion. I've taken a different path but with the same ideals. 1. All my live, working data is in Dropbox. I've been using my (paid) Dropbox account for over a decade. This step isn't really intended to be a backup per se, but you get the backup behavior "for free" because this is how I keep my secondary machine in sync with my primary. My Dropbox folder is also replicated to a 3rd computer in my home as another level of redundancy. 2. Everything in my home dir, especially photography, is ALSO covered by Backblaze. Having some backups elsewhere is mandatory; not enough people really understand how important this is. Should my house burn down, I still have data. 3. My primary system is a Mac, so I use Time Machine. TM is the only backup on this list I've ever actually used as a backup. When our home was robbed a number of years ago in a quickie smash-window-and-grab affair, they got my laptop. I went out and bought a new computer, plugged it into the TM drive, and in an hour or so I was right back where I left off. Hard to beat that. Even my app windows were in the same place. 4. Periodically, I take a full clone of my main machine's drive using a drive imaging tool (Mac specific; I use SuperDuper) and **store that drive at a friend's house**. This probably only happens a few times a year a this point. I should do it more often. That tertiary computer I mentioned in step 1 is also the home NAS server / home media server. It holds the photo archive in a large outboard disk. Backblaze covers that disk, and Time Machine on that computer keeps the outboard disk backed up as well. This data is mostly static, so the images I've taken of it and stored elsewhere don't need to be updated all that often (ie, just when I migrate prior year photo data to that drive).

malshe 5y ago
My set up is strikingly similar to yours! I keep a backup in my office instead of a friend’s home but my logic is the same as yours.

leotaku 5y ago

I currently trust Restic with basically all of my long-term backups, which, according to the author, really isn't a thing I should do. However, I'm still somewhat confident in my strategy as I backup my all of my data to two entirely different repositories, one of them backed by Google Cloud, and another by the server sitting in my pantry. So one of these repositories could get irrecoverably corrupted and I still wouldn't lose any of my data. With cloud storage becoming so cheap I've also thought about adding a third repo. Of course this would not protect me from a hypothetical bug in Restic that corrupts all my repositories before I notice, so maybe I should also add another auto-backup solution into the mix. Doing manual things like moving data to external storage seems like a robust strategy, but I really don't trust myself to do something like that nearly often enough for it to be useful.

wedowhatwedo 5y ago
I have used restic for quite a while. Once in a while, I test that I can restore my backups. That's an important step that lots of people miss. I had a client that asked me to setup their system. I setup the system, they got a tape drive and I had them rotate tapes daily. There was a cronjob to tar everything to tapes. It was great until their hard drive failed and I found I had a typo where it was only backing up the current folder, not the whole drive. Needless to say, they found a new IT provider and I learned an important lesson. If you haven't tested your backups, you have no backups.

ChrisMarshallNY 5y ago

Being an Apple chap, I have found that Time Machine works really well. I use a Synology 5-disk NAS. It has a ton of capacity, and allows me to increase capacity, if needed. My important stuff is in the cloud. I have a nightly "hot backup," that is a CC clone of my main and dev drives. It used to be every 4 hours, but I found that to be overkill. I almost never need anything more than the "hot backup."

visviva 5y ago

Darn, I was hoping this would be an article about organizing one's files. I was really in the mood for reading about that, then spending the rest of the morning reworking my own system.

kawfey 5y ago
Same. I get lost in researching everyone else's file organization and other workflow methods. I always feel like mine is a dog's breakfast, scattered across many incongruent, un-coordinated drives, clouds, shares, etc.
pricecomstock 5y ago
I have used this system in certain areas before. It's decent. But it's interesting to read about if you're in the mood! https://johnnydecimal.com/ https://johnnydecimal.com/
- visviva 5y ago
  That's what I'm talking about... and a wasted Friday afternoon is even better than a wasted Friday morning!
- tailspin2019 5y ago
  Oh man, that's a cool approach. Down the rabbit hole I go...
quacked 5y ago
This one is great too: https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
- olivertaylor 5y ago
  One of my favorites.

Gurdt 5y ago

Can someone point me to the flaws of my method? I simply use Google Drive. Me and my partner have multiple devices (tablet, iPhone, Android phone, multiple Mac and Windows laptops) and we just sync folders to our desktop. We just store everything we don't want to lose in Drive. We share folders we both need. Photo's we take on our phones are automagically backed up in the cloud, music and movies are streamed. My house can burn down overnight and we won't lose any valuable data.

CatAtHeart 5y ago
Although I'm not sure of the rate, people have reported getting banned from google services without notice for seemingly no apparent reason. And there is little recourse to unlocking their account. It's something I personally don't worry about, but something to consider when relying on their services for backing up data.
leephillips 5y ago
The flaw is that you depend on Google. You can not trust them at all, for anything. You can use Drive as a redundant fifth backup either for data that you’ve encrypted yourself or for things that you don’t mind being public, but not as the only place that has your data. Things that could very well happen: you get locked out with no recourse; people at Google browse your data; your data gets mined; your data gets lost; Google messes up and other people get access to your data; Google gets a subpoena and those photos that you thought were perfectly normal are now evidence of a manufactured crime.
mbiondi 5y ago
I do the same with Dropbox. I've been doing this for years and it's saved me a number of times. I don't know if there are flaws with this but it's worked better than anything I've cobbled together myself.
foobarian 5y ago
It should work fine until you get into the territory of having to store multiple terabytes of data, at which point cloud services can get expensive.
5y ago

mongol 5y ago

Something tricky is how to bootstrap restoration of backup. If you have lost "everything", how do you get it back? For example, if you use borg to backup remotely via ssh, you will need ssh keys as well as a passphrase for the encrypted backup. Where do you store those to make sure you have them if your computer is gone? What I did was create a self-extracting restoration script, which embedded everything needed. This is also encrypted, and synced to many places. The idea is, as long as I have the passphrase for that, it takes care of the rest.

beervirus 5y ago
For small stuff like this, print it out and put it in a safe deposit box.
- codesections 5y ago
  This would be the only reason I'd need to pay for a safe deposit box. It might be worth it, just for that, but I haven't made that leap yet.
  - reidjs 5y ago
    If you're on good terms with your family, it might make sense to leave it with them (if you trust them to keep it safe).
  - reaperducer 5y ago
    Check with your bank. If you have enough accounts, or the right kind of accounts, they may give you one for free. It's usually listed included in the long list of benefits that almost nobody ever reads. Each year I get a bill from a major American bank in the amount of $0 for a safe deposit box I don't use because I have a certain amount of money in one account.
- rovr138 5y ago
  I print my important keys to paper but also as a QR code. Easier to restore. Have't needed to use them yet, but testing it was great.

jl6 5y ago

Every good backup practitioner has an origin story. That one time they lost data to a clicking HDD, or to a scratched CD, or to a typo. The pain of never getting that data back. Galvanizing pain that catalyzed a resolution never to lose data again. Never trust someone (including yourself) to truly look after your data unless they know this pain first hand.

mirodin 5y ago
So poetic and so true!

deleted 5y ago

[deleted]

PragmaticPulp 5y ago

I also have a ZFS NAS and a separate backup server for the NAS that only gets turned on once a week. I disagree about his recommendation to avoid encryption. I keep the NAS and backup server drives encrypted. It’s not a big deal to unlock the drives at boot with a password and I’ve never had any issues with it. If someone breaks in and steals one of the servers, at least they won’t have access to years of all of my documents backed up. This author’s setup looks good, but I think most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved. A complex solution isn’t necessarily better if you find yourself not investing the time needed to maintain it. I think most people would be better off trusting one of the online backup services rather than spending thousands of dollars building and maintaining a pair of backup servers. If you’re worried about losing data, signing up for two separate backup services could still be cheaper than building and running two separate NAS servers. Start with easy backup solutions first to get something going. Then consider the more complicated options later.

sgarland 5y ago
> most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved. I didn't see the author's stated OS they use for their NAS, but depending on what it is, yes. {Free, True}NAS are dead easy to use as long as you stay inside the confines of the GUI, and hugely reliable. Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool! I have a setup very similar to the author's, with a bunch of Debian VMs on top of Proxmox. Can confirm that even with Packer and Ansible handling base images and configuration, weird edge cases pop up that take quite a while to track down.
- mbiondi 5y ago
  I thought the same about Free/True NAS until I lost a ton of stuff. Be careful with these. Also, for big backups the performance has been terrible. I carefully followed their guide and the write speed is just dismal.

optimalsolver 5y ago

teekert 5y ago
For your browser history, sure. But I for one make sure I rsync stuff to a Raspberry Pi4 with an external disk, at my parents place every now and then. In the event of my demise they can unplug it and have all our family pictures and videos. I'm really afraid that I make things to complicated for the people that I leave behind.
- adkadskhj 5y ago
  I dabble in "life storage" and your comment made me think that some sort of executable shipped alongside backup locations to read the data, if in some deduplicated backup form, seems valuable. Eg Camlistore/Perkeep had the premise of using JSON to store data. However some random person isn't going to write code to parse all that data, pull files out, etc. A lifeboat .exe might be interesting. Though doing it in the most simple, least configurable, least breakable way seems.. necessary. Yea some baked in UI would be cool, but more moving parts means less likely to work.
- mytdi 5y ago
  I have the same setup for the same reason as you. Only difference, I have my setup at home. As you say, in the event of my demise, I want my relatives to be able to grab the drive and have easy access to the pictures and videos.
scohesc 5y ago
If I can't delete my browsing history before I pass on, you bet your bottom dollar I'm not letting my family have my keys!
- avh02 5y ago

csabakissi 5y ago

My approach is 6 years old and I use Time Capsule. The best ever solution for me. You set it up and you don't case anymore. It's Apple specific unfortunately. Don't know if there is a similar solution for Windows.

diffeomorphism 5y ago
Just fyi those seem to be experiencing a data loss issue: https://appleinsider.com/articles/21/07/08/design-failure-in-apples-time-capsule-leads-to-data-loss https://appleinsider.com/articles/21/07/08/design-failure-in...

beervirus 5y ago

Does anybody use tape? I currently have a Synology RAID6 and I use a cloud backup service. But I’ve been thinking about adding another layer in the form of off-site tapes. Any recommendations welcome.

tomatocracy 5y ago
I have tapes as my last line of defence (ZFS snapshots, backups to NAS, online are my earlier lines of defence). The hardware itself is relatively simple but (at least in my experience - I tried quite a few options but they were all missing at least one key feature) there's no good "set and forget" software solution out there in the same way as there is for online backups so you're going to spend a long time scripting it all and tweaking it properly. You can build a solution on top of whichever other tools fit your needs that way. I have a set of scripts which create my backups in single large files on disk ready to be spooled out, then I use mbuffer to actually put the data onto tape. In terms of hardware, 1 or 2 generation old LTO drives are often available for a decent price in good condition on eBay. You'll also need either an SAS or Fibre Channel interface depending on what drive you buy. As far as I can tell they're pretty much all supported by Linux these days. Given the complexity of scripting and potential for user errors, testing that you can restore properly is pretty well essential, of course.

armoredkitten 5y ago

I've been using btrfs and btrbk for a few months now and really enjoy it. I love that snapshots are easy to make, and btrbk is a fairly simple way to schedule taking snapshots and transferring them to another machine, as well as set up a retention strategy. I agree with the author that simpler is better -- if I lose data/delete a file I want back, it's as simple as mounting the snapshot and grabbing what I need.

allenbina 5y ago

Regarding the issue he mentions with unmounting of drives on raspberry pis, I've found the issue that fixed it for me was a reliable power supply.

mdale 5y ago

I thought most people just get fast internet and let "the cloud" handle it now days ? Seems like a lot of work to ultimately not do things not as well as army of engineers focused on this problem full time.

kawfey 5y ago
Most people do (myself included, I just put everything on Backblaze or Google Drive), it's just that HN isn't most people.
allenbina 5y ago
I took the deep dive into tech trying to fix the 2x cd drive someone gave me for my x86 pc. Part of the passion I have for tech is learning and digging deeper, and reading about topics I haven't learned yet to the point where I get a bit of malaise when I'm not learning something new. I say this as I'm repurposing an old pc to a full time nas and learning about zfs pools. There's always a simpler, faster, more expensive solution out there that takes the complexity out of the equation, but part of the fun is the complexity for me.

anjuan 5y ago

For the past few years, I've created a photo book of the best family pictures from the past year. I store these photobooks in a fireproof safe. I think that's the easiest way to pass down pictures since it's paper doesn't require decryption or passwords.

codetrotter 5y ago
This summer I forgot the PIN of my iPad, because I didn’t use it during all summer. And it’s only 6 digits. I had, however, made note of the PIN elsewhere so I was able to unlock it. I have similarly forgotten PINs and passwords in the past that I didn’t use for a while. I recommend unlocking your safe at a regular interval, so that you remember the combination, if it uses a code lock. And even if it’s a key lock it’s probably a good idea to do it too, so you know where the key is.
- benhurmarcel 5y ago
  This was the reason I started using a password manager in the beginning, I didn’t trust my memory to remember them.

ga3h 5y ago

Thumb drives. Financial stuff, copies of legal documents, large libraries of PDF books, reports, etc., etc. And then back the main drive up to FOUR thumb drive backups. Reasons: (1) USB drives are huge now. (2) Convenient - can move from one PC to another easily. (3) Convenient - I do NOT have time to mess with other solutions. (4) Safety - valuable financial stuff almost always OFFLINE reducing hacker vulnerability. (5) All PCs eventually fail - I'm old and have a collection in my basement - plus I have two fairly new ones awaiting repair in my home office right now. Only disadvantage - I do have multiple legal copies of Microsoft Office on multiple PC to make moving thumb drives around practical.

shreddit 5y ago
Doesn’t thumbdrive data decay over time? Like all flash based storage types do
- gen3 5y ago
  That is my understanding. I've heard numbers around 10yr +-5, but I've never seen a study on it. I wonder if storing them zip-locked in a freezer would help (but I guess you could just use a magnetic hardrive :D )
  - mceachen 5y ago
    That the number I've personally experienced (I also replied to GP). It's an interesting idea to deep-freeze your drives, but the thermal stress from modern frost-free freezers that automatically dethaw periodically may negate the benefit of chilly bits... If you don't have that much to store, and are worried about data durability, I'd suggest looking into M-disc instead.
- dangerface 5y ago
  Yes they do. I used pen drives for backups after I found one that was like 10 years old and still had my old cv on it. I stored my backups on two identical pen drives I stopped doing this when I went to backup and discovered one of the pen drives was corrupt and lost all its data. They are super cheap and fairly reliable but not quite reliable enough for backups.

rsync 5y ago

"While I know of some really good cloud providers, such as rsync.net and Tarsnap, I recommend that you never trust cloud providers blindly." This is very good advice. That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard. ... "There exist some really cool open source backup solutions such as Borg, Restic and duplicity, but you should never rely solely on these "complex" solutions. These tools work really great, until they don't! In the past I have lost data to duplicity and other tools." I think this is very good advice as well - and that is coming from someone who has whole-heartedly endorsed 'borg' as a backup tool and regularly recommends it. It is the "holy grail of backups"[1] after all ... It's also a magic black box for most users and would be difficult to work out failures. The safest and (in my opinion) most useful workflow is to back up your data locally to some kind of NAS or fileserver using plain old rsync and then back up that fileserver to rsync.net (or whomever) using the fancy borg tool. Now you have quick and simple local restores but still have a backup in the cloud that requires zero trust in the provider. [1] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/

BrS96bVxXBLzf5B 5y ago
> That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard. This is implied by the 'blindly' part. Searching "cloud storage provider", seeing rsync.net listed and picking it with a thrown dart would be blind. A quick search to see that it's been around for a while and doesn't have any crazy horror stories attached is part of becoming informed.
- newscracker 5y ago
  In case you didn't notice or realize or recognize, GP is the founder of rsync.net (I'm not saying this to assign any ulterior motive for that comment).

mnemnc 5y ago

I feel like I have a fairly solid backup system for all of my important files. They are protected against elementary disasters and cyber attacks alike. However the only thing I struggle with are my phone photos. I recently caved in and started using iCloud Photos due to the convenience of having all my phone photos alwas available, searchable and tagged, even if the library size exceeds my phone‘s capacity. Does anyone know a reliable and automatable way to back up this iCloud photo library on a self hosted server?

rolobio 5y ago
icloud-photo-downloader: https://github.com/icloud-photos-downloader/icloud_photos_downloader https://github.com/icloud-photos-downloader/icloud_photos_do... I've used this to great success! As far as I could tell, it grabbed every photo and video at the highest quality. Very happy.
- Bellyache5 5y ago
  Brilliant! Thank you.
Bellyache5 5y ago
I'm also interested in this. I have iCloud for Windows installed on a VM with lots of attached storage, but it doesn't seem to persistently/reliably download photos unattended. I suppose another way to do it would be with a Mac and then periodically backing up the local Photo Library, but that still leaves the photos tied up in Apple's proprietary library format. Plus you need a spare Mac just laying around and always on.
wazoox 5y ago
I don't know how it works on iPhones, but on Android Nextcloud does this just fine.
poisonborz 5y ago
I do this with Syncthing (not just photos, but all phone files), a very customizeable file sync tool, it can do trash bin and versioning similar to cloud storage providers. As I heard sadly it doesn't work so well in iOS due to its restrictions though.

dangerface 5y ago

> These tools work really great, until they don't! Huh? Whats the problem? How is ZFS any different because its lower level? > ZFS without ECC memory is no worse than any other file system without ECC memory. You really do need ECC memory unless you are ok with your pool becoming corrupt every 6 months or so. Im just saying this from my experience of running a ZFS server without ECC, the data wasn't critical so I left it like that for 3 years before replacing the memory with ECC now its fixed.

90minuteAPI 5y ago
I was so tempted to jump on this with my anecdata of successfully running ZFS without ECC for a time, but that was in the context of a dirt cheap home tinker box. If you care about data integrity enough to use ZFS, just bite the bullet and use ECC. If you care about data integrity at all, use ECC. If you don’t, what are we even doing here?
alpaca128 5y ago
The bit about ZFS was a really flawed argument, saying it's just as complex but in an unavoidable way so that's fine for some reason: > One could argue that ZFS is complex as well, but that is on the filesystem level, a level on which you cannot avoid complexity no matter what you do.
soupbowl 5y ago
I stored 2tb of actively used data on zfs without ecc ram from freebsd 8 to 12. I had no noticeable corruption. I only post this because there is a weird assumption that not using ecc ram is a death sentence for your zfs data. I use ecc ram now and I think it's the proper way to do zfs, but let's not pretend you forfeit your data by not using ecc.
- vermaden 5y ago
  ECC RAM is useful for ALL FILESYSTEMS - not just only for ZFS. But from all filesystems ZFS works best of all when you do not have ECC RAM. The 'requirement' that ZFS needs ECC RAM to work properly is one of the biggest myths of the Internet ...

bachmeier 5y ago

> Everything can look really nice "on paper" but you don't know what goes on behind the scenes. I have worked with a lot of different people and I have seen too much crazy shit to fully trust anyone with my important data. A cloud provider may have the best of intentions, but sometimes all it takes is a single grumpy employee or even a minor mistake to do a lot of damage. OneDrive and Google Drive are both pretty cheap. Is there anything wrong with keeping a backup of your important data in one of them? At a certain point you have to live your life and take a chance. Sure, I never made it to Italy, but I had a 100% safe backup system for my files said noone ever. > Free Git hosting such as GitHub, GitLab and others can also be utilized for data that you don't mind storing in public. GitLab and other providers does provide free private repositories, just don't rely fully on that. At this point, it's clear the author is looking for arguments to make. Of course you're not going to dump all your stuff into a GitLab repo in the cloud. You're going to clone it on multiple machines! My important work stuff is under version control and cloned on multiple machines in multiple locations. If that's not good enough, I'll live with the consequences.

TchoBeer 5y ago
Alternatively, "sure all my sensitive data was stolen and leaked, but at least I made it to Italy" said no one ever.
- helmholtz 5y ago
  How common an occurence is that? How often is an unimportant, middle class person's data at risk, really? Enough that you'd want to spin up your ZFS storage? Hypothetically, let's say I had my entire life on Google. I have a unique password for it, backed up by 2FA, without the SMS/Authenticator fallback. What's the long term consequence? Google knows everything about me? They already do anyway. Someone can steal my printout of the backup codes? I don't ask this to stir shit. I genuinely have these sorts of discussions with friends and family when I try to tell them that privacy is important, and I fail absolutely at convincing them of it.
  - gurchik 5y ago

dmje 5y ago

I have a Synology NAS. All cloud files sync to it (Dropbox and Gdrive, the latter using "real" files, not just their .gdoc format), all backed up nightly to Synology C2 in the cloud, encrypted. Photos, which are in the "absolutely irreplaceable" category: all of the above but also have Mylio running so have external drives and 2x desktops with this data on them. Feels pretty safe, I think...?

dheera 5y ago
Having copies in at least 2 physical locations is important, because RAID/on-site backups won't help you if there's a fire/flood/theft that destroys everything. Back in school I used to have 2 Synology NAS, one in my dorm, one in my office, each running two drives with RAID 1. I had a gigabit link between my dorm and office (in 2008-2014!) thanks to MIT's awesome network. Now I live in Silicon Valley where even in 2021 I can only have a shitty 15 megabit Comcrap uplink so that setup doesn't work anymore. What I do now is a 4-bay Synology NAS. - The first 2 bays are RAID 1, they are the "live" volume - The 3rd bay is the "nightly backup" volume and syncs to the "live" volume every day late at night. That way if I accidentally delete something or mess up something I can look at yesterday's version at any time, very conveniently. - The 4th bay is usually empty, but once every 2-4 months I bring a drive from a storage closet in a different town, plug it in, sync it, and put it back in storage. Car is still faster than the internet in Silicon Valley. And that also allows me to have an airgapped backup, just in case there is ever a ransomware attack on Synology that destroys all the online drives.
- dmje 5y ago
  Good advice, thanks. I actually have an unused Synology NAS which I moved from because it was slow. I wonder if I can use HyperBackup or similar and put this one down in my garden office. That might add another layer of "distance security"...
  - dheera 5y ago
    HyperBackup does this weird nonstandard "hbk" bullshit and I don't understand it, and I'm afraid of anything brand-specific. Between two Synology NAS I use Shared Folder Sync instead. Within the same NAS, sadly Shared Folder Sync doesn't work (even though there's no reason it shouldn't work) but you can set up an rsync task in the Task scheduler e.g. /bin/rsync -ap --delete --verbose /volume1/FOLDER/ /volume2/oldfiles/FOLDER/ >> /tmp/rsync.FOLDER.log and have it email the logs to you.

kumarsw 5y ago

I'll agree with the author partly on software such as rustic/duplicacy - they are not a good solution for long-term archiving, nor are they marketed as such. I keep a network share that I mirror the home folder on my computers to daily, but I take periodic snapshots of the share with duplicacy and copy the snapshots to Backblaze B2. The comments seem to be conflating Google Drive type cloud storage with more general S3/B2 object storage. I wouldn't rely on Google Drive and friends for backups - typically those have pretty short version histories as they are intended more for file synchonization with backup being an afterthought.

deleted 5y ago
[deleted]
gurchik 5y ago
> I'll agree with the author partly on software such as rustic/duplicacy - they are not a good solution for long-term archiving, nor are they marketed as such. I self host a Nextcloud server and I run duplicity each night to back this up to an offsite location. I use incremental backups because I have a data cap. From there each month I duplicate the most recent versions of the Duplicity directory onto a cold hard drive at the same location. I was surprised to see a couple people in this thread say to not rely on Duplicity for this. What could I be doing better?
- kumarsw 5y ago
  My understanding is that there are two issues. First, modern deduplicating backup software like borg/restic/duplicacy store data in a repository in unique chunks. This avoids the issue that incremental backup software like duplicity have where they can create long chains of incremental changes which is slow to restore and increases the likelihood of errors on restore. Second, both deduplicating and incremental backup solutions aren't suggested for long-term archiving as they chop your files into lots of little pieces and the chances of not being able to read the repositories 10 years down the road are high. For that reason it's good to have a local backup in a simple, standard format like tar/zip or just a folder. As an example, see criticism of the Perkeep software [1] which is marketed as long term storage, but uses chunking deduplication for no particularly good reason. [1] https://perkeep.org https://perkeep.org

2OEH8eoCRo0 5y ago

Is it just me or is there quite a lot of fanaticism surrounding ZFS?

synergy20 5y ago

What's the best way to encrypt files, GunPG, openssl, ccrypt, some AES256 command line utility, or some backup can do it automatically? I do want to encrypt part of my backup before upload them to the cloud.

dandanua 5y ago
I don't know what's the best, but rclone looks like the simplest option https://rclone.org/crypt/ https://rclone.org/crypt/

Johnny555 5y ago

I do pretty much what he does, but use rsync - I rsync up my laptop twice daily to a ZFS fileserver, that fileserver is backed up to the cloud and rsync'ed to a networked hard drive located on the opposite side of the house from the fileserver. Every once in a while I swap out that hard drive with another one, so I have 2 copies of most of the data, with the most up to date data in the cloud. I used to use rsync to make incremental backups, but now I count on the cloud backup to keep versions. I use encfs to encrypt the backups (despite its security weaknesses, it's "good enough" to keep my data safe if a hard drive is lost/stolen). I don't worry about this: Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data. If I die, the only thing my family will care about is the pictures that are on a google photo album, no one is going to care about the RAW source images on the hard drive, no one is going to want to look at the mail archive from my job 10 years ago. My wife is certainly not going to maintain a DLNA server to watch videos that I ripped from disks, she's just going to load the original blu-ray/DVD. And not even I ever play any of the CD's that I spent hours ripping to MP3's - I just stream music from Spotify.

yosito 5y ago

I'm a digital nomad, and I decided I wanted to store all of my data as securely as possible without relying on Google/Amazon/Microsoft/Apple. Here's my setup, in case anyone finds it useful. I store all of my data in Nextcloud hosted on a VPS. It's virtually impossible to guarantee the security of data on a running VPS, so my sensitive data is also encrypted with Cryptomator, so I don't have to trust my VPS host (but I also chose my VPS host carefully with data privacy and security as my main criteria). My host makes daily backups of the VPS off-site, and I also backup my Nextcloud data directory daily to Tardigrade/Storj DCS via their s3 API using Restic. An advantage of Storj DCS is that it's geographically distributed, so you're insulated from natural disasters. I also sync all of my Nextcloud data to both of my laptops, and use Restic to make another backup snapshot on an external USB SSD once a month. My Docker configs for Nextcloud are stored on GitHub, but I wouldn't have trouble recreating them if I somehow lost them. For my own personal risk model, I think my sensitive data is pretty well protected from third parties, and I think all of my data is reliably backed up accounting for multiple types of failure. My biggest vulnerability is probably my password manager, where all of my encryption keys and passwords are stored. If anyone has any suggestions on how I can improve my setup, or any potential problems you see, I'd love it if you share!

aborsy 5y ago
I have heard of Cryptomator, but haven’t looked into its code. Has anyone looked into cryptography in Cryptomator? It’s a Java app, and uses Java Cryptography Extension JCE. But I found limited information about JCE. It was audited, but audits don’t mention some important aspects.
tbronchain 5y ago
Wondering something, if you are using cryptomator and a VPS - so you don't own your data and use a higher level encryption - why not using the cloud storage services (i.e Dropbox, icloud etc.) ?
- onefuncman 5y ago
  I put my 1Password vault on Dropbox specifically because I can install both apps via iOS store on a new phone and bootstrap my setup with just 2 passwords. iCloud is actually too hard to recover in case of hardware loss, I don't recommend it to anyone.

buffalobuffalo 5y ago

I use dropbox, and a rsync script to a portable hard drive around once a month. I travel a lot, so i keep hard drives at several different places, which i think spreads the risk fairly well. The only reason I'm willing to sync my data to dropbox though is that it's all encrypted with cryfs. Works really well normally, although with an m1 chip it's been a bit of a pain.

yepthatsreality 5y ago

> Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data. Terrible advice regarding encryption. Really if you have important data that your family needs at the time of your death you should have a plan for that as well. Not avoid encryption because it’s “too hard”.

hunter-gatherer 5y ago
Exactly. Encrypt with a simple utility, and write down the password and store in a safe place and let someone know. In my case, I have a friend who is very technical and I can trust to assist my spouse (who is not technical) with some items if need be. Part of my hand-written instructions are his phone number/email, and to call him if there are issues decrypting or restoring data.

thom 5y ago

You can already build such a system yourself quite trivially by installing Dropbox.

tedcrilly 5y ago

I'm in the self-hosted camp as well with two Synology NAS boxes in two different countries that all of the family's computers synchronize to using Syncthing. Each of the boxes runs a local backup to an external drive nightly and one of them also runs a backup of a subset of (really important) folders to Backblaze. This uses Synology's proprietary backup tool (HyperBackup) which I intend to replace with an open source solution (most likely rclone but I'd be interested in suggestions). As an additional measure I rotate the external drives in one location weekly. So far it proved quite reliable when switching machines (in combination with git-managed dotfiles and stow) and accessing data on demand. I also make a full image of my laptop on an external drive more or less once a month to enable quick restore in case my OS gets hosed. One problem I still only semi-solved is synchronizing iPhones. Right now we just synchronize photos using Synology's DS File and I do an iTunes backup using a Windows VM which is clunky. High hopes for libimobiledevice here but I've had no time to properly research it yet. All of the above requires some work but it's fun more than a nuisance. Probably not that great of a solution when I'm no longer around though.

einpoklum 5y ago

I really like this guy's web page source. Do a source view, it's all very neat and tidy, you could just read that and not feel you're losing much in terms of experience.

theandrewbailey 5y ago

> While you might consider doing a full encryption for both your personal laptop and/or desktop, in case one of these gets stolen, you should avoid encryption on backup and storage when it really isn't needed because encryption adds yet another layer of complexity. Wrong. Not encrypting your backups when your daily use systems are encrypted makes no sense. Seriously. Why go through the hassle of figuring out a password, when one can steal the unencrypted off site backup? You want simple encryption? Me too! Use dm-crypt/LUKS. It's been in every modern Linux distro for at least 10 years. If you can plug your encrypted external drive into a live/freshly installed Linux desktop and a encryption prompt comes up, you're in luck!

baobabKoodaa 5y ago
I agree with your general sentiment, but I'll provide one redeeming counterpoint anyway: your laptop (that you carry around to places) is at higher risk of theft than an external hard drive that you keep in a storage locker.

simonblack 5y ago

Back up everything of yours at least once daily. Use a cronjob so that it won't matter if you can remember or not. I don't trust the 'Cloud' after several disasters involving DropBox when travelling out of the country. I treat my data like I treat my gold: "If it isn't under your own personal control, you don't own it at all". Store your files such that you are not backing up unnecessary stuff - such as the system itself because that's what system re-installs are for.(Apart from your various config files which are included in the daily backups) I divide my stuff into four main categories: config files and installed-package lists, databases, personal stuff, and "write once, keep forever" archives. The config-files and installed-package lists allow you to reinstall your system itself from scratch in less than an hour, in most cases. Databases are self explanatory. Dump the databases at least once daily, reload it when your system crashes. Personal /home directory stuff is restricted to personal documents and stuff that I am currently programming. That is it. Nothing else. What else would you need? That small amount is generally held down to less than 10 gigs. For all else, (ebooks, videos, photos, music, etc, etc, etc) there is the archive directory. That is stuff that needs to be kept, but isn't actually private and personal. The first three categories are included in my main daily backup as a full snapshot. The last category, the archive directory, is backed up as rsync changes. This goes to several separate backup media. On many days, there will be no changes at all and the rsync backup takes less than a couple of minutes.

bobbydreamer 5y ago

I got two 1TB harddrives Western digital and iOmega. WD scarred me, I had all my family photos in it, childhood, sister'marriage. WD usb pin socket is sort of weak, one time when i pulled the wire, the holder came with it. It was a scary situation. Finally repaired it. Moved it to cloud. That's also scary. Anybody knowing your email can lock you.