10 ms·

Per charts in TFA, it looks like some disks are failing less overall, and failing after a longer period of time. I'm still not sure how to confidently store de

Per charts in TFA, it looks like some disks are failing less overall, and failing after a longer period of time. I'm still not sure how to confidently store decent amounts of (personal) data for over 5 years without 1- giving to cloud, 2- burning to M-disk, or 3- replacing multiple HDD every 5 years on average All whilst regularly checking for bitrot and not overwriting good files with bad corrupted files. Who has the easy, self-service, cost-effective solution for basic, durable file storage? Synology? TrueNAS? Debian? UGreen? (1) and (2) both have their annoyances, so (3) seems "best" still, but seems "too complex" for most? I'd consider myself pretty technical, and I'd say (3) presents real challenges if I don't want it to become a somewhat significant hobby.

LunaSea 1y ago
Would tapes not be an option? Not great for easy read access but other than that it might be decent storage.
- dmoy 1y ago
  I looked at tape a little while ago and decided it wasn't gonna work out for me reliability-wise at home without a more controlled environment (especially humidity).
- gruez 1y ago
  >Would tapes not be an option? AFAIK someone on reddit did the math and the break-even for tapes is between 50TB to 100TB. Any less and it's cheaper to get a bunch of hard drives.
  - ghaff 1y ago
    Unless you're basically a serious data hoarder or otherwise have unusual storage requirements, an 18TB drive (or maybe 2) get you a lot of the way to handling most normal home requirements.
    - t-3 1y ago
      Personally, I buy the drives with the best $/storage ratio. Right now that seems to be ~3-6TB drives. Many PC enclosures and motherboards can fit 8-12 drives, fill it up with the cheapest stuff you're willing to spend money on. It will probably break even or be cheaper than the larger drives.
      - ghaff 1y ago
        It depends on the use case. As with CPUs, I tend not to buy the top-end but it may make sense to just buy for expansion over time. I think my RAID-1 Synology drives are 8TB. But mostly just use external enclosures these days anyway. Pretty much don't build PCs any longer.
- thisislife2 1y ago
  Tapes would be great for backups - but the tape drive market's all "enterprise-y", and the pricing reflects that. There really isn't any affordable retail consumer option (which is surprising as there definitely is a market for it).
- mapkkk 1y ago
  I don't know why you were downvoted, I think for the right purpose tape drives are great. Used drives from a few generations back work just fine, and are affordable. I have an LTO-5 drive, and new tapes are around $30 where I am. One tape holds 1,5TB uncompressed. I think they are great for critical data. I have documents and photos on them.
  - jandrese 1y ago
    Is there not a problem where companies stop manufacturing the tape media because it is obsolete?
    - mapkkk 1y ago
      I'm not 100% up to speed with the current standing of things, but tapes (specifically the LTO technology) is still being relied on very heavily by the enterprise, both in data centers for things like cold storage or critical backups, and other corporate uses. Archival use is also very strong with libraries and other such institutions having large tape libraries with autoloaders and all that automation jazz. The LTO-5 generation I mentioned was released in 2010, and the first LTO generation was released in 2000 I believe. The current generation is LTO-10, with an uncompressed capacity of 30TB. New LTO tapes are still being produced, the last batch I purchased was made in 2023. The LTO consortium consists of HP, IBM and one other company I believe. Now, in my opinion, none of this guarantees the longevity of the medium any more than any other medium, but when I initially looked into it, this was enough to convince me to buy a drive and a couple of tapes. My reasoning was that with the advertised longevity of 30 years under "ideal archival conditions", if I can get 10 years of mileage from tapes that are just sitting on my non-environmentally-controlled shelf, that means I'll only have to hunt down new tapes 3 times in my remaining lifetime, and after that it will be someone else's problem.
palmotea 1y ago
> 2- burning to M-disk, or You can't buy those anymore. I've tried. IIRC, the things currently marketed as MDisc are just regular BD-R discs (perhaps made to a higher standard, and maybe with a slower write speed programmed into them, but still regular BD-Rs).
Gigachad 1y ago
Hard drive failure seems like more of a cost and annoyance problem than a data preservation issue. Even with incredible reliability you still need backups if your house burns down. And if you have a backup system then drive failure matters little.
ssl-3 1y ago
One method that seems appealing: 1. Use ZFS with raidz 2. Scrub regularly to catch the bitrot 3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.) 4. Bring over pizza or something from time to time. As to brands: This method is independent of brand or distro.
- fun444555 1y ago
  This works great although I should really do step 4 :)
- gruez 1y ago
  >3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.) Unless you're storing terabyte levels of data, surely it's more straightforward and more reliable to store on backblaze or aws glacier? The only advantage of the DIY solution is if you value your time at zero and/or want to "homelab".
  - ghaff 1y ago
    I'd much rather just have a backblaze solution and maybe redundant local backups with Time Machine or your local backup of choice (which work fine for terabytes at this point). Maybe create a clone data drive and drop it off with a friend every now and then which should capture most important archive stuff.
  - ssl-3 1y ago
    A chief advantage of storing backup data across town is that a person can just head over and get it (or ideally, a copy of it) in the unlikely event that it becomes necessary to recover from a local disaster that wasn't handled by raidz and local snapshots. The time required to set this stuff up is...not very big. Things like ZFS and Tailscale may sound daunting, but they're very light processes on even the most garbage-tier levels of vaguely-modern PC hardware and are simple to get working.
toast0 1y ago
If you don't have too much stuff, you could probably do ok with mirroring across N+1 (distributed) disks, where N is enough that you're comfortable. Monitor for failure/pre-failure indicators and replace promptly. When building up initially, make a point of trying to stagger purchases and service entry dates. After that, chances are failures will be staggered as well, so you naturally get staggered service entry dates. You can likely hit better than 5 year time in service if you run until failure, and don't accumulate much additional storage. But I just did a 5 year replacement, so I dunno. Not a whole lot of work to replace disks that work.
IgorPartola 1y ago
Get yourself a Xeon powered workstation that supports at least 4 drives. One will be your boot system drive and three or more will be a ZFS mirror. You will use ECC RAM (hence Xeon). I bought a Lenovo workstation like this for $35 on eBay. ZFS with a three way mirror will be incredibly unlikely to fail. You only need one drive for your data to survive. Then get a second setup exactly like this for your backup server. I use rsnapshot for that. For your third copy you can use S3 like a block device, which means you can use an encrypted file system. Use FreeBSD for your base OS.
- bjoli 1y ago
  For am4 it is pretty easy to get things working with ecc udimms. I have a 5900x with 32gb of ddr4 ecc ram ticking in my basement.
Dylan16807 1y ago
I don't understand what you're worried about with 3. Make a box, hide it in a closet with power, every 3 months look at your drive stats to see if any have a buch of uncorrectable errors. If we estimate half an hour per checkup and one hour per replacement that's under three hours per year to maintain your data.
20after4 1y ago
Offline data storage is a good option for files you don't need to access constantly. A hard drive sitting on a shelf in a good environment (not much humidity, reasonable temperature, not a lot of vibration) will last a very very long time. The same can't be said for SSDs which will lose their stored data in a mater of a year or two.

LunaSea 1y ago

Would tapes not be an option? Not great for easy read access but other than that it might be decent storage.

dmoy 1y ago
I looked at tape a little while ago and decided it wasn't gonna work out for me reliability-wise at home without a more controlled environment (especially humidity).
gruez 1y ago
>Would tapes not be an option? AFAIK someone on reddit did the math and the break-even for tapes is between 50TB to 100TB. Any less and it's cheaper to get a bunch of hard drives.
- ghaff 1y ago
  Unless you're basically a serious data hoarder or otherwise have unusual storage requirements, an 18TB drive (or maybe 2) get you a lot of the way to handling most normal home requirements.
  - t-3 1y ago
    Personally, I buy the drives with the best $/storage ratio. Right now that seems to be ~3-6TB drives. Many PC enclosures and motherboards can fit 8-12 drives, fill it up with the cheapest stuff you're willing to spend money on. It will probably break even or be cheaper than the larger drives.
    - ghaff 1y ago
      It depends on the use case. As with CPUs, I tend not to buy the top-end but it may make sense to just buy for expansion over time. I think my RAID-1 Synology drives are 8TB. But mostly just use external enclosures these days anyway. Pretty much don't build PCs any longer.
thisislife2 1y ago
Tapes would be great for backups - but the tape drive market's all "enterprise-y", and the pricing reflects that. There really isn't any affordable retail consumer option (which is surprising as there definitely is a market for it).
mapkkk 1y ago
I don't know why you were downvoted, I think for the right purpose tape drives are great. Used drives from a few generations back work just fine, and are affordable. I have an LTO-5 drive, and new tapes are around $30 where I am. One tape holds 1,5TB uncompressed. I think they are great for critical data. I have documents and photos on them.
- jandrese 1y ago
  Is there not a problem where companies stop manufacturing the tape media because it is obsolete?
  - mapkkk 1y ago
    I'm not 100% up to speed with the current standing of things, but tapes (specifically the LTO technology) is still being relied on very heavily by the enterprise, both in data centers for things like cold storage or critical backups, and other corporate uses. Archival use is also very strong with libraries and other such institutions having large tape libraries with autoloaders and all that automation jazz. The LTO-5 generation I mentioned was released in 2010, and the first LTO generation was released in 2000 I believe. The current generation is LTO-10, with an uncompressed capacity of 30TB. New LTO tapes are still being produced, the last batch I purchased was made in 2023. The LTO consortium consists of HP, IBM and one other company I believe. Now, in my opinion, none of this guarantees the longevity of the medium any more than any other medium, but when I initially looked into it, this was enough to convince me to buy a drive and a couple of tapes. My reasoning was that with the advertised longevity of 30 years under "ideal archival conditions", if I can get 10 years of mileage from tapes that are just sitting on my non-environmentally-controlled shelf, that means I'll only have to hunt down new tapes 3 times in my remaining lifetime, and after that it will be someone else's problem.

palmotea 1y ago

> 2- burning to M-disk, or You can't buy those anymore. I've tried. IIRC, the things currently marketed as MDisc are just regular BD-R discs (perhaps made to a higher standard, and maybe with a slower write speed programmed into them, but still regular BD-Rs).

Gigachad 1y ago

Hard drive failure seems like more of a cost and annoyance problem than a data preservation issue. Even with incredible reliability you still need backups if your house burns down. And if you have a backup system then drive failure matters little.

ssl-3 1y ago

One method that seems appealing: 1. Use ZFS with raidz 2. Scrub regularly to catch the bitrot 3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.) 4. Bring over pizza or something from time to time. As to brands: This method is independent of brand or distro.

fun444555 1y ago
This works great although I should really do step 4 :)
gruez 1y ago
>3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.) Unless you're storing terabyte levels of data, surely it's more straightforward and more reliable to store on backblaze or aws glacier? The only advantage of the DIY solution is if you value your time at zero and/or want to "homelab".
- ghaff 1y ago
  I'd much rather just have a backblaze solution and maybe redundant local backups with Time Machine or your local backup of choice (which work fine for terabytes at this point). Maybe create a clone data drive and drop it off with a friend every now and then which should capture most important archive stuff.
- ssl-3 1y ago
  A chief advantage of storing backup data across town is that a person can just head over and get it (or ideally, a copy of it) in the unlikely event that it becomes necessary to recover from a local disaster that wasn't handled by raidz and local snapshots. The time required to set this stuff up is...not very big. Things like ZFS and Tailscale may sound daunting, but they're very light processes on even the most garbage-tier levels of vaguely-modern PC hardware and are simple to get working.

toast0 1y ago

If you don't have too much stuff, you could probably do ok with mirroring across N+1 (distributed) disks, where N is enough that you're comfortable. Monitor for failure/pre-failure indicators and replace promptly. When building up initially, make a point of trying to stagger purchases and service entry dates. After that, chances are failures will be staggered as well, so you naturally get staggered service entry dates. You can likely hit better than 5 year time in service if you run until failure, and don't accumulate much additional storage. But I just did a 5 year replacement, so I dunno. Not a whole lot of work to replace disks that work.

IgorPartola 1y ago

Get yourself a Xeon powered workstation that supports at least 4 drives. One will be your boot system drive and three or more will be a ZFS mirror. You will use ECC RAM (hence Xeon). I bought a Lenovo workstation like this for $35 on eBay. ZFS with a three way mirror will be incredibly unlikely to fail. You only need one drive for your data to survive. Then get a second setup exactly like this for your backup server. I use rsnapshot for that. For your third copy you can use S3 like a block device, which means you can use an encrypted file system. Use FreeBSD for your base OS.

bjoli 1y ago
For am4 it is pretty easy to get things working with ecc udimms. I have a 5900x with 32gb of ddr4 ecc ram ticking in my basement.

Dylan16807 1y ago

I don't understand what you're worried about with 3. Make a box, hide it in a closet with power, every 3 months look at your drive stats to see if any have a buch of uncorrectable errors. If we estimate half an hour per checkup and one hour per replacement that's under three hours per year to maintain your data.

20after4 1y ago

Offline data storage is a good option for files you don't need to access constantly. A hard drive sitting on a shelf in a good environment (not much humidity, reasonable temperature, not a lot of vibration) will last a very very long time. The same can't be said for SSDs which will lose their stored data in a mater of a year or two.