7 ms·

Note that it doesn't look like it has ECC, so make sure to have backups. Fancy file systems like ZFS don't remove the need for ECC.

coconut08 2y ago
been using zfs on my home nas without ecc for well over a decade and never had any problems. i've seen people claiming this since before i started using zfs and it seems so unnecessary for some random home project.
- EvanAnderson 2y ago
  Unless you've verified hashes of your files over time you may be having problems and not realizing it.
  - theshrike79 2y ago
    If a single byte flips in a 4-10GB video file, nobody will ever notice it. There aren't that many cases where it actually matters.
  - loeg 2y ago
    I believe ZFS does periodic checksuming (scrubbing).
    - yjftsjthsd-h 2y ago
      Strictly speaking I don't think ZFS itself does, but it is very common for distros to ship a cronjob that runs `zpool scrub` on a schedule (often but not always default enabled).
  - ratboy666 2y ago
    They did mention ZFS, so verified hashes of each file block. I hope they are scrubbing, and have at least one snapshot.
    - mmh0000 2y ago
      ZFS does nothing to protect you against RAM corrupting your data before ZFS sees it. All you'll end up with is a valid checksum of the now bad data. You can Google more, but, I'll just leave this from the first page of the openZFS manual: Misinformation has been circulated that ZFS data integrity features are somehow worse than those of other filesystems when ECC RAM is not used. This is not the case: all software needs ECC RAM for reliable operation and ZFS is no different from any other filesystem in that regard.[1] [1] https://openzfs.readthedocs.io/en/latest/introduction.html https://openzfs.readthedocs.io/en/latest/introduction.html
    - mafro 2y ago
      Why would one snapshot help?
      - ratboy666 2y ago
        One snapshot would help because, if EVERYTHING collapses, and you need data recovery, the snapshot provides a basepoint for the recovery. This should allow better recovery of metadata. Not that this should EVER happen -- it is just a good idea. I use Jim Salter's syncoid/sanoid to make snapshots, age them out, and send data to another pool. I agree that ECC is a damn good idea - I use it on my home server. But, my lappy (i5 thinkpad) doesn't have it.
  - coconut08 2y ago
    i've heard people say this, like i said, since before i started using zfs and i've never had an issue with a corrupted file. there's a few things that could be happening: i'm the luckiest person who has ever lived, these bit flip events don't happen nearly as often as people like to pretend they do, or when they do happen they aren't likely to be a big deal.
    - EvanAnderson 2y ago
      I have some JPEGs with bit flips. I could tell because they display ugly artifacts at the point of the bit flip. (You can see the kind of artifacts I'm talking about here: https://xn--andreasvlker-cjb.de/2024/02/28/image-formats-bitflips/ https://xn--andreasvlker-cjb.de/2024/02/28/image-formats-bit...) I'd happened to archive the files to CD-R's incidentally. I was able to compare those archived files to the ones that remained on my file server. There were bit flips randomly in some of the files. After that happened I started hashing all of my files and comparing hashes when I migrate files during server upgrades. Prior to using ZFS I also periodically verified file hashes with a cheapo Perl script.
    - iforgotpassword 2y ago
      If all you have on your Nas is pirated movies, then yes > when they do happen they aren't likely to be a big deal. But with more sensitive data it might matter to you. Ram can go bad like hdds can, and without ecc you have no chance of telling. Zfs won't help you here if the bit flip happens in the page cache. The file will corrupt in ram and Zfs will happily calculate a checksum for that corrupted data and store that alongside the file.
jnovek 2y ago
> so make sure to have backups Can you (or someone) suggest a backup scheme? I have a 28TB NAS. Almost everything I've looked into is expensive or intended more for enterprise tier. Are there options for backup in the "hobbyist" price range?
- Dxtros 2y ago
  if your talking cloud backup Wasabi (which uses S3) is the cheapest i could find it’s pay as you go and they don’t charge for upload/download. The pay as you go is $6.99 per TB which would be pretty pricey at 28 TB, but it’s super cheap for my 4tb NAS.
- vunderba 2y ago
  I have a NAS that has 18 TB effective storage, 36 TB mirrored. It all gets backed up to a B2 back blaze which is about six dollars per terabyte - but I'm currently only using about 8 TB at the moment so it's only about 50 bucks a month. So this might be on the higher end of the price range if you're using up all 28 TB uncompressed since that's about $168 per month though...
- Dylan16807 2y ago
  AWS, GCP, and Azure all offer cold storage for about $1 per TB per month. If you want any cheaper you need to build a second NAS. You could also take the awkward route and add one or two large drives to your desktop, mirror there, and back that up to backblaze (not B2). The other suggestions you got for hot storage strike me as the wrong way to handle this, if you're considering $80 per year per TB for backups then just make another NAS.
  - vunderba 2y ago
    For the OP - be careful with AWS, the closest pricing to one dollar per terabyte is S3 Glacier Deep Archive and you'd be surprised how expensive a full restore can be in the event that you need to do so in terms of restore pricing, egress cost, etc. Another NAS isn't really a good solution (unless you can place it in a different house) - the goal of a cloud back up is that it's offsite.
hitsurume 2y ago
I know ECC is a special type of ram, but how does it help a NAS/Raid setup?
- AlexandrB 2y ago
  Data that's about to be written to disk often resides in ram for some period of time - bit flips in non-ECC ram can silently corrupt the data before writing it out. ZFS doesn't prevent this though it might detect it with checksumming. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
- eric__cartman 2y ago
  If you're unlucky enough to experience memory errors in one of the intermediate buffers files go through while being copied from one computer to another an incorrect copy of the file might get written to disk. When running software RAID, memory errors could also cause data to be replicated erroneously and raise an error the next time it's read. That said if the memory is flaky enough that these errors are common it's highly likely that the operating system will crash very frequently and the user will know something is seriously wrong. If you want to make sure that files have been copied correctly you can flush all kernel buffers and run diff -r between the source and destination directory to make sure that everything is the same. It's probably way more likely to experience data loss due to human error or external factors such as a power surge than bad ram. I personally thoroughly test the memory before a computer gets put into service and assume it's okay until something fails or it gets replaced. The only machine I've ever seen that would corrupt random data on a disk was heavily and carelessly overclocked (teenage me cared about getting moar fps in games, and not having a reliable workstation lol)
  - barnabee 2y ago
    I wonder whether something like Syncthing would notice a hash difference with data corruption caused by such a memory error? And whether it’d correct it or propagate the issue…
theshrike79 2y ago
I've had non-ECC NAS systems for over 20 years and I've had exactly zero cases where memory corruption was an issue. It's OK for corporate systems, but complete overkill for personal setups.
- jjav 2y ago
  > It's OK for corporate systems, but complete overkill for personal setups. My personal files are ultimately a lot more important to me and much more irreplaceable than any files at work. I'd never run a NAS without ZFS and ECC.
  - Aeolun 2y ago
    I don’t think I’ve ever had ECC, and have never had any issues. What kind of problems would you expect to see?
    - averageRoyalty 2y ago
      Non shielded RAM is subject to bit flipping. Non-ECC always carries this risk in general computing, but the problem is compounded when you run a filesystem like ZFS which uses memory as a significant storage element for write cache. If it would hugely impact your life if a bit were flipped from a 0 to 1 in your stored data - say you make videos or store your bitcoin wallet key on your NAS - you are running a risk not using ECC. You may not have had issues or ever have issues with non-ECC. Your car may never be stolen if you leave the keys in either, but it's not a good risk proposition for most people.
    - toast0 2y ago
      Well part of it is you likely won't see the problem until a long time after it happens. But on servers with ECC and reporting I saw several different patterns: a) 99%+ (or something) of the servers had zero reported errors for their lifetime. Memory usually works, no problems experienced. b) Some of the servers reported one error, one time, and then all was well for the rest of their life. If this happens without ECC, and you're lucky, it's in some memory that doesn't really matter and it's no big deal. Or maybe it crashes your kernel because it's in the right spot (flip a bit in a pointer/return address and it's easy to crash). Or maybe it managed to flip a bit in a pending write and your file is stored wrong and the zfs checksum is calculated on the wrong data. If you're really unlucky, you could probably write bad metadata and have a real hard time mounting the filesystem later? c) some servers reported the same address with a correctable error once a day; probably one bit stuck, any time data transits that address, that bit is still stuck, and that will likely cause trouble. If it's used for kernel memory, you'll probably end up crashing sooner or later. d) some servers had a lot more ram errors; sometimes a slow ramp to a hundred a day, once or twice a rapid ramp to so many that the system spent most of its time handling machine check exceptions, but did manage to stay online but developed a queue it could never process. Once you're at these levels, you'll probably get crashes, but you might write some bad data first. Ram testing helps on systems without ECC, but you won't really know if/when the ram has gone from working 100% to working almost 100%. I have a desktop that was running fine for a while, tests fine, but crashes in ways that have to be bad ram, and removing one stick seems to have fixed it.
singron 2y ago
This is only 1 disk, so you are way more likely to lose all your data due to an ordinary single disk failure than to some ram errors.
Spooky23 2y ago
ECC for nerds is like gear heads arguing about motor oil.
- kstrauser 2y ago
  Only if one set of gear heads was arguing that you don't really need it.
Lammy 2y ago
Hate to see this downvoted because I have personally lost files to ZFS on failing non-ECC memory. It was all my most-used stuff too because those were the files I was actually accessing, then it would compare checksum in memory to checksum on disk and decide disk was the one that was wrong. I noticed via audible blips appearing in my FLACs and verified with `flac -t <bad_flac>`.

coconut08 2y ago

been using zfs on my home nas without ecc for well over a decade and never had any problems. i've seen people claiming this since before i started using zfs and it seems so unnecessary for some random home project.

EvanAnderson 2y ago
Unless you've verified hashes of your files over time you may be having problems and not realizing it.
- theshrike79 2y ago
  If a single byte flips in a 4-10GB video file, nobody will ever notice it. There aren't that many cases where it actually matters.
- loeg 2y ago
  I believe ZFS does periodic checksuming (scrubbing).
  - yjftsjthsd-h 2y ago
    Strictly speaking I don't think ZFS itself does, but it is very common for distros to ship a cronjob that runs `zpool scrub` on a schedule (often but not always default enabled).
- ratboy666 2y ago
  They did mention ZFS, so verified hashes of each file block. I hope they are scrubbing, and have at least one snapshot.
  - mmh0000 2y ago
    ZFS does nothing to protect you against RAM corrupting your data before ZFS sees it. All you'll end up with is a valid checksum of the now bad data. You can Google more, but, I'll just leave this from the first page of the openZFS manual: Misinformation has been circulated that ZFS data integrity features are somehow worse than those of other filesystems when ECC RAM is not used. This is not the case: all software needs ECC RAM for reliable operation and ZFS is no different from any other filesystem in that regard.[1] [1] https://openzfs.readthedocs.io/en/latest/introduction.html https://openzfs.readthedocs.io/en/latest/introduction.html
  - mafro 2y ago
    Why would one snapshot help?
    - ratboy666 2y ago
      One snapshot would help because, if EVERYTHING collapses, and you need data recovery, the snapshot provides a basepoint for the recovery. This should allow better recovery of metadata. Not that this should EVER happen -- it is just a good idea. I use Jim Salter's syncoid/sanoid to make snapshots, age them out, and send data to another pool. I agree that ECC is a damn good idea - I use it on my home server. But, my lappy (i5 thinkpad) doesn't have it.
- coconut08 2y ago
  i've heard people say this, like i said, since before i started using zfs and i've never had an issue with a corrupted file. there's a few things that could be happening: i'm the luckiest person who has ever lived, these bit flip events don't happen nearly as often as people like to pretend they do, or when they do happen they aren't likely to be a big deal.
  - EvanAnderson 2y ago
    I have some JPEGs with bit flips. I could tell because they display ugly artifacts at the point of the bit flip. (You can see the kind of artifacts I'm talking about here: https://xn--andreasvlker-cjb.de/2024/02/28/image-formats-bitflips/ https://xn--andreasvlker-cjb.de/2024/02/28/image-formats-bit...) I'd happened to archive the files to CD-R's incidentally. I was able to compare those archived files to the ones that remained on my file server. There were bit flips randomly in some of the files. After that happened I started hashing all of my files and comparing hashes when I migrate files during server upgrades. Prior to using ZFS I also periodically verified file hashes with a cheapo Perl script.
  - iforgotpassword 2y ago
    If all you have on your Nas is pirated movies, then yes > when they do happen they aren't likely to be a big deal. But with more sensitive data it might matter to you. Ram can go bad like hdds can, and without ecc you have no chance of telling. Zfs won't help you here if the bit flip happens in the page cache. The file will corrupt in ram and Zfs will happily calculate a checksum for that corrupted data and store that alongside the file.

jnovek 2y ago

> so make sure to have backups Can you (or someone) suggest a backup scheme? I have a 28TB NAS. Almost everything I've looked into is expensive or intended more for enterprise tier. Are there options for backup in the "hobbyist" price range?

Dxtros 2y ago
if your talking cloud backup Wasabi (which uses S3) is the cheapest i could find it’s pay as you go and they don’t charge for upload/download. The pay as you go is $6.99 per TB which would be pretty pricey at 28 TB, but it’s super cheap for my 4tb NAS.
vunderba 2y ago
I have a NAS that has 18 TB effective storage, 36 TB mirrored. It all gets backed up to a B2 back blaze which is about six dollars per terabyte - but I'm currently only using about 8 TB at the moment so it's only about 50 bucks a month. So this might be on the higher end of the price range if you're using up all 28 TB uncompressed since that's about $168 per month though...
Dylan16807 2y ago
AWS, GCP, and Azure all offer cold storage for about $1 per TB per month. If you want any cheaper you need to build a second NAS. You could also take the awkward route and add one or two large drives to your desktop, mirror there, and back that up to backblaze (not B2). The other suggestions you got for hot storage strike me as the wrong way to handle this, if you're considering $80 per year per TB for backups then just make another NAS.
- vunderba 2y ago
  For the OP - be careful with AWS, the closest pricing to one dollar per terabyte is S3 Glacier Deep Archive and you'd be surprised how expensive a full restore can be in the event that you need to do so in terms of restore pricing, egress cost, etc. Another NAS isn't really a good solution (unless you can place it in a different house) - the goal of a cloud back up is that it's offsite.

hitsurume 2y ago

I know ECC is a special type of ram, but how does it help a NAS/Raid setup?

AlexandrB 2y ago
Data that's about to be written to disk often resides in ram for some period of time - bit flips in non-ECC ram can silently corrupt the data before writing it out. ZFS doesn't prevent this though it might detect it with checksumming. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
eric__cartman 2y ago
If you're unlucky enough to experience memory errors in one of the intermediate buffers files go through while being copied from one computer to another an incorrect copy of the file might get written to disk. When running software RAID, memory errors could also cause data to be replicated erroneously and raise an error the next time it's read. That said if the memory is flaky enough that these errors are common it's highly likely that the operating system will crash very frequently and the user will know something is seriously wrong. If you want to make sure that files have been copied correctly you can flush all kernel buffers and run diff -r between the source and destination directory to make sure that everything is the same. It's probably way more likely to experience data loss due to human error or external factors such as a power surge than bad ram. I personally thoroughly test the memory before a computer gets put into service and assume it's okay until something fails or it gets replaced. The only machine I've ever seen that would corrupt random data on a disk was heavily and carelessly overclocked (teenage me cared about getting moar fps in games, and not having a reliable workstation lol)
- barnabee 2y ago
  I wonder whether something like Syncthing would notice a hash difference with data corruption caused by such a memory error? And whether it’d correct it or propagate the issue…

theshrike79 2y ago

I've had non-ECC NAS systems for over 20 years and I've had exactly zero cases where memory corruption was an issue. It's OK for corporate systems, but complete overkill for personal setups.

jjav 2y ago
> It's OK for corporate systems, but complete overkill for personal setups. My personal files are ultimately a lot more important to me and much more irreplaceable than any files at work. I'd never run a NAS without ZFS and ECC.
- Aeolun 2y ago
  I don’t think I’ve ever had ECC, and have never had any issues. What kind of problems would you expect to see?
  - averageRoyalty 2y ago
    Non shielded RAM is subject to bit flipping. Non-ECC always carries this risk in general computing, but the problem is compounded when you run a filesystem like ZFS which uses memory as a significant storage element for write cache. If it would hugely impact your life if a bit were flipped from a 0 to 1 in your stored data - say you make videos or store your bitcoin wallet key on your NAS - you are running a risk not using ECC. You may not have had issues or ever have issues with non-ECC. Your car may never be stolen if you leave the keys in either, but it's not a good risk proposition for most people.
  - toast0 2y ago
    Well part of it is you likely won't see the problem until a long time after it happens. But on servers with ECC and reporting I saw several different patterns: a) 99%+ (or something) of the servers had zero reported errors for their lifetime. Memory usually works, no problems experienced. b) Some of the servers reported one error, one time, and then all was well for the rest of their life. If this happens without ECC, and you're lucky, it's in some memory that doesn't really matter and it's no big deal. Or maybe it crashes your kernel because it's in the right spot (flip a bit in a pointer/return address and it's easy to crash). Or maybe it managed to flip a bit in a pending write and your file is stored wrong and the zfs checksum is calculated on the wrong data. If you're really unlucky, you could probably write bad metadata and have a real hard time mounting the filesystem later? c) some servers reported the same address with a correctable error once a day; probably one bit stuck, any time data transits that address, that bit is still stuck, and that will likely cause trouble. If it's used for kernel memory, you'll probably end up crashing sooner or later. d) some servers had a lot more ram errors; sometimes a slow ramp to a hundred a day, once or twice a rapid ramp to so many that the system spent most of its time handling machine check exceptions, but did manage to stay online but developed a queue it could never process. Once you're at these levels, you'll probably get crashes, but you might write some bad data first. Ram testing helps on systems without ECC, but you won't really know if/when the ram has gone from working 100% to working almost 100%. I have a desktop that was running fine for a while, tests fine, but crashes in ways that have to be bad ram, and removing one stick seems to have fixed it.

singron 2y ago

This is only 1 disk, so you are way more likely to lose all your data due to an ordinary single disk failure than to some ram errors.

Spooky23 2y ago

ECC for nerds is like gear heads arguing about motor oil.

kstrauser 2y ago
Only if one set of gear heads was arguing that you don't really need it.

Lammy 2y ago

Hate to see this downvoted because I have personally lost files to ZFS on failing non-ECC memory. It was all my most-used stuff too because those were the files I was actually accessing, then it would compare checksum in memory to checksum on disk and decide disk was the one that was wrong. I noticed via audible blips appearing in my FLACs and verified with `flac -t <bad_flac>`.