4 ms·

> Are you scrubbing once a month? If not this will happen again. Linus Media lost 100’s of TB that way. Anybody running 100 TB+ ZFS arrays should really have b

> Are you scrubbing once a month? If not this will happen again. Linus Media lost 100’s of TB that way. Anybody running 100 TB+ ZFS arrays should really have been aware of this class of problem before deploying (and doubly so if using consumer drives). ZFS protects from bitrot and hardware failures... but it's not magic. If data is written once and not read again for a long time, you won't know it's been sitting on bad sectors until it's too late, and you may end up with multiple failing drives at once if you lose a disk and try to resilver. Far better to check periodically so you can throw bad disks early. Hopefully they had a good backup strategy!

aerojoe23 5y ago
Watched the video where Linus talks about it. Apparently they store all of the footage as a nice to have, and to have a use case to make the large storage content around. All that to say yes they lost all that data, no it wasn't backed up. Not critical to the business.
Handytinge 5y ago
I have a 100TB+ ZFS based NAS and have poor understanding of this. I'm yet to see anyone make a really good set of resources to either watch or read that are not much too technically deep.
- JeremyNT 5y ago
  This is tricky, because it's a class of problem that you need to understand exists in order to seek out the relevant information. Once you know what to look for, it's easy to find guidance - even from Oracle themselves [0] A lot of us have "learned the hard way," as it sounds like Linus himself eventually did. I think this highlights an issue with the "learn through youtube video" approach. An internet celebrity may acquire enough knowledge to do accessible demonstrations, presenting totally valid, useful, and correct information in them, and still miss crucial "unknown unknowns" that they simply hadn't encountered in their own research. It's hard to know what to recommend for a class of issue you aren't even aware of! [0] https://blogs.oracle.com/oracle-systems/post/disk-scrub-why-and-when https://blogs.oracle.com/oracle-systems/post/disk-scrub-why-...

aerojoe23 5y ago

Watched the video where Linus talks about it. Apparently they store all of the footage as a nice to have, and to have a use case to make the large storage content around. All that to say yes they lost all that data, no it wasn't backed up. Not critical to the business.

Handytinge 5y ago

I have a 100TB+ ZFS based NAS and have poor understanding of this. I'm yet to see anyone make a really good set of resources to either watch or read that are not much too technically deep.

JeremyNT 5y ago
This is tricky, because it's a class of problem that you need to understand exists in order to seek out the relevant information. Once you know what to look for, it's easy to find guidance - even from Oracle themselves [0] A lot of us have "learned the hard way," as it sounds like Linus himself eventually did. I think this highlights an issue with the "learn through youtube video" approach. An internet celebrity may acquire enough knowledge to do accessible demonstrations, presenting totally valid, useful, and correct information in them, and still miss crucial "unknown unknowns" that they simply hadn't encountered in their own research. It's hard to know what to recommend for a class of issue you aren't even aware of! [0] https://blogs.oracle.com/oracle-systems/post/disk-scrub-why-and-when https://blogs.oracle.com/oracle-systems/post/disk-scrub-why-...