3 ms·

There is a lot of cargo cult about this. Firstly the claim that freebsd has wider testing is utter trash. In terms of TBs installed ZFS on linux >> freenas/fre

There is a lot of cargo cult about this. Firstly the claim that freebsd has wider testing is utter trash. In terms of TBs installed ZFS on linux >> freenas/freebsd the amount of money behind ZoL is now surprisingly large. Also, extensive memtest of ECC ram is pointless, ECC ram has checksumming built into the chip which will tell you if there is a memory error and correct it if possible. As for most drive fail in the first hours: https://www.backblaze.com/blog/how-long-do-disk-drives-last/ https://www.backblaze.com/blog/how-long-do-disk-drives-last/ evidence says other wise. Infant mortality is a thing, but its a matter of months, not hours. (internal QA grabs most of the ones that die within a few hours.) The best way to combat simultaneous failure is to mix hardrive types, this makes it much less likley that a single fault class will be triggered on all disks.

nitin_flanker 11y ago
How does it tell which drive has failed? or is not working properly? I never worked with NAS rigs, but I sure have plan to setup one for my home.
- KaiserPro 11y ago
  There are a number of ways. The most visible, even if you are not using ZFS is using the SMART output http://www.techrepublic.com/blog/linux-and-open-source/using-smartctl-to-get-smart-status-information-on-your-hard-drives/ http://www.techrepublic.com/blog/linux-and-open-source/using... There are a number of metrics that might indicate the health of the drive not all raid cards support this, however most consumer devices do. If you have a card that doesn't do this, either its a highend jobby with another mechanism, or a pile of shite. second when you actually bump into a hard error, you'll see things in /var/log/messages that say "disk timed out, retrying" (but then you're probably fucked by then) However using ZFS/BTRFS data is checksum'd when written and read. This means that bit flips and otherwise silent bit rot has a greater chance of being caught early. Performing "scrub" (consistency checks) often means you're more likely to catch errors early. (this is another reason why you'll want, but ECC ram. as it'll tell you when you get bit flips in ram, which undermines on disk checksums) Part of the reason why people choose raid6/z2 is because it takes a long time to rebuild an array after a disk crash. It involves reading all the data again and computing the hamming code to recover the lost data. when you buy a job lot of disks, they tend to be the same batch, which means they could be prone to the same bug/defect. So this general means that multiple disk failure happen at the same time. having another parity disk means you can withstand more failure before total loss. However, none of these mechanisms are a replacement for a backup. even if you have a 36 gig raid, you still need a backup. There will always be a time where something goes horrifically wrong and you need to get access to an archived version. anyone who fights against this has yet to experience the fun of total failure.
  - nitin_flanker 11y ago
    Cool! Disk failures and losing data scares the shite out of me. >even if you have a 36 gig raid, you still need a backup and a backup of that backup too.
    - Luyt 11y ago
      Yes, multiple backups, kept at remote places. Don't store the backup in the server room! ;-)
      - lsaferite 11y ago
        At different remote locations for each backup.
      - lsaferite 11y ago
        At different remote locations for each backup.
    - KaiserPro 11y ago
      ITs up to you. I'm happy with many copies of my core files that I desperately need (photos and or stuff that gets me money) I have my ZFS box, which is backed up to a remote disk miles away. My time machine has a few local copies too. Just be ware, snapshots are not the same as a full backup. They are brilliant, but not a backup.
  - XorNot 11y ago
    ZFS and BTRFS both provide snapshot capability. This substantially changes the metric for a backup - i.e. if disaster recovery is an absolute priority, then you need it, but your backups will include your snapshot set.
Amezarak 11y ago
> Also, extensive memtest of ECC ram is pointless, ECC ram has checksumming built into the chip which will tell you if there is a memory error and correct it if possible. Yes, but if the chip is getting lots of memory errors, correctable or uncorrectable, I probably don't want it. I don't want to build my system and put it into production and then find out.
- KaiserPro 11y ago
  you'd pick this up straight away, ECC ram detects memory errors in real time. The reason why you run memtest is to systematically go through all the bits and flip them in a known pattern. You can check that known pattern to see if the bit changed properly. With ECC this is redundant, as each byte(or word, or some unit of memory) has a hardware checksum. So using ECC ram in normal operation will indicate if there are errors.
  - Amezarak 11y ago
    Yes, I understand that. The reason to run memtest on ECC memory is to verify that every bit of memory is good before you finish building the system, installing an OS, and putting it into production. If you have, say, 32GB of memory, there's no guarantee you'll hit those bad bits for ECC to log an error until you're doing something memory intensive two weeks later. The primary use case of ECC is to handle random bit flips from cosmic rays and whatnot, not to mitigate bad hardware. If you have bad memory, replace it, don't rely on ECC. The only way to test memory is to run something like memtest, where you read and write from every bit in memory. If you're OK with finding out a month later that ECC is logging a hundred uncorrectable errors an hour whenever you do anything memory intensive, then sure, don't bother memtesting. If you would rather deal with it beforehand, then run memtest.
    - KaiserPro 11y ago
      but you have hardware monitoring linked to an alerting framework right?
      - 11y ago
cm3 11y ago
> Firstly the claim that freebsd has wider testing is utter trash. In terms of TBs installed ZFS on linux >> freenas/freebsd the amount of money behind ZoL is now surprisingly large. In that case OmniOS (or any IllumOS based) distribution is a much better choice. It has a more full featured ZFS (features not yet ported to FreeBSD or ZoL) and a more complete set of DTrace probes to analyze problems. Moreover it's still the OpenZFS upstream for all intents and purposes, but that may change in the future. On top of that ZFS on Solaris doesn't try to fit into a foreign kernel. I'm not sure but FreeBSD's GELI or Linux's LUKS may be missing on Illumos (but exists in Oracle Solaris 11), so that can be a disadvantage. If none of Linux, Illumos or FreeBSD is to your liking, you can try ZFS on NetBSD.
- feld 11y ago
  I wish people would stop with the FUD here. Matthew Ahrens has publicly stated that FreeBSD and Solaris derivatives are equal as far as ZFS goes and OSX and Linux are trailing behind.
  - cm3 11y ago
    If that's true then bsdnow.tv should be informed. In a couple discussions and interviews (Bryan?) they talk about feature disparity and that a newer OpenZFS version needs to be imported into FreeBSD trunk. It's still true that if you want to analyze serious issues, IllumOS DTrace probes will be more complete. With that being said, I'm glad to hear FreeBSD is not behind IllumOS ZFS. FreeBSD has wider hardware support than IllumOS and thus is preferred by most users.
    - feld 11y ago
      We're talking about bleeding edge features that were just created, and yes they're often modeled upstream in Illumos. However, there are other features that have been created by companies using FreeBSD first. But they tend to get pushed up to OpenZFS first and then trickle back down to vanilla FreeBSD. When it comes to stability and performance FreeBSD == Illumos. I wouldn't worry so much about features anymore, because as soon as they're available they get pulled in. Last I looked the awesome ZFS features that are coming still weren't fully accepted upstream yet.

nitin_flanker 11y ago

How does it tell which drive has failed? or is not working properly? I never worked with NAS rigs, but I sure have plan to setup one for my home.

KaiserPro 11y ago
There are a number of ways. The most visible, even if you are not using ZFS is using the SMART output http://www.techrepublic.com/blog/linux-and-open-source/using-smartctl-to-get-smart-status-information-on-your-hard-drives/ http://www.techrepublic.com/blog/linux-and-open-source/using... There are a number of metrics that might indicate the health of the drive not all raid cards support this, however most consumer devices do. If you have a card that doesn't do this, either its a highend jobby with another mechanism, or a pile of shite. second when you actually bump into a hard error, you'll see things in /var/log/messages that say "disk timed out, retrying" (but then you're probably fucked by then) However using ZFS/BTRFS data is checksum'd when written and read. This means that bit flips and otherwise silent bit rot has a greater chance of being caught early. Performing "scrub" (consistency checks) often means you're more likely to catch errors early. (this is another reason why you'll want, but ECC ram. as it'll tell you when you get bit flips in ram, which undermines on disk checksums) Part of the reason why people choose raid6/z2 is because it takes a long time to rebuild an array after a disk crash. It involves reading all the data again and computing the hamming code to recover the lost data. when you buy a job lot of disks, they tend to be the same batch, which means they could be prone to the same bug/defect. So this general means that multiple disk failure happen at the same time. having another parity disk means you can withstand more failure before total loss. However, none of these mechanisms are a replacement for a backup. even if you have a 36 gig raid, you still need a backup. There will always be a time where something goes horrifically wrong and you need to get access to an archived version. anyone who fights against this has yet to experience the fun of total failure.
- nitin_flanker 11y ago
  Cool! Disk failures and losing data scares the shite out of me. >even if you have a 36 gig raid, you still need a backup and a backup of that backup too.
  - Luyt 11y ago
    Yes, multiple backups, kept at remote places. Don't store the backup in the server room! ;-)
    - lsaferite 11y ago
      At different remote locations for each backup.
    - lsaferite 11y ago
      At different remote locations for each backup.
  - KaiserPro 11y ago
    ITs up to you. I'm happy with many copies of my core files that I desperately need (photos and or stuff that gets me money) I have my ZFS box, which is backed up to a remote disk miles away. My time machine has a few local copies too. Just be ware, snapshots are not the same as a full backup. They are brilliant, but not a backup.
- XorNot 11y ago
  ZFS and BTRFS both provide snapshot capability. This substantially changes the metric for a backup - i.e. if disaster recovery is an absolute priority, then you need it, but your backups will include your snapshot set.

Amezarak 11y ago

> Also, extensive memtest of ECC ram is pointless, ECC ram has checksumming built into the chip which will tell you if there is a memory error and correct it if possible. Yes, but if the chip is getting lots of memory errors, correctable or uncorrectable, I probably don't want it. I don't want to build my system and put it into production and then find out.

KaiserPro 11y ago
you'd pick this up straight away, ECC ram detects memory errors in real time. The reason why you run memtest is to systematically go through all the bits and flip them in a known pattern. You can check that known pattern to see if the bit changed properly. With ECC this is redundant, as each byte(or word, or some unit of memory) has a hardware checksum. So using ECC ram in normal operation will indicate if there are errors.
- Amezarak 11y ago
  Yes, I understand that. The reason to run memtest on ECC memory is to verify that every bit of memory is good before you finish building the system, installing an OS, and putting it into production. If you have, say, 32GB of memory, there's no guarantee you'll hit those bad bits for ECC to log an error until you're doing something memory intensive two weeks later. The primary use case of ECC is to handle random bit flips from cosmic rays and whatnot, not to mitigate bad hardware. If you have bad memory, replace it, don't rely on ECC. The only way to test memory is to run something like memtest, where you read and write from every bit in memory. If you're OK with finding out a month later that ECC is logging a hundred uncorrectable errors an hour whenever you do anything memory intensive, then sure, don't bother memtesting. If you would rather deal with it beforehand, then run memtest.
  - KaiserPro 11y ago
    but you have hardware monitoring linked to an alerting framework right?
    - 11y ago

cm3 11y ago

> Firstly the claim that freebsd has wider testing is utter trash. In terms of TBs installed ZFS on linux >> freenas/freebsd the amount of money behind ZoL is now surprisingly large. In that case OmniOS (or any IllumOS based) distribution is a much better choice. It has a more full featured ZFS (features not yet ported to FreeBSD or ZoL) and a more complete set of DTrace probes to analyze problems. Moreover it's still the OpenZFS upstream for all intents and purposes, but that may change in the future. On top of that ZFS on Solaris doesn't try to fit into a foreign kernel. I'm not sure but FreeBSD's GELI or Linux's LUKS may be missing on Illumos (but exists in Oracle Solaris 11), so that can be a disadvantage. If none of Linux, Illumos or FreeBSD is to your liking, you can try ZFS on NetBSD.

feld 11y ago
I wish people would stop with the FUD here. Matthew Ahrens has publicly stated that FreeBSD and Solaris derivatives are equal as far as ZFS goes and OSX and Linux are trailing behind.
- cm3 11y ago
  If that's true then bsdnow.tv should be informed. In a couple discussions and interviews (Bryan?) they talk about feature disparity and that a newer OpenZFS version needs to be imported into FreeBSD trunk. It's still true that if you want to analyze serious issues, IllumOS DTrace probes will be more complete. With that being said, I'm glad to hear FreeBSD is not behind IllumOS ZFS. FreeBSD has wider hardware support than IllumOS and thus is preferred by most users.
  - feld 11y ago
    We're talking about bleeding edge features that were just created, and yes they're often modeled upstream in Illumos. However, there are other features that have been created by companies using FreeBSD first. But they tend to get pushed up to OpenZFS first and then trickle back down to vanilla FreeBSD. When it comes to stability and performance FreeBSD == Illumos. I wouldn't worry so much about features anymore, because as soon as they're available they get pulled in. Last I looked the awesome ZFS features that are coming still weren't fully accepted upstream yet.