4 ms·

Storage is a really weak area of mine, but it is important, and I do not see much technical discussion here anyway, so I am going to expose myself: How does in

Storage is a really weak area of mine, but it is important, and I do not see much technical discussion here anyway, so I am going to expose myself: How does integrity protection work in practice? I know that the bits on the HDD itself do not map one to one to bits you can actually use and that it uses that to protect us from flipped bits. I know that beyond that RAID/RAIDZ is supposed to help. I guess redundancy alone does not help. Two copies would not help to determine which bit is the correct one, you would need three. Once you have these, I guess you would be able to detect corruption. Am I guessing correct that duplication beyond 3 times would just help with read speed and in the case when a drive fails while one is already dead (e.g. when you are rebuilding)? What then? Does RAID/ZFS automatically fix it? I imagine at some point the drive would have to be replaced, do you have to check that manually or does it detect that by itself? I imagine after that you would have to run a rebuild. So I guess my question would be: Could I put a FreeNas in a corner a room and leave it there? Would it just blink when I need to put a new drive in or would it need more maintenance? Of course talking about worst case scenarios here, so say I want this to live for 10 years sitting in the corner.

codemac 10y ago
> Two copies would not help to determine which bit is the correct one, you would need three. Once you have these, I guess you would be able to detect corruption. Am I guessing correct that duplication beyond 3 times would just help with read speed and in the case when a drive fails while one is already dead (e.g. when you are rebuilding)? You're confusing two topics. Integrity vs. availability. 3x copies, RAID4 XOR, RAIDZ (a form of RAID6), etc are about increasing the availability of your data in the face of device failure/partitions. The integrity of you data is computed different ways in different systems, but you can safely think of it in your head as a pile of checksums on top of eachother. There is a half decent blog entry[0] on Oracle's website, but you can mentally model it as having checksum on disk for each block, which then is combined with other blocks into some larger object that is also checksummed, and on and on until you have "end-to-end integrity". It is a feature of all serious storage systems. I'd love if someone who was more familiar with ZFS could respond with how ZFS's internals work for you.. I worked at a direct competitor of Sun's that sued eachother over this stuff, so I actively tried to not learn about ZFS. I regret this. [0]: https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data
- bcantrill 10y ago
  It seems that you might have edited your penultimate sentence, but just to make sure that the record is unequivocally clear on this: NetApp and Sun didn't simply "sue each other", NetApp initiated patent litigation against Sun in East Texas, and Sun countersued as a defensive maneuver.[1] As for your ZFS question, Bonwick's blog is certainly a good source -- though if one is looking for a thorough treatment, I might recommend "Reliability Analysis of ZFS" by Asim Kadav and Abhishek Rajimwale.[2] [1] https://news.ycombinator.com/item?id=9129784 https://news.ycombinator.com/item?id=9129784 [2] http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf
  - codemac 10y ago
    I did not edit that part IIRC, I try not to edit anything substantially that I post online. As far as NetApp v. Sun, it was very unfortunate. I could rant for days about how crappy the business reasons were for NetApp going after Sun (zomg coraid!). I use your quote about Oracle being a lawn mower from your usenix presentation all the time. Thank you for your reply.
Nacraile 10y ago
RAID generally does not protect against bit-rot (i.e. undetected errors); it only protects against detectable errors / catastrophic disk failure. ZFS does protect against undetected (at the disk level) errors. To a first approximation, it does this by keeping block checksums alongside pointers so that it can verify that a block has not been changed since it was referenced, by keeping multiple checksummed root blocks, and by having enough redundancy to reconstruct blocks that fail their checksums. Naturally, there are information-theoretical limits to the number of corruptions that may occur for detection/correction to be guaranteed. You should refer to the relevant wikipedia pages if you want more detail than that :p edit: To answer your actual question, yes, you should be able to leave a ZFS box in the corner unattended for years with reasonable confidence that your data is safe and uncorrupted (tune redundancy settings to your taste on the space efficiency vs. error tolerance trade-off). Two caveats: 1) the machine must have an effective means of communicating a catastrophic disk failure for you to resolve (this should hold for any made-for-purpose NAS device, but you'll need to do some work if you're DIY). 2) ZFS does not actively patrol for corruptions, it fixes problems when it encounters them. If your data is not being periodically accessed, there will need to be some provision for walking the filesystem and correcting any accumulated errors (the ZFS scrub util exists for this purpose, but it has to be used)
- allendoerfer 10y ago
  > Naturally, there are information-theoretical limits to the number of corruptions that may occur for detection/correction to be guaranteed. To my practical point: Does it tell me, when it approaches that limit or do I have to put in more maintenance? Can it be fixed by swapping one of the drives?
  - Nacraile 10y ago
    The limits are on the number of simultaneous failures. Data is safe provided that too many errors do not accumulate before they can be detected and corrected. This is fundamental fact: no real storage system can tolerate an unbounded number of simultaneous errors without an unbounded number of space for replicas. You can control the number of allowable simultaneous errors by tuning redundancy settings (the trade off is space efficiency vs. probability of data loss). It is straightforward to put in place an automated process to guarantee that errors are detected within some finite period of time.
FireBeyond 10y ago
Give your computer ECC memory. Configure ZFS to scrub data regularly. It can be configured to email you of impending (SMART) or actual failure. You can label drives by ID/SN, or if your chassis or controller supports it, it can blink to tell you what drive to replace. It can be configured with a hot spare so the "resilver" starts immediately on failure detection, not "when you happen to fix".
- gmfawcett 10y ago
  Let me add that you really want to have a hot spare, and if you can't do that (e.g. no extra bay in the chassis), then you should not delay on replacing a failed disk. Have a spare disk on hand, and take the time to replace it ASAP. We lost everything on a FreeNAS system once because we were alerted to a failed-disk error, but decided to wait until the next week to replace it. But then a second disk failed, and we lost the storage pool. Lesson learned! But this was an operator error, not a system problem. FreeNAS has been tremendously stable and reliable for us.
  - johngalt 10y ago
    Everyone makes that mistake once. It is an understated problem in building fault tolerant storage arrays. We buy a batch of 20 drives all at the same time, and they are all the same manufacturer, model, size etc... Possibly even from the same batch or date of manufacture. Then we put them in continuous use in the same room, at the same temperature, in the same chassis. Finally they have an almost identical amount of reads/writes. Then we act shocked that two drives fail within a short interval of each other :)
    - mioelnir 10y ago
      Just a couple of weeks ago, I upgraded my home storage system from 9 disks (4x3 raidz) from the same manufacturer with partially consecutive serials. Now its 18 disks (3x6 raidz2) from 3 different manufacturers and every vdev has 2 of each. And the vdevs are physically evenly spread throughout the case. I sleep so much better. It was kind of a miracle the first setup survived the 4.5 years it did.
cbsmith 10y ago
So many levels to this. Drives themselves remap sectors to handle bad sectors, and some have integrated ECC bits... there are also things at the sensor level to help recover from errors. Once you are at the block level, you have simple parity checking with some redundancy models, but the good ones use ECC bits. Usually this is done at the block device layer, but ZFS and some other filesystems do this at the filesystem layer (sometimes you have multiple layers of this). You're right that beyond a few copies, the main win tends to be read speed, but there is one other factor: surviving rebuilds. As drives get larger, the failure rate is such that the chance of failure during a rebuild gets too disturbingly high, so you need additional redundancy to avoid catastrophic failure during recovery. As for "automatically fix it", the short answer is there is a lot of stuff that automatically fixes problems, but it is a leaky abstraction. RAID-5 rebuilds are notoriously terrible for performance, and often it is easier to have logic for dealing with failures & redundancy at the application layer. FreeNAS and similar projects are definitely intended to be turnkey storage solutions. They have their strengths & weaknesses, but the notion that you just plug it in and go isn't too far off. Usually you don't go with the blinky light, but with an alerting mechanism (e-mail, SMS, whatever) that you integrate with it for notifications about problems. In principle, it is a fire & forget kind of thing.
- baruch 10y ago
  An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. ECC is always written alongside the actual block and the overhead for ECC is the reason for the move from 512b sectors to 4Kb sectors in HDDs. For SSDs the data is already written in different block sizes depending on the NAND and the internal representation and ECC is done for larger than 512b units. The probability of failure during rebuild is not really directly linked to drive size, the usual interpretation of the drive BER is wrong (media BER is stated across a large population of drives rather than just one drive).
  - cmurf 10y ago
    Yes, the two key things: population of drives (model/version possibly, but ultimately not publicly known what the sample size is), and the rate is generally a maximum. i.e. use of "less than" symbol with the rate. Therefore, it's wrong to say every x bytes you should expect an unrecoverable read error. We have no idea how good the drives actually are, the reporting is in orders of magnitude anyway so it could be 8 or 9 times more reliable than the reported rate; or even multiple orders of magnitude. It's not untrue to say < 1 URE in 10^14 bits, but for the population to actually experience 1 URE in 10^16 bits. Why not advertise that? Well, maybe there's a product that costs a little more than advertises 1 URE in 10^15 bits. And other product classes that promise better.
dragontamer 10y ago
> How does integrity protection work in practice? I know that the bits on the HDD itself do not map one to one to bits you can actually use and that it uses that to protect us from flipped bits. Checksums. If you get big enough checksums (aka: Reed-Solomon Codes), you can not only detect errors but correct them as well. See https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor... for the math. Now that you have this "error-correcting checksum", where do you put it? Raid5 means you place the error-correcting checksum on different disks. If you have three disks: A, B, and C. You'll put the data on A & B, then the checksum on C. This is RAID4 (which is never used). RAID5 is much like RAID4, except you also cycle between the drives. So the checksum information is stored on A, B, or C. Sometimes the data is on A&B, sometimes is B&C, and sometimes its on A&C. ----------------------- > Could I put a FreeNas in a corner a room and leave it there? Would it just blink when I need to put a new drive in or would it need more maintenance? Of course talking about worst case scenarios here, so say I want this to live for 10 years sitting in the corner. Maybe, maybe not. If all the drives fail in those 10 years, of course not (Hard Drive arms may lose lubricant. If the arms stop moving, you won't be able to read the data). "Good practice" means that you want to boot up the ZFS box and run a "scrub" on it every few months, to ensure all the hard drives are actually working. If one fails, you replace it and rebuild all the checksums (or the data from the checksums). RAID / ZFS isn't a magic bullet. They just buy you additional time: time where your rig begins to break but is still in a repairable position. ZFS has more checksums everywhere to check for a few more cases than simple RAID5. But otherwise, the fundamentals remain the same. You need to regularly check for broken hard drives and then replace them before too many hard drives break. --------- This also means that no single box can protect you from a natural disaster: fires, floods, earthquakes... these can destroy your backup all at once. If all hard drives fail at the same time, you lose your data.
ldehaan 10y ago
if you intend to try this, I would recommend getting spinrite at grc.com, and run it a couple times every year, if you're going to be doing a lot of io on it. I run it on all my hdd's and its kept them alive and running smoothly for years at a time.
- voltagex_ 10y ago
  http://serverfault.com/questions/51681/does-spinrite-do-what-it-claims-to-do http://serverfault.com/questions/51681/does-spinrite-do-what...

codemac 10y ago

> Two copies would not help to determine which bit is the correct one, you would need three. Once you have these, I guess you would be able to detect corruption. Am I guessing correct that duplication beyond 3 times would just help with read speed and in the case when a drive fails while one is already dead (e.g. when you are rebuilding)? You're confusing two topics. Integrity vs. availability. 3x copies, RAID4 XOR, RAIDZ (a form of RAID6), etc are about increasing the availability of your data in the face of device failure/partitions. The integrity of you data is computed different ways in different systems, but you can safely think of it in your head as a pile of checksums on top of eachother. There is a half decent blog entry[0] on Oracle's website, but you can mentally model it as having checksum on disk for each block, which then is combined with other blocks into some larger object that is also checksummed, and on and on until you have "end-to-end integrity". It is a feature of all serious storage systems. I'd love if someone who was more familiar with ZFS could respond with how ZFS's internals work for you.. I worked at a direct competitor of Sun's that sued eachother over this stuff, so I actively tried to not learn about ZFS. I regret this. [0]: https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data

bcantrill 10y ago
It seems that you might have edited your penultimate sentence, but just to make sure that the record is unequivocally clear on this: NetApp and Sun didn't simply "sue each other", NetApp initiated patent litigation against Sun in East Texas, and Sun countersued as a defensive maneuver.[1] As for your ZFS question, Bonwick's blog is certainly a good source -- though if one is looking for a thorough treatment, I might recommend "Reliability Analysis of ZFS" by Asim Kadav and Abhishek Rajimwale.[2] [1] https://news.ycombinator.com/item?id=9129784 https://news.ycombinator.com/item?id=9129784 [2] http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf
- codemac 10y ago
  I did not edit that part IIRC, I try not to edit anything substantially that I post online. As far as NetApp v. Sun, it was very unfortunate. I could rant for days about how crappy the business reasons were for NetApp going after Sun (zomg coraid!). I use your quote about Oracle being a lawn mower from your usenix presentation all the time. Thank you for your reply.

Nacraile 10y ago

RAID generally does not protect against bit-rot (i.e. undetected errors); it only protects against detectable errors / catastrophic disk failure. ZFS does protect against undetected (at the disk level) errors. To a first approximation, it does this by keeping block checksums alongside pointers so that it can verify that a block has not been changed since it was referenced, by keeping multiple checksummed root blocks, and by having enough redundancy to reconstruct blocks that fail their checksums. Naturally, there are information-theoretical limits to the number of corruptions that may occur for detection/correction to be guaranteed. You should refer to the relevant wikipedia pages if you want more detail than that :p edit: To answer your actual question, yes, you should be able to leave a ZFS box in the corner unattended for years with reasonable confidence that your data is safe and uncorrupted (tune redundancy settings to your taste on the space efficiency vs. error tolerance trade-off). Two caveats: 1) the machine must have an effective means of communicating a catastrophic disk failure for you to resolve (this should hold for any made-for-purpose NAS device, but you'll need to do some work if you're DIY). 2) ZFS does not actively patrol for corruptions, it fixes problems when it encounters them. If your data is not being periodically accessed, there will need to be some provision for walking the filesystem and correcting any accumulated errors (the ZFS scrub util exists for this purpose, but it has to be used)

allendoerfer 10y ago
> Naturally, there are information-theoretical limits to the number of corruptions that may occur for detection/correction to be guaranteed. To my practical point: Does it tell me, when it approaches that limit or do I have to put in more maintenance? Can it be fixed by swapping one of the drives?
- Nacraile 10y ago
  The limits are on the number of simultaneous failures. Data is safe provided that too many errors do not accumulate before they can be detected and corrected. This is fundamental fact: no real storage system can tolerate an unbounded number of simultaneous errors without an unbounded number of space for replicas. You can control the number of allowable simultaneous errors by tuning redundancy settings (the trade off is space efficiency vs. probability of data loss). It is straightforward to put in place an automated process to guarantee that errors are detected within some finite period of time.

FireBeyond 10y ago

Give your computer ECC memory. Configure ZFS to scrub data regularly. It can be configured to email you of impending (SMART) or actual failure. You can label drives by ID/SN, or if your chassis or controller supports it, it can blink to tell you what drive to replace. It can be configured with a hot spare so the "resilver" starts immediately on failure detection, not "when you happen to fix".

gmfawcett 10y ago
Let me add that you really want to have a hot spare, and if you can't do that (e.g. no extra bay in the chassis), then you should not delay on replacing a failed disk. Have a spare disk on hand, and take the time to replace it ASAP. We lost everything on a FreeNAS system once because we were alerted to a failed-disk error, but decided to wait until the next week to replace it. But then a second disk failed, and we lost the storage pool. Lesson learned! But this was an operator error, not a system problem. FreeNAS has been tremendously stable and reliable for us.
- johngalt 10y ago
  Everyone makes that mistake once. It is an understated problem in building fault tolerant storage arrays. We buy a batch of 20 drives all at the same time, and they are all the same manufacturer, model, size etc... Possibly even from the same batch or date of manufacture. Then we put them in continuous use in the same room, at the same temperature, in the same chassis. Finally they have an almost identical amount of reads/writes. Then we act shocked that two drives fail within a short interval of each other :)
  - mioelnir 10y ago
    Just a couple of weeks ago, I upgraded my home storage system from 9 disks (4x3 raidz) from the same manufacturer with partially consecutive serials. Now its 18 disks (3x6 raidz2) from 3 different manufacturers and every vdev has 2 of each. And the vdevs are physically evenly spread throughout the case. I sleep so much better. It was kind of a miracle the first setup survived the 4.5 years it did.

cbsmith 10y ago

So many levels to this. Drives themselves remap sectors to handle bad sectors, and some have integrated ECC bits... there are also things at the sensor level to help recover from errors. Once you are at the block level, you have simple parity checking with some redundancy models, but the good ones use ECC bits. Usually this is done at the block device layer, but ZFS and some other filesystems do this at the filesystem layer (sometimes you have multiple layers of this). You're right that beyond a few copies, the main win tends to be read speed, but there is one other factor: surviving rebuilds. As drives get larger, the failure rate is such that the chance of failure during a rebuild gets too disturbingly high, so you need additional redundancy to avoid catastrophic failure during recovery. As for "automatically fix it", the short answer is there is a lot of stuff that automatically fixes problems, but it is a leaky abstraction. RAID-5 rebuilds are notoriously terrible for performance, and often it is easier to have logic for dealing with failures & redundancy at the application layer. FreeNAS and similar projects are definitely intended to be turnkey storage solutions. They have their strengths & weaknesses, but the notion that you just plug it in and go isn't too far off. Usually you don't go with the blinky light, but with an alerting mechanism (e-mail, SMS, whatever) that you integrate with it for notifications about problems. In principle, it is a fire & forget kind of thing.

baruch 10y ago
An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. ECC is always written alongside the actual block and the overhead for ECC is the reason for the move from 512b sectors to 4Kb sectors in HDDs. For SSDs the data is already written in different block sizes depending on the NAND and the internal representation and ECC is done for larger than 512b units. The probability of failure during rebuild is not really directly linked to drive size, the usual interpretation of the drive BER is wrong (media BER is stated across a large population of drives rather than just one drive).
- cmurf 10y ago
  Yes, the two key things: population of drives (model/version possibly, but ultimately not publicly known what the sample size is), and the rate is generally a maximum. i.e. use of "less than" symbol with the rate. Therefore, it's wrong to say every x bytes you should expect an unrecoverable read error. We have no idea how good the drives actually are, the reporting is in orders of magnitude anyway so it could be 8 or 9 times more reliable than the reported rate; or even multiple orders of magnitude. It's not untrue to say < 1 URE in 10^14 bits, but for the population to actually experience 1 URE in 10^16 bits. Why not advertise that? Well, maybe there's a product that costs a little more than advertises 1 URE in 10^15 bits. And other product classes that promise better.

dragontamer 10y ago

> How does integrity protection work in practice? I know that the bits on the HDD itself do not map one to one to bits you can actually use and that it uses that to protect us from flipped bits. Checksums. If you get big enough checksums (aka: Reed-Solomon Codes), you can not only detect errors but correct them as well. See https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor... for the math. Now that you have this "error-correcting checksum", where do you put it? Raid5 means you place the error-correcting checksum on different disks. If you have three disks: A, B, and C. You'll put the data on A & B, then the checksum on C. This is RAID4 (which is never used). RAID5 is much like RAID4, except you also cycle between the drives. So the checksum information is stored on A, B, or C. Sometimes the data is on A&B, sometimes is B&C, and sometimes its on A&C. ----------------------- > Could I put a FreeNas in a corner a room and leave it there? Would it just blink when I need to put a new drive in or would it need more maintenance? Of course talking about worst case scenarios here, so say I want this to live for 10 years sitting in the corner. Maybe, maybe not. If all the drives fail in those 10 years, of course not (Hard Drive arms may lose lubricant. If the arms stop moving, you won't be able to read the data). "Good practice" means that you want to boot up the ZFS box and run a "scrub" on it every few months, to ensure all the hard drives are actually working. If one fails, you replace it and rebuild all the checksums (or the data from the checksums). RAID / ZFS isn't a magic bullet. They just buy you additional time: time where your rig begins to break but is still in a repairable position. ZFS has more checksums everywhere to check for a few more cases than simple RAID5. But otherwise, the fundamentals remain the same. You need to regularly check for broken hard drives and then replace them before too many hard drives break. --------- This also means that no single box can protect you from a natural disaster: fires, floods, earthquakes... these can destroy your backup all at once. If all hard drives fail at the same time, you lose your data.

ldehaan 10y ago

if you intend to try this, I would recommend getting spinrite at grc.com, and run it a couple times every year, if you're going to be doing a lot of io on it. I run it on all my hdd's and its kept them alive and running smoothly for years at a time.

voltagex_ 10y ago
http://serverfault.com/questions/51681/does-spinrite-do-what-it-claims-to-do http://serverfault.com/questions/51681/does-spinrite-do-what...