7 ms·

Disks lie. And the controllers that run them are partners in crime.

141 points by pwg 14y ago

jat850 14y ago
I wish I could post the video from the talk, but here are some slides at least. Good supporting talk on the pitfalls of making assumptions about how disks work, from OSCON '11: http://www.slideshare.net/iammutex/what-every-data-programmer-needs-to-know-about-disks http://www.slideshare.net/iammutex/what-every-data-programme...
ChuckMcM 14y ago
During my tenure at NetApp I got to see all sorts of really really interesting disk problems and the lengths software had to go to reliably store data on them. Two scars burned into me from that were 1) disks suck, and 2) disks are not 'storage'. The first part is pretty easy to understand, storage manufacturers have competed for years in a commodity market where consumers often choose price per gigabyte over URER (the unrecoverable read error rate), further at the scale of the market small savings of cents adds up to better margins. And while the 'enterprise' fibre channel and SCSI drives could (and did) command a hefty premium, the shift to SATA drives really put a crimp in the over all margin picture. So the disk manufacturers are stuck between the reliability of the drive and the cost of the drive. They surf the curve where it is just reliable enough to keep people buying drives. This trend bites back, making the likely hood of an error while reading more and more probable. Not picking on Seagate here, they are all similar, but lets look at their Barracuda drive's spec sheet [1]. You will notice a parameter 'Nonrecoverable Read Errors per Bits read', and you'll see that its 1x10e14 across the sheet, from 3TB down to 250GB. It is a statistical thing, the whole magnetic field domain to digital bit pipeline is one giant analog loop of a error extraction. 1x10E14 bits. So what does that mean? Lets say each byte is encoded in 10 bits on the disk. Three trillion bytes is 30 trillion bits in that case, or 3x10E13 bits. Best case, if you read that disk from one end to the other (like you were re-silvering a mirror from a RAID 10 setup) you have a 1 in 3 chance that one of those sectors won't be readable for a perfectly working disk. Amazing isn't it? Trying to reconstruct a RAID5 group with 4 disks remaining out of 5, look out. So physics is not on your side, but we've got RAID 6 Chuck! Of course you do, and that is a good thing. But what about when you write to the disk, the disk replies "Yeah sure boss! Got that on the platters now" but it didn't actually right it? Now you've got a parity failure waiting to bite you sitting on the drive, or worse, it does write the sector but writes it somewhere else (saw this a number of times at NetApp as well). There are three million lines of code in the disk firmware. One of the manufacturers (my memory says Maxtor) showed a demo where they booted Linux on the controller of the drive. Bottom line is that the system works mostly, which is a huge win, and a lot of people blame the OS when the drive is at fault, another bonus for manufacturers, but unless your data is in a RAID 6 group or on at least 3 drives, its not really 'safe' in the "I am absolutely positive I could read this back" sense of the word. [1] http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/disc/barracuda-ds1737-1-1111us.pdf http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/...
- biot 14y ago
  What were the least problematic drives in your experience?
  - protomyth 14y ago
    Not original poster, but I favor WD over Seagate because lately all Seagate drives we have got (about 20) failed in a couple of months. I have had 2 WD failures this years (about 100 drives in various RAID configs). Whatever brand ships with HP netbooks is basically a failure waiting to happen (we bought 100 HP netbooks with 3G from Verizon - they are very sick of us calling).
    - mbreese 14y ago
      Everyone has their own horror stories about drive manufacturers. I personally had a bunch of WD drives fail a while ago, so I've been hesitant to buy WD. Now, this was 10-ish years ago, so I know reliability has improved, but that doesn't change my hesitation. It used to be that Maxtor (when they were around and independent) had the most reliable drives. Now, it's who knows...
      - tjoff 14y ago
        Maxtor had horrible reputation (by far the worst) among most of my friends and local forums. Things is, all drive manufacturers have batches and you might end up with 8 dead out of 10 drives for any manufacturer at any given time. The only thing you can do is try to get drives from different batches and thus trying to minimize having all eggs in one basket (what good is raid6 if all drives die?). Also when a lot of drives die you should look for other culprits as well, is the temperature OK? Is the power stable? (Vibrations?) There isn't much statistics available (that I know of) but a french store published their return-rate for drives (which of course doesn't count drives returned directly to the manufacturer) http://www.hardware.fr/articles/831-6/disques-durs.html http://www.hardware.fr/articles/831-6/disques-durs.html Or a write up in Swedish about it (probably easier to read if you don't know either language, lot of graphs): http://www.sweclockers.com/nyhet/13859-fransk-datorsajt-publicerar-rma-statistik-for-april-manad http://www.sweclockers.com/nyhet/13859-fransk-datorsajt-publ... But all in all, if you have many drives failing you've hit a bad batch or are treating your drives badly, nothing you can do about it (the former at least).
recoiledsnake 14y ago
The article glosses over the fact that fsync() itself has major issues. For example, on ext3, if you call fsync() on a single file, all file system cached data is written to disk, leading to an extreme slowdown. This led to a Firefox bug, when the sqlite db used for the awesome bar and for bookmarks called fsync() and slowed everything down. http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/ http://shaver.off.net/diary/2008/05/25/fsyncers-and-curvebal... OS X does a fake fsync() if you call it with the defaults. https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdPersistentStores.html https://developer.apple.com/library/mac/#documentation/Cocoa... fsync in Mac OS X: Since in Mac OS X the fsync command does not make the guarantee that bytes are written, SQLite sends a F_FULLFSYNC request to the kernel to ensures that the bytes are actually written through to the drive platter. This causes the kernel to flush all buffers to the drives and causes the drives to flush their track caches. Without this, there is a significantly large window of time within which data will reside in volatile memory—and in the event of system failure you risk data corruption
- btrask 14y ago
  So in summary, I believe that the comments in the MySQL news posting are slightly confused. On MacOS X fsync() behaves the same as it does on all Unices. That's not good enough if you really care about data integrity and so we also provide the F_FULLFSYNC fcntl. As far as I know, MacOS X is the only OS to provide this feature for apps that need to truly guarantee their data is on disk. The full post goes into detail. http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072...
scdeepak 14y ago
Me and my colleagues had worked on a solution to this problem a few years back. The results were published and is linked here. ACM digital library link - http://dl.acm.org/citation.cfm?id=2056434 http://dl.acm.org/citation.cfm?id=2056434 The paper - http://research.cs.wisc.edu/adsl/Publications/cce-dsn11.pdf http://research.cs.wisc.edu/adsl/Publications/cce-dsn11.pdf
- baruch 14y ago
  That's a really neat hack, but how many disks have you seen that do not adhere to their write-cache-disabled setting?
rarrrrrr 14y ago
At SpiderOak, every storage cluster writes new customer data to three RAID6 volumes on 3 different machines, which wait for fdatasync, before we consider it written. ... and before that begins, the data is added to a giant ring buffer that records the last 30 TB of new writes, on machines with UPS. If a rack power failure happens, the ring buffer is kept until storage audits complete. I think every company that deals with large data they can't lose develops appropriate paranoia. The behavior of hard drives is like decaying atoms. You can't make accurate predictions about what any one of them will do. Only in aggregate can you say something like "the half life of this pile of hardware is 12 years" or "if we write this data N times we can reasonably expect to read it it again."
- baruch 14y ago
  Interesting, do you do disk scrubbing to make sure the data is still out there after a while? That would be the key method to ensure you can still recover the data upon need.
kabdib 14y ago
I had fun once diagnosing a bad bit in a drive's cache memory. Writes would go out, and sometimes come back with a bit set. All the disk-resident CRCs in the world won't help if your data is mangled before it makes it to the media. File systems with end-to-end checking are good. (These can turn into accidental memory tests for your host, too, and with a large enough population you'll see interesting failures).
- baruch 14y ago
  At the storage level there is T10 DIF that is supposed to help with such things (among many other things), though it is used in a very limited fashion and is only supported on the higher end disks.

141 points by pwg 14y ago

I wish I could post the video from the talk, but here are some slides at least. Good supporting talk on the pitfalls of making assumptions about how disks work, from OSCON '11: http://www.slideshare.net/iammutex/what-every-data-programmer-needs-to-know-about-disks http://www.slideshare.net/iammutex/what-every-data-programme...

ChuckMcM 14y ago

During my tenure at NetApp I got to see all sorts of really really interesting disk problems and the lengths software had to go to reliably store data on them. Two scars burned into me from that were 1) disks suck, and 2) disks are not 'storage'. The first part is pretty easy to understand, storage manufacturers have competed for years in a commodity market where consumers often choose price per gigabyte over URER (the unrecoverable read error rate), further at the scale of the market small savings of cents adds up to better margins. And while the 'enterprise' fibre channel and SCSI drives could (and did) command a hefty premium, the shift to SATA drives really put a crimp in the over all margin picture. So the disk manufacturers are stuck between the reliability of the drive and the cost of the drive. They surf the curve where it is just reliable enough to keep people buying drives. This trend bites back, making the likely hood of an error while reading more and more probable. Not picking on Seagate here, they are all similar, but lets look at their Barracuda drive's spec sheet [1]. You will notice a parameter 'Nonrecoverable Read Errors per Bits read', and you'll see that its 1x10e14 across the sheet, from 3TB down to 250GB. It is a statistical thing, the whole magnetic field domain to digital bit pipeline is one giant analog loop of a error extraction. 1x10E14 bits. So what does that mean? Lets say each byte is encoded in 10 bits on the disk. Three trillion bytes is 30 trillion bits in that case, or 3x10E13 bits. Best case, if you read that disk from one end to the other (like you were re-silvering a mirror from a RAID 10 setup) you have a 1 in 3 chance that one of those sectors won't be readable for a perfectly working disk. Amazing isn't it? Trying to reconstruct a RAID5 group with 4 disks remaining out of 5, look out. So physics is not on your side, but we've got RAID 6 Chuck! Of course you do, and that is a good thing. But what about when you write to the disk, the disk replies "Yeah sure boss! Got that on the platters now" but it didn't actually right it? Now you've got a parity failure waiting to bite you sitting on the drive, or worse, it does write the sector but writes it somewhere else (saw this a number of times at NetApp as well). There are three million lines of code in the disk firmware. One of the manufacturers (my memory says Maxtor) showed a demo where they booted Linux on the controller of the drive. Bottom line is that the system works mostly, which is a huge win, and a lot of people blame the OS when the drive is at fault, another bonus for manufacturers, but unless your data is in a RAID 6 group or on at least 3 drives, its not really 'safe' in the "I am absolutely positive I could read this back" sense of the word. [1] http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/disc/barracuda-ds1737-1-1111us.pdf http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/...

biot 14y ago
What were the least problematic drives in your experience?
- protomyth 14y ago
  Not original poster, but I favor WD over Seagate because lately all Seagate drives we have got (about 20) failed in a couple of months. I have had 2 WD failures this years (about 100 drives in various RAID configs). Whatever brand ships with HP netbooks is basically a failure waiting to happen (we bought 100 HP netbooks with 3G from Verizon - they are very sick of us calling).
  - mbreese 14y ago
    Everyone has their own horror stories about drive manufacturers. I personally had a bunch of WD drives fail a while ago, so I've been hesitant to buy WD. Now, this was 10-ish years ago, so I know reliability has improved, but that doesn't change my hesitation. It used to be that Maxtor (when they were around and independent) had the most reliable drives. Now, it's who knows...
    - tjoff 14y ago
      Maxtor had horrible reputation (by far the worst) among most of my friends and local forums. Things is, all drive manufacturers have batches and you might end up with 8 dead out of 10 drives for any manufacturer at any given time. The only thing you can do is try to get drives from different batches and thus trying to minimize having all eggs in one basket (what good is raid6 if all drives die?). Also when a lot of drives die you should look for other culprits as well, is the temperature OK? Is the power stable? (Vibrations?) There isn't much statistics available (that I know of) but a french store published their return-rate for drives (which of course doesn't count drives returned directly to the manufacturer) http://www.hardware.fr/articles/831-6/disques-durs.html http://www.hardware.fr/articles/831-6/disques-durs.html Or a write up in Swedish about it (probably easier to read if you don't know either language, lot of graphs): http://www.sweclockers.com/nyhet/13859-fransk-datorsajt-publicerar-rma-statistik-for-april-manad http://www.sweclockers.com/nyhet/13859-fransk-datorsajt-publ... But all in all, if you have many drives failing you've hit a bad batch or are treating your drives badly, nothing you can do about it (the former at least).

recoiledsnake 14y ago

The article glosses over the fact that fsync() itself has major issues. For example, on ext3, if you call fsync() on a single file, all file system cached data is written to disk, leading to an extreme slowdown. This led to a Firefox bug, when the sqlite db used for the awesome bar and for bookmarks called fsync() and slowed everything down. http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/ http://shaver.off.net/diary/2008/05/25/fsyncers-and-curvebal... OS X does a fake fsync() if you call it with the defaults. https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdPersistentStores.html https://developer.apple.com/library/mac/#documentation/Cocoa... fsync in Mac OS X: Since in Mac OS X the fsync command does not make the guarantee that bytes are written, SQLite sends a F_FULLFSYNC request to the kernel to ensures that the bytes are actually written through to the drive platter. This causes the kernel to flush all buffers to the drives and causes the drives to flush their track caches. Without this, there is a significantly large window of time within which data will reside in volatile memory—and in the event of system failure you risk data corruption

btrask 14y ago
So in summary, I believe that the comments in the MySQL news posting are slightly confused. On MacOS X fsync() behaves the same as it does on all Unices. That's not good enough if you really care about data integrity and so we also provide the F_FULLFSYNC fcntl. As far as I know, MacOS X is the only OS to provide this feature for apps that need to truly guarantee their data is on disk. The full post goes into detail. http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072...

scdeepak 14y ago

Me and my colleagues had worked on a solution to this problem a few years back. The results were published and is linked here. ACM digital library link - http://dl.acm.org/citation.cfm?id=2056434 http://dl.acm.org/citation.cfm?id=2056434 The paper - http://research.cs.wisc.edu/adsl/Publications/cce-dsn11.pdf http://research.cs.wisc.edu/adsl/Publications/cce-dsn11.pdf

baruch 14y ago
That's a really neat hack, but how many disks have you seen that do not adhere to their write-cache-disabled setting?

rarrrrrr 14y ago

At SpiderOak, every storage cluster writes new customer data to three RAID6 volumes on 3 different machines, which wait for fdatasync, before we consider it written. ... and before that begins, the data is added to a giant ring buffer that records the last 30 TB of new writes, on machines with UPS. If a rack power failure happens, the ring buffer is kept until storage audits complete. I think every company that deals with large data they can't lose develops appropriate paranoia. The behavior of hard drives is like decaying atoms. You can't make accurate predictions about what any one of them will do. Only in aggregate can you say something like "the half life of this pile of hardware is 12 years" or "if we write this data N times we can reasonably expect to read it it again."

baruch 14y ago
Interesting, do you do disk scrubbing to make sure the data is still out there after a while? That would be the key method to ensure you can still recover the data upon need.

kabdib 14y ago

I had fun once diagnosing a bad bit in a drive's cache memory. Writes would go out, and sometimes come back with a bit set. All the disk-resident CRCs in the world won't help if your data is mangled before it makes it to the media. File systems with end-to-end checking are good. (These can turn into accidental memory tests for your host, too, and with a large enough population you'll see interesting failures).

baruch 14y ago
At the storage level there is T10 DIF that is supposed to help with such things (among many other things), though it is used in a very limited fashion and is only supported on the higher end disks.