3 ms·

"It feels like the author's main complaint is the frustration of not understanding SSD hardware as well." What is so frustrating about SSDs is how very poorly

by rsync 8y ago

"It feels like the author's main complaint is the frustration of not understanding SSD hardware as well." What is so frustrating about SSDs is how very poorly they compare to previous incarnations of solid state storage. Using Disk-On-Chip and/or IDE-pin-compatible CF cards, I had many, many devices in the field that lasted, mounted read-only, for decades An entire sect of the computing industry came to rely on these parts as alternatives to spinning media that could not mechanically fail. This is not the case with SSDs at all. They fail left and right, even mounted read-only, for all manner of complicated and interesting reasons. It's very frustrating that SSDs are not a step forward in reliability from spinning media and are a step downward compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000. rsync.net filers, which need a boot mirror, are always constructed with two unrelated SSDs - usually one Intel part and one Samsung part - so that when the inevitable usage-related failure occurs, it does not occur simultaneously to both members of the mirror which have, being a mirror, been subjected to identical usage-lives.[1] We shouldn't have to do that. [1] I can't overstate this - if you need a RAID mirror, do not use identical SSDs for the two members of the mirror. There are many, many cases of SSDs failing not due to "wear" or end-of-life, but due to weird usage edge cases that cause them to puke ... and in a mirror, you give the two parts identical usage ... we either get two different generations of Intel part (current gen and just-previous gen) or we get current Intel and current Samsung ...

Mister_Snuggles 8y ago
I've heard very similar advice for non-SSD mirrors too. Use different manufacturers or, at the very least, use different batches of disks from the same manufacturer.
- blattimwind 8y ago
  Most importantly, never assemble an array from drives that took the same shipping path i.e. came out of the same parcel.
- lsc 8y ago
  Most people use different batches but the same manufacturer, mostly because they remember advice from back when you could get a hardware raid card that would synchronize your scsi drives. Also, though, because most serious RAIDs contain more drives than you can find manufacturers of hard drives.
asdfadsfgfdda 8y ago
On modern MLC/TLC SSDs, read-only mode doesn't really exist. The NAND blocks must be re-programmed after a number of read accesses to mitigate read disturb. If anything, mounting read-only is probably a corner case, stressing the firmware's read disturb mitigation. The tradeoff from worse read disturb characteristics is NAND that is 100x cheaper per GB than in 2000.
- vkuruthers 8y ago
  Do you have any more details on the re-programming that would be occurring on non SLC flash cells, even if mounted in read only mode? This is something I was always concerned about too.
endorphone 8y ago
Every failure of an SSD feels like an exceptional event. Some harbinger of doom that needs to be shouted from the rooftops. The prions of storage. But magnetic hard drives failed all the time. I have a giant stack in my office closet just from my dev machines over the years. But it wasn't new and scary -- it was just a hard drive failing -- so it was just normal. Some had controllers fail, suddenly blinking out of existence. Another had cache memory corrupt so it just gave ridiculous readings occasionally. Others had physical failures. I don't know where to begin relative to prior flash memory (e.g. CF cards) which were absolutely notorious trash. It is worth noting that every smartphone the world over has an "SSD" in it. We spend remarkably little of our mental power concerned about the flash storage. It is the cause of a negligible amount of device failures.
amatecha 8y ago
Yeah agreed 100%, for my RAID mirror setup I use drives from distinct manufacturers for that exact reason -- I can presumably expect a different failure rate (hopefully). :)
userbinator 8y ago
compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000. That would almost certainly be small-block SLC flash rated for 100K program/erase cycles and 10 years of retention. The huge-block TLC now is ~1K program/erase cycles and 2-3 years of of retention depending on where you look (the manufacturers are not surprisingly quite reluctant to release these specs...)
nickpsecurity 8y ago
"We shouldn't have to do that." I don't know about that. I know most managers in embedded software go cheap on engineers and software assurance on purpose to get bonuses and such. The hardware side makes me think you haven't studied deep, sub-micron hardware much. I started looking into it a few years ago or so, just reading the slides on lots of stuff even though not understanding much of it. They helpfully put a lot in lay terms, though, with lots of comparisons. To say the stuff gets harder every time you shrink to a smaller node is an understatement, esp for solid state. If anything, the modern flash should probably be considered broken right as they ship out of the factory. If not, the process nodes after 90nm or so just keep having more and more ways for individual components to screw up or change behavior across same wafer. Some happen instantly by design. Some happen later with aging. The memory technologies are closer to that analog level of things than most with harder verification. The high-density flash on newer nodes uses less-reliable tech than most just to operate at that low cost. So, they add all kinds of firmware and hardware tricks to try to make it work for a period of time like it's a whole, functional unit of storage despite pieces of it misbehaving all throughout. It's a nice, man-made miracle these techs even work at all. Those that last longer like you mentioned still exist. I'll add a comment with links to one type so you can compare price/storage/performance to these broken-by-design SSD's you use. I'll throw in another two that mention shrinking challenges so you can see what they face every time they have to upgrade or just deploy new designs in mixed-signal. https://news.ycombinator.com/item?id=18528023 https://news.ycombinator.com/item?id=18528023 https://www.electronicdesign.com/digital-ics/understanding-28-nm-soc-design-arm-based-cores https://www.electronicdesign.com/digital-ics/understanding-2... https://anysilicon.com/ic-design-impact-in-moving-from-28nm-to-16nm/ https://anysilicon.com/ic-design-impact-in-moving-from-28nm-...

by rsync 8y ago

Mister_Snuggles 8y ago

I've heard very similar advice for non-SSD mirrors too. Use different manufacturers or, at the very least, use different batches of disks from the same manufacturer.

blattimwind 8y ago
Most importantly, never assemble an array from drives that took the same shipping path i.e. came out of the same parcel.
lsc 8y ago
Most people use different batches but the same manufacturer, mostly because they remember advice from back when you could get a hardware raid card that would synchronize your scsi drives. Also, though, because most serious RAIDs contain more drives than you can find manufacturers of hard drives.

asdfadsfgfdda 8y ago

On modern MLC/TLC SSDs, read-only mode doesn't really exist. The NAND blocks must be re-programmed after a number of read accesses to mitigate read disturb. If anything, mounting read-only is probably a corner case, stressing the firmware's read disturb mitigation. The tradeoff from worse read disturb characteristics is NAND that is 100x cheaper per GB than in 2000.

vkuruthers 8y ago
Do you have any more details on the re-programming that would be occurring on non SLC flash cells, even if mounted in read only mode? This is something I was always concerned about too.

endorphone 8y ago

Every failure of an SSD feels like an exceptional event. Some harbinger of doom that needs to be shouted from the rooftops. The prions of storage. But magnetic hard drives failed all the time. I have a giant stack in my office closet just from my dev machines over the years. But it wasn't new and scary -- it was just a hard drive failing -- so it was just normal. Some had controllers fail, suddenly blinking out of existence. Another had cache memory corrupt so it just gave ridiculous readings occasionally. Others had physical failures. I don't know where to begin relative to prior flash memory (e.g. CF cards) which were absolutely notorious trash. It is worth noting that every smartphone the world over has an "SSD" in it. We spend remarkably little of our mental power concerned about the flash storage. It is the cause of a negligible amount of device failures.

amatecha 8y ago

Yeah agreed 100%, for my RAID mirror setup I use drives from distinct manufacturers for that exact reason -- I can presumably expect a different failure rate (hopefully). :)

userbinator 8y ago

compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000. That would almost certainly be small-block SLC flash rated for 100K program/erase cycles and 10 years of retention. The huge-block TLC now is ~1K program/erase cycles and 2-3 years of of retention depending on where you look (the manufacturers are not surprisingly quite reluctant to release these specs...)

nickpsecurity 8y ago

"We shouldn't have to do that." I don't know about that. I know most managers in embedded software go cheap on engineers and software assurance on purpose to get bonuses and such. The hardware side makes me think you haven't studied deep, sub-micron hardware much. I started looking into it a few years ago or so, just reading the slides on lots of stuff even though not understanding much of it. They helpfully put a lot in lay terms, though, with lots of comparisons. To say the stuff gets harder every time you shrink to a smaller node is an understatement, esp for solid state. If anything, the modern flash should probably be considered broken right as they ship out of the factory. If not, the process nodes after 90nm or so just keep having more and more ways for individual components to screw up or change behavior across same wafer. Some happen instantly by design. Some happen later with aging. The memory technologies are closer to that analog level of things than most with harder verification. The high-density flash on newer nodes uses less-reliable tech than most just to operate at that low cost. So, they add all kinds of firmware and hardware tricks to try to make it work for a period of time like it's a whole, functional unit of storage despite pieces of it misbehaving all throughout. It's a nice, man-made miracle these techs even work at all. Those that last longer like you mentioned still exist. I'll add a comment with links to one type so you can compare price/storage/performance to these broken-by-design SSD's you use. I'll throw in another two that mention shrinking challenges so you can see what they face every time they have to upgrade or just deploy new designs in mixed-signal. https://news.ycombinator.com/item?id=18528023 https://news.ycombinator.com/item?id=18528023 https://www.electronicdesign.com/digital-ics/understanding-28-nm-soc-design-arm-based-cores https://www.electronicdesign.com/digital-ics/understanding-2... https://anysilicon.com/ic-design-impact-in-moving-from-28nm-to-16nm/ https://anysilicon.com/ic-design-impact-in-moving-from-28nm-...