11 ms·
SSDs: A gift and a curse
- justinsb 11y agoI am looking forward to the day when all SSDs ship minimal firmware, and offload all the complex work to (main-CPU) software.
- rwmj 11y agoUnfortunately that day isn't going to happen. In fact the opposite is happening -- interesting spinning rust technology like Shingled Magnetic Recording (SMR) drives are getting more complex translation layers.
- deleted 11y ago[deleted]
- Turing_Machine 11y agoI wouldn't say that it's never going to happen based on current trends. This kind of thing tends to be quasi-cyclic. At one time, desktop computers had separate FPUs. Over time, those got integrated onto the main CPU, but then the trend switched back, with heavy calculations moving off-CPU to the GPU. The mainframe->desktop->cloud story is similar. I say "quasi"-cyclic because (e.g.) cloud isn't really the same as mainframe, but the transition back and forth between centralized and decentralized is still striking nonetheless.
- marssaxman 11y agoI'm eagerly awaiting the part of the cycle where the GPU gets folded back into the processor. I'm really not interested in writing code against mystery-meat architectures with no documented ABI...
- gcr 11y agoWe're already seeing that with integrated, "soldered-on" graphics cards, and with frameworks like OpenCL, and higher-level abstractions like Theano, Torch7, etc.
- marssaxman 11y agoAll we need now is documentation for the low-level abstractions so we can scrap all the vendor code and compile our own GPU machine language.
- greggyb 11y agoI am not sure based on reading your post whether or not I am providing new information to you: Keep an eye on the AMD APUs moving forward and their HSA progress.
- marssaxman 11y agoThanks, that sounds like an interesting development.
- nine_k 11y agoSSDs that use SATA aren't going to allow any finer control, I suspect, due to the interface's limitations. PCIe SSD devices, though, already feature an advanced controller (that is, a CPU core). This core could be programmed from the "computer" side somehow — but building any public API is always a major commitment, and takes much more effort than a proprietary API which you can change at will.
- Retric 11y agoIn the long term latency makes that a non starter.
- wtallis 11y agoHaving the flash translation layer on the drive itself is the only practical way for the drive to be usable by multiple operating systems. Almost everyone who uses a SSD wants it to be accessible to at least two operating systems (UEFI, and whatever OS resides on the SSD). Doing the FTL on the host system would basically relegate the SSD to being just a cache device.
- zokier 11y agoWe have these things called standards which allow nice things like interoperability. Besides, you do not need the entire drive to be accessible from UEFI, it is enough that you can load a small bootloader which contains the drivers to handle the disk. Thirdly, UEFI has extensible right in the name, shouldn't be too hard for OSes to throw a disk driver blob into UEFI.
- bduerst 11y agoDoesn't that require partitioning the disk? That doesn't seem like an OOB solution.
- wtallis 11y agoIt's not the requirement to partition the drive that kills the idea. Setting aside the first small chunk of the drive for the firmware to read is how it's always been done. The problem is that this would require the partitioning to be done below the wear leveling layer, which reduces the effectiveness of the wear leveling slightly and means you can only change your bootloader settings a thousand times before the drive is dead.
- pgeorgi 11y agosadly with NAND even reads contribute to wear.
- Dylan16807 11y agoThat's not really what kills it. You can use a small ring buffer to boost that to 100 thousand, and/or use the flash in a more reliable way there (like SLC or devoting 2/3 of the bits to ECC).
- VLM 11y agoThe user experience of win-modems, win-printers, and win-scanners was absolutely horrific so you really need to click edit and add the sarcasm tag.
- ebbv 11y agoThis is a bad idea for a lot of reasons. A well designed system has components that just work and have standardized interfaces. What you're suggesting is that each device should be totally unusable unless the machine has the specific software running that knows how to use it. That's poor design.
- zanny 11y agoThats how... pretty much every peripheral in existance works. You need webcam drivers. Video card drivers. Sound card drivers. USB drivers, at the least for generic HID devices. Nothing stops there from being common standard generic SSD drivers, where vendors could publish their own if they had an optimization. Right now the SATA interface prevents anything like that from happening, and the market has already moved to these proprietary broken controllers with braindead GL-class protocols to access them. (SATA, NVMe)
- wtallis 11y agoAny such interface for SSDs would need to be able to work with a dozen different kinds of flash, a few sizes for each, a wide range of quantities of flash chips, all while exposing the performance and endurance characteristics of the flash and the topology of its connection to the host bus so that the software FTL can properly tune itself. You're asking for a hardware interface of unprecedented complexity and cross-platform vendor neutral drivers of comparable complexity to 3d graphics drivers, all for the sake of letting SSDs have "dumb" controllers for a generation or two before the technology changes enough that the controllers need to start translating things again.
- AYBABTME 11y agoWouldn't that make performance a lot worse, for both CPU and disks? Offloading the job to something closer to the flash chips and fully dedicated to servicing them sounds like a better (and the current) path. Same for high speed network links.
- rwmj 11y agoThis is appropriate: http://www.catb.org/jargon/html/W/wheel-of-reincarnation.html http://www.catb.org/jargon/html/W/wheel-of-reincarnation.htm...
- AYBABTME 11y agoAlways have that in the back of my mind =]. However I don't think that's the current trend right now. Except where special hardware is integrated to the chip, but the work isn't left to the CPU itself. I'm not so versed in the area so I'm just blind guessing here.
- wtallis 11y agoFlash translation layers don't really need to be offloaded for the sake of performance. You need some kind of controller to multiplex access to a dozen or more flash chips from a narrow PCIe link, so you might as well throw in an ARM core to do the translation and wear leveling and garbage collection, and then you can stick a few supercaps in the drive too and have a robust hardware platform.
- wmf 11y agoThe FTL runs much faster on a 3GHz Xeon core than on a 150MHz ARM core. Of course, the cost and power consumption is also much higher.
- bbulkow 11y agoFusionIO cards run with FTL maintenance on the main CPU. Run a workload and you'll see the driver spinning in a way that isn't accounted for by your operations. It's not pretty - at high workloads, the network interrupt traffic starts interfering with all the bus traffic to the cards. A controller can be designed to run exactly at the speed of the chips, no more no less, leaving the CPU doing what it does best. That being said, if you build your system correctly, there is not any extra CPU load created by maintaining the FTL at app level. The main problem is the lack of an API for wear detection. I have discussed this with senior chip designers at the major manufacturers, and they are all loath to nail down to one api, as they believe how wear is specified will have to change over time. That's a broad brush - some manufacturers are more open to these discussions than others. I'm still working on them.
- nqzero 11y agothis would be a huge win for database design - FTL is incredibly wasteful in terms of unnecessary block erases for cache
- sliverstorm 11y agoWelcome back to driver hell. We thought you weren't ever coming back, but here you are!
- moonbug 11y agoIf you think that, I've a crate of ST-506s to sell you.
- ColinDabritz 11y agoWhat a wonderfully informative article. I really appreciated all the specific scenarios and cases.
- mavhc 11y agoAll these drives were on hardware RAID cards it seems, is it feasible to do without them?
- Freaky 11y agoPreferable, I'd say. The last thing you want between your OS and your disks is a buggy unknowable black box you can only talk to with some crappy binary blob management tools.
- bbulkow 11y agoI recommend using the RAID card as an HBA (JBOD), turn on the fast path, and use software that can share multiple devices evenly.
- mrb 11y agoOf course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash chips on a PCB, take the controller's manufacturer's reference firmware implementation, tweak the dozens of knobs provided by the reference implementation (ignore FLUSH commands, change the amount of reserved sectors, disable this, enable that, etc), change the device ID strings to "Company Foobar SSD 2000", and release the product on the market, with minimal QA and testing. And that's exactly what happened in the last 5-7 years with dozens and dozens of companies around the world designing SSDs. But with traditional HDDs, the amount of engineering and domain-specific knowledge to manufacture and assemble the platters, moving heads, casing, etc, is such that there are only a handful of companies around the world who can do this (Seagate, WDC, Hitachi, etc). They have decades of experience doing that, so the firmware part of HDDs happen to be very robust as these companies have seen everything that can and will go wrong in an HDD. So it boils down to this: which would you trust more, an HDD firmware code base that is 20 years old, or an SSD firmware code base that is 4 years old? Combine this with the fact SSD firmware is much more complex (a flash translation layer must minimize write amplification, do wear leveling, spread writes on multiple chips, etc), and you are guaranteed that many SSDs on the market are going to be very buggy.
- jacquesm 11y agoAnd, contrary to a fly-by-night SSD manufacturer that's here today and gone tomorrow they have a reputation to protect and they suffer big time if a batch of faulty drives hits the market. See 'deathstar' and other re-runs of that concept. So they work very hard to make sure that doesn't happen to them.
- deleted 11y ago[deleted]
- VLM 11y agoOh there used to be a lot of HDD mfgr in perhaps 1985, all new products like this have an explosion of competitors, then they all get bought out or otherwise exit the market. There used to be something like one hundred car manufacturers in the USA, around a century ago! Another way to look at oligopolies is they no longer need to compete on quality or specs if they slack in unison, and the collapse and commodification of their product means they'll get nickel and dimed to death so I wouldn't expect perfection from them, they find ways to screw up. You can't "wish" or "push" commodity status on a market if mother nature and the economy in general prevent it from happening, forcing commodity style management onto a non-commodity product/market just makes a mess.
- abecedarius 11y agoSo there are bugs in drive firmware. How about security bugs? Should we expect quality drives to have had a security audit?
- zokier 11y agoYes, firmwares have security bugs too: http://spritesmods.com/?art=hddhack http://spritesmods.com/?art=hddhack
- cmdrfred 11y agoIf you are doing your security at the hard drive layer, haven't you already lost?
- abecedarius 11y agoSo another reason for whole-disk encryption is to defend against your own drive firmware? I guess that may make sense nowadays, though it hadn't occurred to me.
- rdl 11y agoThe new flash DIMMs (which thus bypass PCIe bridges and ATA layer, since they plug directly into the memory controller) are really interesting. Not a commodity yet, but seems like a case where simpler -> better -> cheaper.
- Rafert 11y agoI am surprised they haven't mentioned Crucial SSDs. With cheap drivers like the MX100 having features as power loss protection and Opal 2.0 support, I preferred these over the slightly faster Samsung products at the time.
- rosser 11y agoTurns out, the Crucial drives' "power loss protection" doesn't actually preserve data "in-flight" at the moment of power loss. It just prevents data "at rest" from being corrupted. This appears to apply to all the consumer-grade Crucial drives. See [1], money quote: "In the MX100 review, I was still under the impression that there was full power-loss protection in the drive, but my impression was wrong. The client-level implementation only guarantees that data-at-rest is protected, meaning that any in-flight data will be lost, including the user data in the DRAM buffer. In other words the M500, M550 and MX100 do not have power-loss protection -- what they have is circuitry that protects against corruption of existing data in the case of a power-loss." [1] http://www.anandtech.com/show/8528/micron-m600-128gb-256gb-1tb-ssd-review-nda-placeholder http://www.anandtech.com/show/8528/micron-m600-128gb-256gb-1...
- userbinator 11y agoThat is true of ordinary HDDs too. Until data is written to the flash or platter itself, it hasn't been actually written. People don't expect files they haven't saved to be written and magically come back later if the power goes out, but they do expect that their drives will power back up with all the data that was last written intact.
- rosser 11y agoYes, that's true. However, Crucial explicitly marketed their SSDs as having, quote, "power loss protection", pointing out the array of capacitors on the drive's PCB, and strongly implying that this feature included in-flight writes surviving a power loss (i.e., that the caps had enough capacity to enable flushing the DRAM buffer to the NAND media, much like the non Sandforce-based Intel SSDs — e.g., the 320 series from a few years back, or the current DC-3500 and -3700 drives). That, it turns out, isn't true. And that's a problem, because I and many others bought these drives on the basis of Crucial's implying they were power-loss durable. When enough people started reporting to Crucial support that their drives didn't, in point of actual fact, offer this feature, their marketing literature changed so as not to imply they did, and forum posts and reviews, like my previously-linked Anand Tech article, started pointing out that they didn't.
- yellowapple 11y agoParts of this sound more like hardware RAID controller issues than SSD issues, which is why I typically avoid hardware RAID in production environments unless there's a specific reason for it. RAID controllers tend to be buggy pieces of shit, usually implementing some RAID method that's more-or-less proprietary and unique even between different RAID cards from the same vendor (meaning that if your RAID controller fails, you might as well kiss your data goodbye, since the replacement - 9 times out of 10 - won't be able to make sense of its predecessor's RAID setup). Also, RAID6 is a bad idea, almost as bad as RAID5. There have been numerous studies and reports [0] indicating that both are very susceptible to subtle bit errors ("cosmic rays"), and this is made even worse when SSDs are involved. If you need absolute data integrity, RAID1 is your only option; if you need a balance between integrity, performance, and capacity, go with RAID10, which is still leaps-and-bounds better than RAID5/6. [0]: http://www.miracleas.com/BAARF/Why_RAID5_is_bad_news.pdf http://www.miracleas.com/BAARF/Why_RAID5_is_bad_news.pdf
- JoeAltmaier 11y agoTo expand: in my 20-year experience, no small-computer RAID 5/6 controller I have had experience with has ever saved any data, ever. All failures are hard and non-recoverable. Which begs the question: why use them at all? I mirror only.
- bbulkow 11y agoI have seen data recovery. Once. 22 years into my career, and it was still a home system. Which I believe proves your point.
- Laforet 11y agoRAIDs are in place to for your guaranteed uptime in the SLA. With the current storage density RAID5/6 is probably more fragile than single disks because UREs are very likely during a rebuild. Nonetheless having a degraded array is probably better than having an offline system and it will buy you some time to migrate. Mirrors are ideal, but it is hard to justify the upfront cost.
- Lagged2Death 11y agoOn the upside, [the ridiculously expensive HP SSDs] do have fancy detailed stats (like wear levelling) exposed via the controller and ILO, and none have failed yet almost 3 years on (in fact, they’re all showing 99% health). You get what you pay for, luckily. Call me a huge cynic if you must, but given the other problems observed, I think there's a really simple explanation for perfectly uniform "99% health" after three years of service that doesn't involve "you get what you pay for."
- wtallis 11y agoA SSD can be 85% of the way through its lifespan and still be operating with the same performance and reliability expectations it had when it was almost new, so the drive can reasonably be said to still be healthy. It's only when it has to start retiring bad blocks and expending the spare area that the drive is operating in a degraded mode, and it's not until you hit that point that the drive can start making accurate predictions about how much more use it can take until it dies.
- baruch 11y agoNot quite. An SSD at 85% wearout will take longer to program and erase the blocks and it will require more refreshes (data retention suffers) so there will be considerably more background operations. In fact SSD vendors do quite a bit to make the disk perform its best early on when they absolutely can and then start to break down further down the road after the benchmark is over. After all the users are mostly only testing it for a month or two and they never get to a real wearout condition during the test and then they buy in bulk and run it for a few years but by that time the SSD model is already outdated anyway (about 18 month cycle for SSD models).
- MaulingMonkey 11y ago"There's already been silent failures"?
- AHHspiders 11y agoI bought two consumer intel.. 80gb? Ssds. They both lasted less than a year before they stopped posting or allowing a system boot on separate devices. The 850 pro is ok, but it's slowed down a lot lately. Might be an OS thing, which i doubt. All in all i keep a redundant backup on old school hdds too since the failure rate of SSDs isn't so great in my experiences so far. Anyone try one of the newer M2's yet? Or i think i mean the pci-E types?
- deelowe 11y agoIsn't running raid1/5/6 on ssds silly b/c they'll all die at the same time? And hardware raid on top of that? Why? SSDs have a fairly consistent failure curve (exusing firmware bugs and other random events) for a given model, so they'll wear evenly in a raid setup. This means they'll all die at the same time as writes/reads are distributed fairly evenly across the disks. Given the size of today's drives, you may not complete a rebuild before losing another disk. Has this been proven to not be true within the past few years? I don't run redundant raid on ssds. It's either raid0 or jbod.
- GeorgeBeech 11y agoNot necessarily. We've been running Intel SSDs in productions at Stack Exchange for 4+ years, and just recently had our first 2.5" drive die. That said, most of the drives in this article are consumer drives. The problem with consumer drives is that they don't have capacitors. And since your writes are cached by the drive before they go to the NAND, if you lose power all of your drives will be corrupted in the exact same way at the exact same time. If you don't care about the data, go ahead and use them. If you do pay the extra for Enterprise drives. They really aren't _that_ much more expensive these days.
- deelowe 11y agoInteresting. Do you have more info on what you mean by "Not necessarily?" From what I've seen during reliability studies on SSDs, they have a fairly tight failure curve. This is very dissimilar from hard disks where there's much more variance from drive to drive. I'm genuinely interested.
- GeorgeBeech 11y agoNothing that would pass deep scrutiny. Just our experience running SSDs in almost every server at Stack. We've only had one mass failure of drives. That was when 5/8 Samsung drives died around the same time in our packet capture box. The remaining 3 are still alive, although we don't really use them. We have only had two Intel drives die on us. I'm interested (well academically, not professionally) if they will die at the same time or keep dropping off one at a time. We tend to retire the machines or the drives in them before they fail.
- ksec 11y agoAm I missing something? Most of the issues seems to be Hardware Raid related.
- insaneirish 11y agoMoral of the story: * Don't use hardware RAID controllers. * Don't buy hardware from people who are going to change SKUs out from under you, or worse, change what's actually delivered for a given SKU.
- simplexion 11y agoThis is exactly what I took from the article "Don't use hardware RAID controllers."
- willejs 11y agoTotally agree with you. I was just about to write the exact same thing. I've had problems with dell hardware, hardware raid controllers, and hundreds of ssds. The most stable systems I've worked on use lsi sas controllers, perhaps software raid, and Intel/crucial ssds in fault tolerant distributed systems. Therefore, I also question if it's etsy's tendency to use non distributed datastores, with less effective or complex fault tolerance makes ssd failure a bigger deal? Failure is inevitable, and should be expected at every level.