8 ms·

A major problem with SSDs seems to be “firmware death” - where the flash chips are physically fine (or mostly fine), but the firmware (or firmware memory) has g

A major problem with SSDs seems to be “firmware death” - where the flash chips are physically fine (or mostly fine), but the firmware (or firmware memory) has gotten corrupted due to some programming error, electrical glitch, or cosmic ray. I’ve had scores of older SSDs die after things like power outages and sudden shutdown events. This is super frustrating because the data is physically OK but the controller just isn’t responding to any requests anymore. I wonder if there’s an easy way to distinguish a controller failure from a flash failure from the behavior of the device over the last few seconds/minutes of operation. In theory a controller failure should cause a fairly abrupt loss of service, but I’m sure there are soft lockup failure modes too.

darkmighty 8y ago
If that's the case, I wonder if controller failure could be prevented by ECC controller memory? Or would software failure recovery be sufficient to make a highly reliable controller?
- astrodust 8y ago
  With drives you used to be able to rip off a controller from an identical model and wire it back on to read the data. With SSD or NVMe the controller isn't really a separate component you can just replace. Maybe it's possible to saw off the broken part and bodge-wire it to a working surrogate, but that would be extreme. A tear-down of a broken SSD might reveal more about what could be done.
mkirklions 8y ago
Does this explain why I often can access my 'dead' SSDs with my Ubuntu install rather than Windows? I havent run into this issue for ~4 years, so I'm not sure if this is a solved problem or I got lucky a few years back.
- soundwave106 8y ago
  Several years ago I was able to revive a "dead" SSD simply by updating the firmware to the latest version. I haven't had a dead SSD problem in a while though, so I don't know how common firmware issues are now.
  - nneonneo 8y ago
    FWIW I switched entirely to using Samsung’s SSDs and haven’t had any issues. They seem to be above average for firmware quality, but I don’t have statistics to back that feeling up. Sometimes the dead SSDs will respond to a handful of commands anyway, meaning that you can attempt a firmware upgrade or reset. That’s the lucky case. But in the majority of cases I’ve seen, the firmware/controller goes into some hard lockup where it no longer processes SATA commands at all. I wish these things had JTAG ports...
- pnutjam 8y ago
  I have a laptop where the SSD wouldn't accept a Windows install, but worked fine with Linux. Months later, it took Windows with no issue...
sebazzz 8y ago
I have seen some weird issues with SSDs. I had an OCZ Vertex 2 die on me multiple times, but one thing that stood out most is that after a power cycle or complete system shutdown (note: reboots were just fine) everything I had done last time - install software, update Windows, create files - was gone. The state was reverted to before the time it booted. It was like my computer contained some kind of Reborn chip, except it was the Sandforce controller malfunctioning.
- ricardobeat 8y ago
  Modern SSDs can have very large DRAM write buffers (8-256MB), this is pretty plausible if it was failing to flush it.
  - Skunkleton 8y ago
    It was probably some kind of wear leveling metadata that didnt get flushed.
- deleted 8y ago
  [deleted]
- yzmtf2008 8y ago
  Modern SSDs have incredibly large caches. For example, HP EX920, which is a TLC (triple layer cell) SSD, in its 1TB model contains a whooping 200GB of SLC (single layer cell) cache. It's entirely possible that your changes are simply in cache only, and hasn't been committed to the actual storage.
  - undersuit 8y ago
    I don't know if I'm getting exactly what you're saying, but a sudden power off isn't going to wipe your SSD's SLC cache like it would do the DRAM cache on more expensive drives.
    - nightfly 8y ago
      But if the controller gets into a weird state and "forgets" to flush the cache and then doesn't even look at the SLC cache on the next boot...
cimmanom 8y ago
What's involved in recovering the data if the controller fails?
- sigstoat 8y ago
  just based on having seen data recovered from other flash devices, and the advertising copy of these devices: hot-air rework to lift all the flash chips off and get them hooked up to something (probably custom) that can read them. if the controller was encrypting everything that went to flash, you also get to try and find the key in the controller's memory.
  - listic 8y ago
    why something custom? shouldn't you just solder the chips into an identical donor ssd?
    - sigstoat 8y ago
      a new controller is probably worthless to you, even if the old one wasn't storing the data encrypted. you don't have the map between real sectors and locations in flash. your data is probably jumbled up one way or the other, but at least a custom board will let you read all of the underlying flash, instead of just the portions of it that a new controller would believe are in use. (keep in mind that the SSDs will have more physical blocks than they advertise logically.)
userbinator 8y ago
but the firmware (or firmware memory) has gotten corrupted due to some programming error, electrical glitch, or cosmic ray. A lot of SSDs also store their main firmware in the same flash that is used to hold user data... this is something which was done with hard drives too (hence why dead/dying HDDs sometimes show up as a small drive with a weird name --- that's the "recovery mode").

darkmighty 8y ago

If that's the case, I wonder if controller failure could be prevented by ECC controller memory? Or would software failure recovery be sufficient to make a highly reliable controller?

astrodust 8y ago
With drives you used to be able to rip off a controller from an identical model and wire it back on to read the data. With SSD or NVMe the controller isn't really a separate component you can just replace. Maybe it's possible to saw off the broken part and bodge-wire it to a working surrogate, but that would be extreme. A tear-down of a broken SSD might reveal more about what could be done.

mkirklions 8y ago

Does this explain why I often can access my 'dead' SSDs with my Ubuntu install rather than Windows? I havent run into this issue for ~4 years, so I'm not sure if this is a solved problem or I got lucky a few years back.

soundwave106 8y ago
Several years ago I was able to revive a "dead" SSD simply by updating the firmware to the latest version. I haven't had a dead SSD problem in a while though, so I don't know how common firmware issues are now.
- nneonneo 8y ago
  FWIW I switched entirely to using Samsung’s SSDs and haven’t had any issues. They seem to be above average for firmware quality, but I don’t have statistics to back that feeling up. Sometimes the dead SSDs will respond to a handful of commands anyway, meaning that you can attempt a firmware upgrade or reset. That’s the lucky case. But in the majority of cases I’ve seen, the firmware/controller goes into some hard lockup where it no longer processes SATA commands at all. I wish these things had JTAG ports...
pnutjam 8y ago
I have a laptop where the SSD wouldn't accept a Windows install, but worked fine with Linux. Months later, it took Windows with no issue...

sebazzz 8y ago

I have seen some weird issues with SSDs. I had an OCZ Vertex 2 die on me multiple times, but one thing that stood out most is that after a power cycle or complete system shutdown (note: reboots were just fine) everything I had done last time - install software, update Windows, create files - was gone. The state was reverted to before the time it booted. It was like my computer contained some kind of Reborn chip, except it was the Sandforce controller malfunctioning.

ricardobeat 8y ago
Modern SSDs can have very large DRAM write buffers (8-256MB), this is pretty plausible if it was failing to flush it.
- Skunkleton 8y ago
  It was probably some kind of wear leveling metadata that didnt get flushed.
deleted 8y ago
[deleted]
yzmtf2008 8y ago
Modern SSDs have incredibly large caches. For example, HP EX920, which is a TLC (triple layer cell) SSD, in its 1TB model contains a whooping 200GB of SLC (single layer cell) cache. It's entirely possible that your changes are simply in cache only, and hasn't been committed to the actual storage.
- undersuit 8y ago
  I don't know if I'm getting exactly what you're saying, but a sudden power off isn't going to wipe your SSD's SLC cache like it would do the DRAM cache on more expensive drives.
  - nightfly 8y ago
    But if the controller gets into a weird state and "forgets" to flush the cache and then doesn't even look at the SLC cache on the next boot...

cimmanom 8y ago

What's involved in recovering the data if the controller fails?

sigstoat 8y ago
just based on having seen data recovered from other flash devices, and the advertising copy of these devices: hot-air rework to lift all the flash chips off and get them hooked up to something (probably custom) that can read them. if the controller was encrypting everything that went to flash, you also get to try and find the key in the controller's memory.
- listic 8y ago
  why something custom? shouldn't you just solder the chips into an identical donor ssd?
  - sigstoat 8y ago
    a new controller is probably worthless to you, even if the old one wasn't storing the data encrypted. you don't have the map between real sectors and locations in flash. your data is probably jumbled up one way or the other, but at least a custom board will let you read all of the underlying flash, instead of just the portions of it that a new controller would believe are in use. (keep in mind that the SSDs will have more physical blocks than they advertise logically.)

userbinator 8y ago

but the firmware (or firmware memory) has gotten corrupted due to some programming error, electrical glitch, or cosmic ray. A lot of SSDs also store their main firmware in the same flash that is used to hold user data... this is something which was done with hard drives too (hence why dead/dying HDDs sometimes show up as a small drive with a weird name --- that's the "recovery mode").