5 ms·

Of course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash

by mrb 11y ago

Of course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash chips on a PCB, take the controller's manufacturer's reference firmware implementation, tweak the dozens of knobs provided by the reference implementation (ignore FLUSH commands, change the amount of reserved sectors, disable this, enable that, etc), change the device ID strings to "Company Foobar SSD 2000", and release the product on the market, with minimal QA and testing. And that's exactly what happened in the last 5-7 years with dozens and dozens of companies around the world designing SSDs. But with traditional HDDs, the amount of engineering and domain-specific knowledge to manufacture and assemble the platters, moving heads, casing, etc, is such that there are only a handful of companies around the world who can do this (Seagate, WDC, Hitachi, etc). They have decades of experience doing that, so the firmware part of HDDs happen to be very robust as these companies have seen everything that can and will go wrong in an HDD. So it boils down to this: which would you trust more, an HDD firmware code base that is 20 years old, or an SSD firmware code base that is 4 years old? Combine this with the fact SSD firmware is much more complex (a flash translation layer must minimize write amplification, do wear leveling, spread writes on multiple chips, etc), and you are guaranteed that many SSDs on the market are going to be very buggy.

jacquesm 11y ago
And, contrary to a fly-by-night SSD manufacturer that's here today and gone tomorrow they have a reputation to protect and they suffer big time if a batch of faulty drives hits the market. See 'deathstar' and other re-runs of that concept. So they work very hard to make sure that doesn't happen to them.
- deleted 11y ago
  [deleted]
VLM 11y ago
Oh there used to be a lot of HDD mfgr in perhaps 1985, all new products like this have an explosion of competitors, then they all get bought out or otherwise exit the market. There used to be something like one hundred car manufacturers in the USA, around a century ago! Another way to look at oligopolies is they no longer need to compete on quality or specs if they slack in unison, and the collapse and commodification of their product means they'll get nickel and dimed to death so I wouldn't expect perfection from them, they find ways to screw up. You can't "wish" or "push" commodity status on a market if mother nature and the economy in general prevent it from happening, forcing commodity style management onto a non-commodity product/market just makes a mess.
zokier 11y ago
That is a nice theory, except for the fact that firmware bugs have hit big players equally badly, if not even worse. Intel and Samsung come easily to mind, and Sandforce was notorious for not getting their own firmware to work (for their own controllers).
- scott_karana 11y ago
  Since when has Intel been hit equally?
  - zokier 11y ago
    The "8MB bug" (https://communities.intel.com/thread/24205 https://communities.intel.com/thread/24205) was the most notorious, but I remember also some bricked Intel SSDs too.
    - drzaiusapelord 11y ago
      Ignoring that, Intel is rock solid. The larger issue isn't incompetents making SSDs, but SSDs being sold soley on performance. Intel has always been a step or two behind the performance game but manages to deliver some really reliable SSD's. I have yet to see a consumer Intel fail, which is incredible as that kind of reliability, outside of the super expensive PCIe cards is unheard of. I think we're a few more years of shaking out this industry and leaving behind all the growing pains. No idea who will be left standing, but I suspect Intel will probably be around.
      - pja 11y ago
        I managed to brick an Intel 320 last week. Going to see if they’ll replace it under warranty - those devices were upgraded to a 5 years so it should still be covered. (Apparently the drive really didn’t like being issued with an 'hdparm --security-set-pass X /dev/sdb' command. I was intending to do a secure wipe, for which the password setting is a pre-requisite, but the drive never came back from setting the password. After that it just returned junk data to any command, including 'hdparm -I /dev/sdb' & failed the power-on BIOS SMART test.)
JoeAltmaier 11y ago
Early SSD failures were all firmware bugs. As time goes on that has receeded into noise. Now SSD failure stats show sensible wear patterns.
noobie 11y ago
Excuse me but most of what you typed went right over my head. Where can one start learning about SSD/HDD firmware and most of what you said?
- aguki 11y ago
  The OpenSSD project[1] might be worth a look. If you don't fancy paying thousands of dollars, you can buy SSD controller boards and raw NAND flash chips off Taobao and assemble them together. There's plenty of information on various Chinese forums (Mydigit, Upan etc.), admittedly more towards USB flash drives. Some older flash chips are TSOP rather than BGA which makes hand soldering much easier. [1] http://www.openssd-project.org/wiki/The_OpenSSD_Project http://www.openssd-project.org/wiki/The_OpenSSD_Project
- 72deluxe 11y ago
  Firmware is the software on the hardware, basically. If you take a hard disk and look underneath it, there is a PCB with chips on it. The firmware lives in there and is responsible for doing the job of reading/writing data to the disk in a specific format. If you take the platters out of one hard disk and put it into another hard disk with different firmware, the other hard disk PCB won't be able to read it or write to it. I found this out by having a dead Hitachi drive (yes, the IBM "Deathstar" disks after they were sold to Hitachi) so I bought another identical drive and swapped PCBs but on startup the read/write head would not settle as there was a mismatch between the PCB firmware number and the firmware identifier written on the start of the disk. So, although I thought they were identical disks they were not.
zanny 11y ago
Sounds almost like having free software SSD firmware would be a nice thing. I just love how I have microcontrollers on all my persistent storage devices now that operate as a black box made by incompetent or possibly malicious developers.
- theatrus2 11y ago
  You've always had this.
  - pjc50 11y ago
    Not back in the days of loading programs from cassette tape I didn't.
- deleted 11y ago
  [deleted]
kevin_thibedeau 11y ago
Not with Seagate. I have a drive waiting to be fixed because of their silly BSY bug that renders a drive inaccessible. I'll never buy their garbage again.
Klinky 11y ago
From personal experience I would trust neither to be infallible. I remember the IBM Deathstar drives, my Maxtor drives repeatedly dying along with their RMA replacements & the issues Seagate seems to have with 1.5TB & 3TB drives. When was the last time you had an SSD die from a dead motor or crashed heads? Do you have intimate knowledge of the firmware writing process for hard drive manufacturing? Do you think knobs aren't twisted and codebases aren't rewritten to take into account the changes required for the latest in magnetic storage? Geometry calculations for perpendicular and shingled media is less complicated than a flash translation layer? Also just because something is old and hasn't broken yet, doesn't mean it's well written, bug free or will work in the future. This isn't to say that hard drives are reliable, as they're notoriously the least reliable thing in your computer.
dragontamer 11y ago
50%+ failure rate on 3TB Seagates from Backblaze studies. Something _horrible_ happened on the 3TB Seagate batch. 4TB and 2TB are reasonable... but 3TB... wtf happened?
- semi-extrinsic 11y ago
  This. I had one at work fail on me after 10 months of usage; got almost everything back from backups, but the timing was incredibly bad and I had to work some really late nights to pull through. Got a free replacement, but I'm pretty distrusting of it so now it's in Raid1 with a WD drive. That may've been a decent one though, because it's been working for two years now.
yuhong 11y ago
I think StorageReviews.com has an article on exactly this.
lewisl9029 11y ago
It would be interesting if SSD firmware were all open-sourced. Manufacturers could then eventually converge all of their development and QA efforts on a few different codebases designed for different tradeoffs rather than everybody rolling their own thing and learning everybody else's lessons the hard way. But I guess as long as custom firmware remains a major competitive advantage, this would never happen.

by mrb 11y ago

jacquesm 11y ago

And, contrary to a fly-by-night SSD manufacturer that's here today and gone tomorrow they have a reputation to protect and they suffer big time if a batch of faulty drives hits the market. See 'deathstar' and other re-runs of that concept. So they work very hard to make sure that doesn't happen to them.

deleted 11y ago
[deleted]

VLM 11y ago

Oh there used to be a lot of HDD mfgr in perhaps 1985, all new products like this have an explosion of competitors, then they all get bought out or otherwise exit the market. There used to be something like one hundred car manufacturers in the USA, around a century ago! Another way to look at oligopolies is they no longer need to compete on quality or specs if they slack in unison, and the collapse and commodification of their product means they'll get nickel and dimed to death so I wouldn't expect perfection from them, they find ways to screw up. You can't "wish" or "push" commodity status on a market if mother nature and the economy in general prevent it from happening, forcing commodity style management onto a non-commodity product/market just makes a mess.

zokier 11y ago

That is a nice theory, except for the fact that firmware bugs have hit big players equally badly, if not even worse. Intel and Samsung come easily to mind, and Sandforce was notorious for not getting their own firmware to work (for their own controllers).

scott_karana 11y ago
Since when has Intel been hit equally?
- zokier 11y ago
  The "8MB bug" (https://communities.intel.com/thread/24205 https://communities.intel.com/thread/24205) was the most notorious, but I remember also some bricked Intel SSDs too.
  - drzaiusapelord 11y ago
    Ignoring that, Intel is rock solid. The larger issue isn't incompetents making SSDs, but SSDs being sold soley on performance. Intel has always been a step or two behind the performance game but manages to deliver some really reliable SSD's. I have yet to see a consumer Intel fail, which is incredible as that kind of reliability, outside of the super expensive PCIe cards is unheard of. I think we're a few more years of shaking out this industry and leaving behind all the growing pains. No idea who will be left standing, but I suspect Intel will probably be around.
    - pja 11y ago
      I managed to brick an Intel 320 last week. Going to see if they’ll replace it under warranty - those devices were upgraded to a 5 years so it should still be covered. (Apparently the drive really didn’t like being issued with an 'hdparm --security-set-pass X /dev/sdb' command. I was intending to do a secure wipe, for which the password setting is a pre-requisite, but the drive never came back from setting the password. After that it just returned junk data to any command, including 'hdparm -I /dev/sdb' & failed the power-on BIOS SMART test.)

JoeAltmaier 11y ago

Early SSD failures were all firmware bugs. As time goes on that has receeded into noise. Now SSD failure stats show sensible wear patterns.

noobie 11y ago

Excuse me but most of what you typed went right over my head. Where can one start learning about SSD/HDD firmware and most of what you said?

aguki 11y ago
The OpenSSD project[1] might be worth a look. If you don't fancy paying thousands of dollars, you can buy SSD controller boards and raw NAND flash chips off Taobao and assemble them together. There's plenty of information on various Chinese forums (Mydigit, Upan etc.), admittedly more towards USB flash drives. Some older flash chips are TSOP rather than BGA which makes hand soldering much easier. [1] http://www.openssd-project.org/wiki/The_OpenSSD_Project http://www.openssd-project.org/wiki/The_OpenSSD_Project
72deluxe 11y ago
Firmware is the software on the hardware, basically. If you take a hard disk and look underneath it, there is a PCB with chips on it. The firmware lives in there and is responsible for doing the job of reading/writing data to the disk in a specific format. If you take the platters out of one hard disk and put it into another hard disk with different firmware, the other hard disk PCB won't be able to read it or write to it. I found this out by having a dead Hitachi drive (yes, the IBM "Deathstar" disks after they were sold to Hitachi) so I bought another identical drive and swapped PCBs but on startup the read/write head would not settle as there was a mismatch between the PCB firmware number and the firmware identifier written on the start of the disk. So, although I thought they were identical disks they were not.

zanny 11y ago

Sounds almost like having free software SSD firmware would be a nice thing. I just love how I have microcontrollers on all my persistent storage devices now that operate as a black box made by incompetent or possibly malicious developers.

theatrus2 11y ago
You've always had this.
- pjc50 11y ago
  Not back in the days of loading programs from cassette tape I didn't.
deleted 11y ago
[deleted]

kevin_thibedeau 11y ago

Not with Seagate. I have a drive waiting to be fixed because of their silly BSY bug that renders a drive inaccessible. I'll never buy their garbage again.

Klinky 11y ago

From personal experience I would trust neither to be infallible. I remember the IBM Deathstar drives, my Maxtor drives repeatedly dying along with their RMA replacements & the issues Seagate seems to have with 1.5TB & 3TB drives. When was the last time you had an SSD die from a dead motor or crashed heads? Do you have intimate knowledge of the firmware writing process for hard drive manufacturing? Do you think knobs aren't twisted and codebases aren't rewritten to take into account the changes required for the latest in magnetic storage? Geometry calculations for perpendicular and shingled media is less complicated than a flash translation layer? Also just because something is old and hasn't broken yet, doesn't mean it's well written, bug free or will work in the future. This isn't to say that hard drives are reliable, as they're notoriously the least reliable thing in your computer.

dragontamer 11y ago

50%+ failure rate on 3TB Seagates from Backblaze studies. Something _horrible_ happened on the 3TB Seagate batch. 4TB and 2TB are reasonable... but 3TB... wtf happened?

semi-extrinsic 11y ago
This. I had one at work fail on me after 10 months of usage; got almost everything back from backups, but the timing was incredibly bad and I had to work some really late nights to pull through. Got a free replacement, but I'm pretty distrusting of it so now it's in Raid1 with a WD drive. That may've been a decent one though, because it's been working for two years now.

yuhong 11y ago

I think StorageReviews.com has an article on exactly this.

lewisl9029 11y ago

It would be interesting if SSD firmware were all open-sourced. Manufacturers could then eventually converge all of their development and QA efforts on a few different codebases designed for different tradeoffs rather than everybody rolling their own thing and learning everybody else's lessons the hard way. But I guess as long as custom firmware remains a major competitive advantage, this would never happen.