4 ms·

A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives tha

A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them. Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired... This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing. Huh. So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it. So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up. The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that. I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.

db48x 3y ago
Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.
rincebrain 3y ago
One more for the road. So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons. The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus. Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun. One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked. I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot. The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features. The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic. But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.

db48x 3y ago

Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.

rincebrain 3y ago

One more for the road. So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons. The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus. Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun. One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked. I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot. The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features. The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic. But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.