4 ms·

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day whe

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot. If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC. Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them". They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things. Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system". They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing. But that was exciting. [1] - https://github.com/openzfs/zfs/pull/15588 https://github.com/openzfs/zfs/pull/15588

rincebrain 3y ago
A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them. Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired... This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing. Huh. So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it. So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up. The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that. I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.
- db48x 3y ago
  Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.
- rincebrain 3y ago
  One more for the road. So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons. The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus. Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun. One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked. I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot. The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features. The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic. But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.

rincebrain 3y ago

A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them. Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired... This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing. Huh. So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it. So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up. The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that. I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.

db48x 3y ago
Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.
rincebrain 3y ago
One more for the road. So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons. The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus. Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun. One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked. I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot. The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features. The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic. But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.