3 ms·

Once upon a time, a local computer club chapter I was in bought a bunch of 1Gb PCI/PCI-X Intel e1000 NICs to upgrade our systems, and a shiny used GbE switch.

Once upon a time, a local computer club chapter I was in bought a bunch of 1Gb PCI/PCI-X Intel e1000 NICs to upgrade our systems, and a shiny used GbE switch. We installed them, and a curious thing happened - on _a few_ systems, eventually the NIC would effectively hang, flooding dmesg with "Rx Unit Hang" or "Tx Unit Hang", and since the systems were NFS+LDAP based, people without local account access were SOL, and a reboot was required to get the NIC back into a good state. When swapping NICs with a system (also using an e1000) where this _never_ happened, the problem did not follow the NIC, it stayed on the original system - same statement if you duplicated the OS install from the working one onto the broken one. So we reached out to Intel's Linux support (which, at the time, took posts on their sourceforge page), and they replied basically "hey, yeah, we've heard a few reports of that, but we haven't been able to repro it on our testbeds yet". "We have two identical systems which both do it, would you like to borrow one?" Their reply was a shipping label, so we shipped it to them. A little while later, I got a phone call, and the first words out of the engineer's mouth were "Do you have the BIOS that was on this motherboard?" "...what do you mean, was?" Apparently they plugged in a PCI bus analyzer that made assumptions that were true on Intel x86 and not AMD x86, and we had shipped them a Sempron, so comedy ensued. After a number of rabbit holes, the root cause came back - our affected systems were _so_ cheap and terrible, _they lied about bus DMAs succeeding sometimes_, so sometimes the "hey I put/cleared this buffer on the NIC" messages were getting lost but not reporting they were lost, and eventually, you run out of buffers... They couldn't reproduce it because none of their testbeds were bargain-basement AMD boards. XD They asked us to keep the system for their test set, and shipped us a brand spanking new Core 2 Quad with all the bells and whistles.

rincebrain 5y ago
Another strange one: Once, when testing a whole bunch of various-grade SSDs, a curious thing happened. We had a lot of Windows servers at a prior job for a variety of reasons, so we did a lot with Windows, in addition to Linux. On Windows only, when using a SAS HBA only, a certain vendor's certain line of SSDs[1] would not show up as disks. (They showed up as "Unknown device" if you dug through Device Manager, under "Other devices", not disk drives.) If you used the onboard SATA, it was good. If you used Linux or Solaris, it worked either way. But on Windows, with a SAS HBA, nope. It turns out, the SSDs left a mandatory field blank in their identify response, and Windows cared when the field was empty on a SATL-encapsulated disk versus just normal AHCI. Once the underlying cause was identified, they cut a FW update real fast (and updated the firmware on the site without bumping the FW number, as I recall), because it also affected their enterprise drives with the same controller, and that was causing them issues... [1] - OCZ Vertex 3s
- rincebrain 5y ago
  Help I can't stop. When we were testing ZFS (at the same prior job) while I convinced my bosses to go with it for a big setup on an upcoming cluster, we bought a couple hundred of a consumer disk to play with (my workplace was obsessed with the idea of using consumer disks over enterprise ones and just using increased redundancy+backups to deal with the gap), threw them in a couple of disk enclosures, and went to play. Set up a pretty vanilla OpenIndiana system, build a big pool out of dozens of disks with raidz3 vdevs, and start writing bunches of data to them and then scrubbing, and a curious thing occurred... ...every time we scrubbed, the checksum error count on not just one or two, but _all_ of the drives would go up sometimes - never any read or write errors or even a blip out of any disk or controller, no SMART counter bump, nothing. So after a lot of experimenting (not the CPU, ECC RAM is fine, controller is fine...), it turned out... ...Samsung managed to manufacture disks that made SMART requests dangerous. [1] You see, if you issued a SMART command while there was data in the drive's write cache (which it has, remember, already told the OS that it wrote safely), it would just...drop it. So my background monitoring of the counters as I did all this IO was making writes get lost before they reached the platters, and so on scrub, then they would naturally be noticed as wrong... ...and if SMART lined up properly there too, they'd still have some on scrubbing again... The "best" part was that Samsung did cut a firmware update to fix this, eventually...but it _doesn't change the firmware version reported_, so you can't know if your drive is fixed just from looking. (We kept a spreadsheet of them and updated it as we flashed them.) [1] - https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks
  - rincebrain 5y ago
    One more bonus, for anyone who yet comes upon this. I bought a monitor, once upon a time, a Dell UP2414Q, as I recall. It was a nice monitor, when it worked. Unfortunately, there's a terrible secret in high-framerate high-resolution stream transport - for at least some protocols (hello DisplayPort), they do this by sending two video streams for halves of the screen and the monitor muxes them. Unfortunately, on this monitor, it would go...wrong, sometimes. Sometimes, it would think that the picture on the "right" half should be stretched to fit the entire width, and then the left side displayed over that on the left (or vice-versa). Sometimes, it would just display nothing for one side. Most excitingly, sometimes it would just do what I would describe as "crash" - if the monitor was allowed to go to sleep, it would sometimes just...stop responding to any inputs on the physical buttons or commands from the computer, and require a hard power removal and reinsertion of the power cable before behaving again. So you got to leave this monitor non-asleep the entire time or endure suffering. You can see a fun comparison of what a screenshot thinks is onscreen versus a photo of it in [1][2]. And all of those things could all happen any time the resolution shifted - switching to/from fullscreen in something, for example. If you look, you can find references to a revision of the monitor which ships with a firmware that lacks these problems, but there exists no documented path to update from one to the other. I could not, when I looked, find someone who had the access and desire to try and extract the firmware from the board without any sort of provided tooling to do it, and I was not going to buy monitors with that model and hope I found one of the right rev. Dell support might have been willing to replace it under warranty, but I had bought it refurbished, so "ha ha". Ultimately, I gave the monitor to someone as a gift when theirs was woefully insufficient (with complete disclosure of the flaws in it) before I tried flashing any exciting payloads. Probably the least satisfying story ending of these, but true. [1] - https://i.imgur.com/NAG5hEU.jpg https://i.imgur.com/NAG5hEU.jpg [2] - https://i.imgur.com/9R7TWQE.jpg https://i.imgur.com/9R7TWQE.jpg

rincebrain 5y ago

Another strange one: Once, when testing a whole bunch of various-grade SSDs, a curious thing happened. We had a lot of Windows servers at a prior job for a variety of reasons, so we did a lot with Windows, in addition to Linux. On Windows only, when using a SAS HBA only, a certain vendor's certain line of SSDs[1] would not show up as disks. (They showed up as "Unknown device" if you dug through Device Manager, under "Other devices", not disk drives.) If you used the onboard SATA, it was good. If you used Linux or Solaris, it worked either way. But on Windows, with a SAS HBA, nope. It turns out, the SSDs left a mandatory field blank in their identify response, and Windows cared when the field was empty on a SATL-encapsulated disk versus just normal AHCI. Once the underlying cause was identified, they cut a FW update real fast (and updated the firmware on the site without bumping the FW number, as I recall), because it also affected their enterprise drives with the same controller, and that was causing them issues... [1] - OCZ Vertex 3s

rincebrain 5y ago
Help I can't stop. When we were testing ZFS (at the same prior job) while I convinced my bosses to go with it for a big setup on an upcoming cluster, we bought a couple hundred of a consumer disk to play with (my workplace was obsessed with the idea of using consumer disks over enterprise ones and just using increased redundancy+backups to deal with the gap), threw them in a couple of disk enclosures, and went to play. Set up a pretty vanilla OpenIndiana system, build a big pool out of dozens of disks with raidz3 vdevs, and start writing bunches of data to them and then scrubbing, and a curious thing occurred... ...every time we scrubbed, the checksum error count on not just one or two, but _all_ of the drives would go up sometimes - never any read or write errors or even a blip out of any disk or controller, no SMART counter bump, nothing. So after a lot of experimenting (not the CPU, ECC RAM is fine, controller is fine...), it turned out... ...Samsung managed to manufacture disks that made SMART requests dangerous. [1] You see, if you issued a SMART command while there was data in the drive's write cache (which it has, remember, already told the OS that it wrote safely), it would just...drop it. So my background monitoring of the counters as I did all this IO was making writes get lost before they reached the platters, and so on scrub, then they would naturally be noticed as wrong... ...and if SMART lined up properly there too, they'd still have some on scrubbing again... The "best" part was that Samsung did cut a firmware update to fix this, eventually...but it _doesn't change the firmware version reported_, so you can't know if your drive is fixed just from looking. (We kept a spreadsheet of them and updated it as we flashed them.) [1] - https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks
- rincebrain 5y ago
  One more bonus, for anyone who yet comes upon this. I bought a monitor, once upon a time, a Dell UP2414Q, as I recall. It was a nice monitor, when it worked. Unfortunately, there's a terrible secret in high-framerate high-resolution stream transport - for at least some protocols (hello DisplayPort), they do this by sending two video streams for halves of the screen and the monitor muxes them. Unfortunately, on this monitor, it would go...wrong, sometimes. Sometimes, it would think that the picture on the "right" half should be stretched to fit the entire width, and then the left side displayed over that on the left (or vice-versa). Sometimes, it would just display nothing for one side. Most excitingly, sometimes it would just do what I would describe as "crash" - if the monitor was allowed to go to sleep, it would sometimes just...stop responding to any inputs on the physical buttons or commands from the computer, and require a hard power removal and reinsertion of the power cable before behaving again. So you got to leave this monitor non-asleep the entire time or endure suffering. You can see a fun comparison of what a screenshot thinks is onscreen versus a photo of it in [1][2]. And all of those things could all happen any time the resolution shifted - switching to/from fullscreen in something, for example. If you look, you can find references to a revision of the monitor which ships with a firmware that lacks these problems, but there exists no documented path to update from one to the other. I could not, when I looked, find someone who had the access and desire to try and extract the firmware from the board without any sort of provided tooling to do it, and I was not going to buy monitors with that model and hope I found one of the right rev. Dell support might have been willing to replace it under warranty, but I had bought it refurbished, so "ha ha". Ultimately, I gave the monitor to someone as a gift when theirs was woefully insufficient (with complete disclosure of the flaws in it) before I tried flashing any exciting payloads. Probably the least satisfying story ending of these, but true. [1] - https://i.imgur.com/NAG5hEU.jpg https://i.imgur.com/NAG5hEU.jpg [2] - https://i.imgur.com/9R7TWQE.jpg https://i.imgur.com/9R7TWQE.jpg