4 ms·

Help I can't stop. When we were testing ZFS (at the same prior job) while I convinced my bosses to go with it for a big setup on an upcoming cluster, we bought

Help I can't stop. When we were testing ZFS (at the same prior job) while I convinced my bosses to go with it for a big setup on an upcoming cluster, we bought a couple hundred of a consumer disk to play with (my workplace was obsessed with the idea of using consumer disks over enterprise ones and just using increased redundancy+backups to deal with the gap), threw them in a couple of disk enclosures, and went to play. Set up a pretty vanilla OpenIndiana system, build a big pool out of dozens of disks with raidz3 vdevs, and start writing bunches of data to them and then scrubbing, and a curious thing occurred... ...every time we scrubbed, the checksum error count on not just one or two, but _all_ of the drives would go up sometimes - never any read or write errors or even a blip out of any disk or controller, no SMART counter bump, nothing. So after a lot of experimenting (not the CPU, ECC RAM is fine, controller is fine...), it turned out... ...Samsung managed to manufacture disks that made SMART requests dangerous. [1] You see, if you issued a SMART command while there was data in the drive's write cache (which it has, remember, already told the OS that it wrote safely), it would just...drop it. So my background monitoring of the counters as I did all this IO was making writes get lost before they reached the platters, and so on scrub, then they would naturally be noticed as wrong... ...and if SMART lined up properly there too, they'd still have some on scrubbing again... The "best" part was that Samsung did cut a firmware update to fix this, eventually...but it _doesn't change the firmware version reported_, so you can't know if your drive is fixed just from looking. (We kept a spreadsheet of them and updated it as we flashed them.) [1] - https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks https://www.smartmontools.org/wiki/SamsungF4EGBadBlocks

rincebrain 5y ago
One more bonus, for anyone who yet comes upon this. I bought a monitor, once upon a time, a Dell UP2414Q, as I recall. It was a nice monitor, when it worked. Unfortunately, there's a terrible secret in high-framerate high-resolution stream transport - for at least some protocols (hello DisplayPort), they do this by sending two video streams for halves of the screen and the monitor muxes them. Unfortunately, on this monitor, it would go...wrong, sometimes. Sometimes, it would think that the picture on the "right" half should be stretched to fit the entire width, and then the left side displayed over that on the left (or vice-versa). Sometimes, it would just display nothing for one side. Most excitingly, sometimes it would just do what I would describe as "crash" - if the monitor was allowed to go to sleep, it would sometimes just...stop responding to any inputs on the physical buttons or commands from the computer, and require a hard power removal and reinsertion of the power cable before behaving again. So you got to leave this monitor non-asleep the entire time or endure suffering. You can see a fun comparison of what a screenshot thinks is onscreen versus a photo of it in [1][2]. And all of those things could all happen any time the resolution shifted - switching to/from fullscreen in something, for example. If you look, you can find references to a revision of the monitor which ships with a firmware that lacks these problems, but there exists no documented path to update from one to the other. I could not, when I looked, find someone who had the access and desire to try and extract the firmware from the board without any sort of provided tooling to do it, and I was not going to buy monitors with that model and hope I found one of the right rev. Dell support might have been willing to replace it under warranty, but I had bought it refurbished, so "ha ha". Ultimately, I gave the monitor to someone as a gift when theirs was woefully insufficient (with complete disclosure of the flaws in it) before I tried flashing any exciting payloads. Probably the least satisfying story ending of these, but true. [1] - https://i.imgur.com/NAG5hEU.jpg https://i.imgur.com/NAG5hEU.jpg [2] - https://i.imgur.com/9R7TWQE.jpg https://i.imgur.com/9R7TWQE.jpg

rincebrain 5y ago

One more bonus, for anyone who yet comes upon this. I bought a monitor, once upon a time, a Dell UP2414Q, as I recall. It was a nice monitor, when it worked. Unfortunately, there's a terrible secret in high-framerate high-resolution stream transport - for at least some protocols (hello DisplayPort), they do this by sending two video streams for halves of the screen and the monitor muxes them. Unfortunately, on this monitor, it would go...wrong, sometimes. Sometimes, it would think that the picture on the "right" half should be stretched to fit the entire width, and then the left side displayed over that on the left (or vice-versa). Sometimes, it would just display nothing for one side. Most excitingly, sometimes it would just do what I would describe as "crash" - if the monitor was allowed to go to sleep, it would sometimes just...stop responding to any inputs on the physical buttons or commands from the computer, and require a hard power removal and reinsertion of the power cable before behaving again. So you got to leave this monitor non-asleep the entire time or endure suffering. You can see a fun comparison of what a screenshot thinks is onscreen versus a photo of it in [1][2]. And all of those things could all happen any time the resolution shifted - switching to/from fullscreen in something, for example. If you look, you can find references to a revision of the monitor which ships with a firmware that lacks these problems, but there exists no documented path to update from one to the other. I could not, when I looked, find someone who had the access and desire to try and extract the firmware from the board without any sort of provided tooling to do it, and I was not going to buy monitors with that model and hope I found one of the right rev. Dell support might have been willing to replace it under warranty, but I had bought it refurbished, so "ha ha". Ultimately, I gave the monitor to someone as a gift when theirs was woefully insufficient (with complete disclosure of the flaws in it) before I tried flashing any exciting payloads. Probably the least satisfying story ending of these, but true. [1] - https://i.imgur.com/NAG5hEU.jpg https://i.imgur.com/NAG5hEU.jpg [2] - https://i.imgur.com/9R7TWQE.jpg https://i.imgur.com/9R7TWQE.jpg