3 ms·

So, for once, it actually was a hardware problem…

AlotOfReading 4y ago
Hardware bugs make the best war stories. I once spent a few days debugging a robot arm that would only manifest the erroneous behavior if it was in the right room. If you physically moved the arm and all its associated cables to another room, the problem would disappear only to manifest again when the hardware was all moved back to the original location. Turns out the problem was an undocumented revision to a passive port cover. The new design introduced a short between pins without changing external markings. The only distinguishing mark was a slight color difference between different batches of plastic.
- rawling 4y ago
  And you... changed port covers when you moved the robot?
  - AlotOfReading 4y ago
    Yes, or omitted them. They weren't supposed to be / documented to be necessary or functional, just keep the port clean of grime (robots operate in dirty environments) and mitigate signal loss.
plasticchris 4y ago
I once had a bug that was triggered by sunlight. We fixed it by painting the device black.
- Simon_O_Rourke 4y ago
  Similarly, I worked on a team that experienced backend failures in a custom distributed hash table that some crazy person decided to deploy on the cheapest server tier available from Hetzner hosting. Every hot summer day around 3pm, we started to see cascading failures in the DHT nodes. Turned out Hetzner were to cheap to pay for cooling for their cheapest server racks and they all failed like clockwork when it got hot enough outside.
- waste_monk 4y ago
  Is that something like erasing an EEPROM with sunlight? I recall Raspberry PI's had an issue where certain types of flash photography would cause them to shut down :) https://www.raspberrypi.com/news/xenon-death-flash-a-free-physics-lesson/ https://www.raspberrypi.com/news/xenon-death-flash-a-free-ph...
dilyevsky 4y ago
Early in google life they skimped on ecc memory for their 30 thousand something serves (it was way back then). All sorts of nasty hardware bugs… Even in my time with ecc there was all sorts of cpu bugs that would occasionally mess up every job that happened to land on a particular node
- H8crilA 4y ago
  Bitflips are still very much a thing, but you need to be processing datasets at least several petabytes in size to get a decent probability of a bitflip/instruction error. It's always about the probability of a flip/error, not about whether they exist at all. They always exist.

Hardware bugs make the best war stories. I once spent a few days debugging a robot arm that would only manifest the erroneous behavior if it was in the right room. If you physically moved the arm and all its associated cables to another room, the problem would disappear only to manifest again when the hardware was all moved back to the original location. Turns out the problem was an undocumented revision to a passive port cover. The new design introduced a short between pins without changing external markings. The only distinguishing mark was a slight color difference between different batches of plastic.

rawling 4y ago
And you... changed port covers when you moved the robot?
- AlotOfReading 4y ago
  Yes, or omitted them. They weren't supposed to be / documented to be necessary or functional, just keep the port clean of grime (robots operate in dirty environments) and mitigate signal loss.

plasticchris 4y ago

I once had a bug that was triggered by sunlight. We fixed it by painting the device black.

Simon_O_Rourke 4y ago
Similarly, I worked on a team that experienced backend failures in a custom distributed hash table that some crazy person decided to deploy on the cheapest server tier available from Hetzner hosting. Every hot summer day around 3pm, we started to see cascading failures in the DHT nodes. Turned out Hetzner were to cheap to pay for cooling for their cheapest server racks and they all failed like clockwork when it got hot enough outside.
waste_monk 4y ago
Is that something like erasing an EEPROM with sunlight? I recall Raspberry PI's had an issue where certain types of flash photography would cause them to shut down :) https://www.raspberrypi.com/news/xenon-death-flash-a-free-physics-lesson/ https://www.raspberrypi.com/news/xenon-death-flash-a-free-ph...

dilyevsky 4y ago

Early in google life they skimped on ecc memory for their 30 thousand something serves (it was way back then). All sorts of nasty hardware bugs… Even in my time with ecc there was all sorts of cpu bugs that would occasionally mess up every job that happened to land on a particular node

H8crilA 4y ago
Bitflips are still very much a thing, but you need to be processing datasets at least several petabytes in size to get a decent probability of a bitflip/instruction error. It's always about the probability of a flip/error, not about whether they exist at all. They always exist.