3 ms·
Not quite a bug, but I think typing 'shutdown -h now' on the wrong tab (which I thought was a test box) of a bunch of tabs monitoring live systems wasn't my bes
by jacquesm 3y ago
Not quite a bug, but I think typing 'shutdown -h now' on the wrong tab (which I thought was a test box) of a bunch of tabs monitoring live systems wasn't my best moment. Worse so because it required an actual trip to the DC to get that box up again.
3 hours of downtime on a very busy service...
The very worst bug I've ever encountered (and that still bugs me to this day more than 20 years later) was an occasion in a hosting facility called 'Webcity' in Hoofddorp, NL. A machine called 'chopper' would periodically register as 'down' according to the watchdog and then it would be automatically power cycled. This happened frequently enough to be a real problem. Investigating that showed that the 'ping' packets that were supposed to test whether or not the box was up would randomly disappear (the responses).
After many, many hours of head scratching, changing out all kinds of gear and cabling (coaxial ethernet) we finally had one part left that we had not changed out. The BNC 'T' connector at the back of the machine. We all looked at each other and said 'that can't be it' and went for another round of verifying everything. Then, upon completion and just before again coming to the conclusion that that couldn't be it my colleague J suggested that maybe we should swap it out any way. Crazy idea but still. Problem solved.
Long story short: the BNC connector was a near perfect sink for a part of the bitstream that made up the ping packet. I still can't figure out how that could be the case but the signal dropped low enough that there was a good chance that the receiving machine didn't decode it properly. And once three packets in a row had been mangled it would reboot the box.
Total elapsed time was many hours and the fix was $1 or so. We ritually destroyed the old splitter.
- armchairhacker 3y agoThe first time I used `shutdown` on Linux console I wondered why it caused every other terminal to print something and then waited 60 seconds. Then I realized you could run it over ssh, on a remote server
- jacquesm 3y agoThat's why using that '-h now' felt particularly dumb in hindsight but I'm always impatient.