9 ms·

My “favorite” magtape story

91 points by zdw 4y ago

mikewarot 4y ago
Similar stories talk about the bits of wire floor scrubbing brushes getting sucked up into multi-million dollar computers in the 1960s.
killjoywashere 4y ago
Also filed under threats you never thought of: I can't find it at the moment, but there was a case report in the New England Journal of Medicine of an iron worker who presented, as I recall, vomiting blood. Turns out some years ago, he had been working with a wire wheel and felt a sharp sensation in the back of his throat, but it subsided. Over the next decade or so, he had intermittent chest pain. A wire had flown out of the spinning wire brush, he had inhaled it midflight, it had landed in his trachea. Over the next decade, it slowly migrated through the trachea and penetrated one of the great vessels of the heart. Retrieving the wire and repairing both the trachea and the great vessel was a delicate matter.
- causi 4y ago
  Full face shields are vastly underutilized pieces of safety equipment. Highly recommended, especially if you fog up safety glasses.
rrauenza 4y ago
When I worked for HP, I was visiting an HP partner’s datacenter. While there, an electrician was doing work cutting metal (conduit if I recall) with a chop saw in the server room. I did report it … since we had systems we were loaning them in that data center.
yummybear 4y ago
In a similar vein we recently had an issue with one employees monitors turning off when a coworker sat down. We traced it to the cylinder of the chair apprently making some electric disturbance. Our minds were blown for a while, but I remembered reading something similar.
- spongeb00b 4y ago
  Ok so this literally happens to me! There’s a desk behind my back. When the guy there stands up, roughly 10% of the time my Dell monitor will go blank and reset.
  - tecleandor 4y ago
    Seems like this is a relatively common bug. Some months ago, in the HN thread for a blog post by the Lunar app developer and weird bugs people sent him, a commenters posts how this is a known bug and, for example, it's in DisplayLink's support database, and they even suggest some fixes: "Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue." HN Thread: https://news.ycombinator.com/item?id=32631017 https://news.ycombinator.com/item?id=32631017 DisplayLink support: https://support.displaylink.com/knowledgebase/articles/738618-display-intermittently-blanking-flickering-or-los https://support.displaylink.com/knowledgebase/articles/73861... Lunar app bugs: https://notes.alinpanaitiu.com/Weird%20monitor%20bugs https://notes.alinpanaitiu.com/Weird%20monitor%20bugs Edit: Looking around I've found this funny/interesting short paper from 1999 called "Unusual Forms of ESD and Their Effects". They refer to a Japanese paper already in 1993 pointing to chairs ESD disruption, and also to using a ziplock bag with a bunch of coins to test ESD protection... Short and interesting! : https://www.emcesd.com/pdf/uesd99-w.pdf https://www.emcesd.com/pdf/uesd99-w.pdf Edit 2: Interesting 2006 research about one of the researchers for the 93 paper: "ESD noise radiated from walkers was observed in the 5-GHz-band to study the influence on the quality degradation of radio communication. We determined that the radiated ESD noise in the 5-Gk-band is originated by "Collision ESD" and "Induced ESD" when people walk while wearing metal objects." https://ieeexplore.ieee.org/document/5256785 https://ieeexplore.ieee.org/document/5256785
    - BizarroLand 4y ago
wrldos 4y ago
My one. I walked into the office once when we had the cable monkeys in and the DLT drives and server pile was covered in plaster dust where they had been drilling above it. Both drives died within a month. Had to replace them and all the tapes. Cheap ass director bought a rack after that. Edit: found potato cam pictures (this was nearly 20 years ago) https://imgur.com/a/gjyCOQj https://imgur.com/a/gjyCOQj
ilyt 4y ago
https://minnie.tuhs.org/pipermail/tuhs/2023-January/027547.html https://minnie.tuhs.org/pipermail/tuhs/2023-January/027547.h... is also good.
Taniwha 4y ago
Back in the early days of large screen CRTs (19 inch color), late 80s. I was working on bringing X up on a new display board .... suddenly the whole screen sort of twisted and shifted, and the went OK again .... outside my window a really long truck with recently replaced BART (electric train) rails was going by .... essentially a large bar magnet ....
- geon 4y ago
  In 2005 I live next to a railway. I could tell when the trains left the station a few kilometers away, because the peak current would make my crt image vibrate.
daver898 4y ago
In olden days i worked at a datacenter that had problems when work on the floor above included grinding (and creating some iron fillings all over the place). The tops of the dg computers below were not covered and the iron fillings caused all sorts of grief inside the computer itself.
tragomaskhalos 4y ago
Reminded me of this classic about irradiated cows messing with a computer at a railway junction: https://beza1e1.tuxen.de/lore/crash_cows.html https://beza1e1.tuxen.de/lore/crash_cows.html
- emmelaich 4y ago
  Also "Always mount a scratch monkey" https://edp.org/monkey.htm https://edp.org/monkey.htm
  - benj111 4y ago
    I thought the monkeys in this story were going to be users. Well. They are, but you know what I mean.
- dspillett 4y ago
  “Upon discovering this, Sergei immediately filed immigration papers with any country that would listen.” — an appropriate level of faith that the system can be trusted correct itself, I think.
  - thriftwy 4y ago
    The tricky part at the time was not being admitted in any country but getting out of USSR.
- thriftwy 4y ago
  I wonder if anybody ever fact checked this story. I expected to see something related to Mayak plant disaster upon reading Sverdlovsk. https://en.m.wikipedia.org/wiki/Kyshtym_disaster https://en.m.wikipedia.org/wiki/Kyshtym_disaster
linker3000 4y ago
Random crashes on a Netware server (1-3-ish times a week) = dishwasher in the kitchen, which was immediately behind the server. Fix: Move the server (which was just on a table) to another corner of the computer room. Regular increase in coax Ethernet network error rate = moon tides affecting nearby river and increased ground water during heavy rain affecting ground voltage level on side of building next to the river. Poor or unable-to-connect modem link = nail through telephone cable on wall. Misbehaving fibre optic token ring topology; 'data getting corrupted in transit' = EMP being injected into one electrical/optical transceiver on one network card in a PC closest to 'The largest laser in Europe' (and its power supply). This was at a UK Government defence research lab in the 1990s. 200MB hard disk in PC - would spin down when the case screws were inserted. Never did find the root cause (pressure on 'something'). This was my work PC so I just left off the screws. Computer room overheats = Delivery of listing paper put on floor in front of self-contained circulating air cooling system in the corner of the computer room. Fix: Move the boxes.
- linsomniac 4y ago
  Back in the early '90s I worked at a PC builder. They got in a batch of drives that had their PCBs closer than usual to the screw holes and installed quite a few before they noticed. They tried to RMA the drives, but were refused. In the end the got a bunch of shorter screws for those drives. They were a fairly large local business, but they priced out big quotes for the University based on the price of parts dropping over 3 years, and then one year in the late 90s prices went up rather than down, and it suddenly put them out of business.
  - linker3000 4y ago
    I can relate to the screws issue. We also did have a 'The cleaner unplugged the packet network termination unit to use the socket' episode.
- linker3000 4y ago
  Forgot one of the best ones: A large leisure centre/sports complex had 'ghost typing' occurring on several new Olivetti PCs. Root cause: Due to the size of the site, the staff had 5W walkie talkies to keep in touch. The new batch of Olivetti computers had cheapened keyboards, and instead of the electronics being encased in a metal clamshell, there was just a foil-backed piece of card folded over the PCB; this did not work as an effective rf screen and the walkie talkie signal strength was hitting the circuitry and causing random keypresses - including CTRL-P to keep dumping screen images to the dot matrix printers (this was DOS world late 1980s/early 1990s). Immediate fix: Replace new keyboards with earlier revisions. Longer-term fix: I seem to recall it was 'a few more capacitors' for rf decoupling.
NoboruWataya 4y ago
Is there a list somewhere of these crazy bug stories? In addition to some great ones in these comments, the one that always sticks out at me is the "500 mile email" story.[0] I'm sure someone must have a big collection of these somewhere. 0: https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
- psychphysic 4y ago
  You're in the list right now!
nicbou 4y ago
I could listen to such stories all day. Any idea of where I can find more?
- andorov 4y ago
  the Magic / More Magic switch is a favorite http://www.catb.org/jargon/html/magic-story.html http://www.catb.org/jargon/html/magic-story.html
  - nicbou 4y ago
    I have an unused switch on my wall. This gives me an idea
  - dreamcompiler 4y ago
    "In my house there's this light switch that doesn't do anything. Every so often I would flick it on and off just to check. Yesterday, I got a call from a woman in Madagascar. She said, 'Cut it out.'" --Steven Wright
porbelm 4y ago
On a rollout of new workstations for the local bank, which were moving into a nice, newly built, office building. On the trader floor, one of the workstations would not finish setup and would constantly reboot. And one of the (three! awesome back in the late '90s) monitors wouldn't work properly. Turned itself off and on all the time. I said to a colleague that the power circuit was probably borked, off-spec and faulty somehow. Since the building had just been put up, there was still a flurry of electricians and plumbers and painters and everything else inside trying to get everything ready over that weekend. After my colleagues and others had tried figuring out the problem for a good time by trying everything else, I asked one of the passing electricians to get the circuit for that specific desk checked out. First time I had right in the workplace by just deducing a cause from experience, as a fresh IT worker in a big company. There was indeed something wrong with the power circuit. Felt good. Never learned exactly what was wrong though. Bad cable perhaps.
cheese_van 4y ago
Around 1993. An overseas government office, high ranking guy,light-years beyond my rank, brilliant in his political field, complains he's lost data, AGAIN, furiously summons me to make amends about the deplorable system of which I am in charge and FIX THIS SHIT. Office has a low standing safe next to his desk. Each safe has a little magnetic sign, one side says "open" the other "closed". While the safe was open, he put the sign on top of the safe, out of the way. It seems (who knew?) that placing your floppy disks on top of magnetic signs might corrupt your data. I accepted responsibility, I told him, because I didn't anticipate he might do this, and I should have briefed him. On one hand it was politically expedient to end the quarrel quickly. On the other, perhaps true: never assume your users or clients have a single shred of intelligence beyond their field.
1970-01-01 4y ago
AM radio was (still is) one of the best tools for finding issues like this sparking motor. The real trick is to find an AM radio station that you enjoy listening to every day, and when you hear interference, drop what you are doing to find the source. Engines and motors close to your server room will create disk errors and absolutely wreck any signal in unshielded wiring.
Ensorceled 4y ago
We had a server that was failing intermittently at about the same time of day, around 3am. Looked like a power failure from the logs but there was no problems with other servers and this server was on a dedicated circuit that was power backed up and the server had a UPS attached. So one of my team volunteered to stay and watch the server. Around 3am, one of the custodians buzzes himself into the secure server room that has limited access, at the time only 3 of us on the IT team, our head of Regulatory and the CFO, were supposed to have access. The custodian walks over to the server, says hi to Chris and proceeds to unplug the server’s UPS from the bright orange socket panel and plug in his vacuum cleaner. He then starts to vacuum the raised floor in our air filtered server room. The failure was intermittent because sometimes he finished vacuuming before the UPS ran out. Apparently our facilities manager considered herself and her staff exempt from our corporate security policies. She also told the contract custodian to “just unplug something if they didn’t have the courtesy to leave you a free plug”.

91 points by zdw 4y ago

Similar stories talk about the bits of wire floor scrubbing brushes getting sucked up into multi-million dollar computers in the 1960s.

killjoywashere 4y ago

Also filed under threats you never thought of: I can't find it at the moment, but there was a case report in the New England Journal of Medicine of an iron worker who presented, as I recall, vomiting blood. Turns out some years ago, he had been working with a wire wheel and felt a sharp sensation in the back of his throat, but it subsided. Over the next decade or so, he had intermittent chest pain. A wire had flown out of the spinning wire brush, he had inhaled it midflight, it had landed in his trachea. Over the next decade, it slowly migrated through the trachea and penetrated one of the great vessels of the heart. Retrieving the wire and repairing both the trachea and the great vessel was a delicate matter.

causi 4y ago
Full face shields are vastly underutilized pieces of safety equipment. Highly recommended, especially if you fog up safety glasses.

rrauenza 4y ago

When I worked for HP, I was visiting an HP partner’s datacenter. While there, an electrician was doing work cutting metal (conduit if I recall) with a chop saw in the server room. I did report it … since we had systems we were loaning them in that data center.

yummybear 4y ago

In a similar vein we recently had an issue with one employees monitors turning off when a coworker sat down. We traced it to the cylinder of the chair apprently making some electric disturbance. Our minds were blown for a while, but I remembered reading something similar.

spongeb00b 4y ago
Ok so this literally happens to me! There’s a desk behind my back. When the guy there stands up, roughly 10% of the time my Dell monitor will go blank and reset.
- tecleandor 4y ago
  Seems like this is a relatively common bug. Some months ago, in the HN thread for a blog post by the Lunar app developer and weird bugs people sent him, a commenters posts how this is a known bug and, for example, it's in DisplayLink's support database, and they even suggest some fixes: "Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue." HN Thread: https://news.ycombinator.com/item?id=32631017 https://news.ycombinator.com/item?id=32631017 DisplayLink support: https://support.displaylink.com/knowledgebase/articles/738618-display-intermittently-blanking-flickering-or-los https://support.displaylink.com/knowledgebase/articles/73861... Lunar app bugs: https://notes.alinpanaitiu.com/Weird%20monitor%20bugs https://notes.alinpanaitiu.com/Weird%20monitor%20bugs Edit: Looking around I've found this funny/interesting short paper from 1999 called "Unusual Forms of ESD and Their Effects". They refer to a Japanese paper already in 1993 pointing to chairs ESD disruption, and also to using a ziplock bag with a bunch of coins to test ESD protection... Short and interesting! : https://www.emcesd.com/pdf/uesd99-w.pdf https://www.emcesd.com/pdf/uesd99-w.pdf Edit 2: Interesting 2006 research about one of the researchers for the 93 paper: "ESD noise radiated from walkers was observed in the 5-GHz-band to study the influence on the quality degradation of radio communication. We determined that the radiated ESD noise in the 5-Gk-band is originated by "Collision ESD" and "Induced ESD" when people walk while wearing metal objects." https://ieeexplore.ieee.org/document/5256785 https://ieeexplore.ieee.org/document/5256785
  - BizarroLand 4y ago

wrldos 4y ago

My one. I walked into the office once when we had the cable monkeys in and the DLT drives and server pile was covered in plaster dust where they had been drilling above it. Both drives died within a month. Had to replace them and all the tapes. Cheap ass director bought a rack after that. Edit: found potato cam pictures (this was nearly 20 years ago) https://imgur.com/a/gjyCOQj https://imgur.com/a/gjyCOQj

ilyt 4y ago

https://minnie.tuhs.org/pipermail/tuhs/2023-January/027547.html https://minnie.tuhs.org/pipermail/tuhs/2023-January/027547.h... is also good.

Taniwha 4y ago

Back in the early days of large screen CRTs (19 inch color), late 80s. I was working on bringing X up on a new display board .... suddenly the whole screen sort of twisted and shifted, and the went OK again .... outside my window a really long truck with recently replaced BART (electric train) rails was going by .... essentially a large bar magnet ....

geon 4y ago
In 2005 I live next to a railway. I could tell when the trains left the station a few kilometers away, because the peak current would make my crt image vibrate.

daver898 4y ago

In olden days i worked at a datacenter that had problems when work on the floor above included grinding (and creating some iron fillings all over the place). The tops of the dg computers below were not covered and the iron fillings caused all sorts of grief inside the computer itself.

tragomaskhalos 4y ago

Reminded me of this classic about irradiated cows messing with a computer at a railway junction: https://beza1e1.tuxen.de/lore/crash_cows.html https://beza1e1.tuxen.de/lore/crash_cows.html

emmelaich 4y ago
Also "Always mount a scratch monkey" https://edp.org/monkey.htm https://edp.org/monkey.htm
- benj111 4y ago
  I thought the monkeys in this story were going to be users. Well. They are, but you know what I mean.
dspillett 4y ago
“Upon discovering this, Sergei immediately filed immigration papers with any country that would listen.” — an appropriate level of faith that the system can be trusted correct itself, I think.
- thriftwy 4y ago
  The tricky part at the time was not being admitted in any country but getting out of USSR.
thriftwy 4y ago
I wonder if anybody ever fact checked this story. I expected to see something related to Mayak plant disaster upon reading Sverdlovsk. https://en.m.wikipedia.org/wiki/Kyshtym_disaster https://en.m.wikipedia.org/wiki/Kyshtym_disaster

linker3000 4y ago

Random crashes on a Netware server (1-3-ish times a week) = dishwasher in the kitchen, which was immediately behind the server. Fix: Move the server (which was just on a table) to another corner of the computer room. Regular increase in coax Ethernet network error rate = moon tides affecting nearby river and increased ground water during heavy rain affecting ground voltage level on side of building next to the river. Poor or unable-to-connect modem link = nail through telephone cable on wall. Misbehaving fibre optic token ring topology; 'data getting corrupted in transit' = EMP being injected into one electrical/optical transceiver on one network card in a PC closest to 'The largest laser in Europe' (and its power supply). This was at a UK Government defence research lab in the 1990s. 200MB hard disk in PC - would spin down when the case screws were inserted. Never did find the root cause (pressure on 'something'). This was my work PC so I just left off the screws. Computer room overheats = Delivery of listing paper put on floor in front of self-contained circulating air cooling system in the corner of the computer room. Fix: Move the boxes.

linsomniac 4y ago
Back in the early '90s I worked at a PC builder. They got in a batch of drives that had their PCBs closer than usual to the screw holes and installed quite a few before they noticed. They tried to RMA the drives, but were refused. In the end the got a bunch of shorter screws for those drives. They were a fairly large local business, but they priced out big quotes for the University based on the price of parts dropping over 3 years, and then one year in the late 90s prices went up rather than down, and it suddenly put them out of business.
- linker3000 4y ago
  I can relate to the screws issue. We also did have a 'The cleaner unplugged the packet network termination unit to use the socket' episode.
linker3000 4y ago
Forgot one of the best ones: A large leisure centre/sports complex had 'ghost typing' occurring on several new Olivetti PCs. Root cause: Due to the size of the site, the staff had 5W walkie talkies to keep in touch. The new batch of Olivetti computers had cheapened keyboards, and instead of the electronics being encased in a metal clamshell, there was just a foil-backed piece of card folded over the PCB; this did not work as an effective rf screen and the walkie talkie signal strength was hitting the circuitry and causing random keypresses - including CTRL-P to keep dumping screen images to the dot matrix printers (this was DOS world late 1980s/early 1990s). Immediate fix: Replace new keyboards with earlier revisions. Longer-term fix: I seem to recall it was 'a few more capacitors' for rf decoupling.

NoboruWataya 4y ago

Is there a list somewhere of these crazy bug stories? In addition to some great ones in these comments, the one that always sticks out at me is the "500 mile email" story.[0] I'm sure someone must have a big collection of these somewhere. 0: https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles

psychphysic 4y ago
You're in the list right now!

nicbou 4y ago

I could listen to such stories all day. Any idea of where I can find more?

andorov 4y ago
the Magic / More Magic switch is a favorite http://www.catb.org/jargon/html/magic-story.html http://www.catb.org/jargon/html/magic-story.html
- nicbou 4y ago
  I have an unused switch on my wall. This gives me an idea
- dreamcompiler 4y ago
  "In my house there's this light switch that doesn't do anything. Every so often I would flick it on and off just to check. Yesterday, I got a call from a woman in Madagascar. She said, 'Cut it out.'" --Steven Wright

porbelm 4y ago

On a rollout of new workstations for the local bank, which were moving into a nice, newly built, office building. On the trader floor, one of the workstations would not finish setup and would constantly reboot. And one of the (three! awesome back in the late '90s) monitors wouldn't work properly. Turned itself off and on all the time. I said to a colleague that the power circuit was probably borked, off-spec and faulty somehow. Since the building had just been put up, there was still a flurry of electricians and plumbers and painters and everything else inside trying to get everything ready over that weekend. After my colleagues and others had tried figuring out the problem for a good time by trying everything else, I asked one of the passing electricians to get the circuit for that specific desk checked out. First time I had right in the workplace by just deducing a cause from experience, as a fresh IT worker in a big company. There was indeed something wrong with the power circuit. Felt good. Never learned exactly what was wrong though. Bad cable perhaps.

cheese_van 4y ago

Around 1993. An overseas government office, high ranking guy,light-years beyond my rank, brilliant in his political field, complains he's lost data, AGAIN, furiously summons me to make amends about the deplorable system of which I am in charge and FIX THIS SHIT. Office has a low standing safe next to his desk. Each safe has a little magnetic sign, one side says "open" the other "closed". While the safe was open, he put the sign on top of the safe, out of the way. It seems (who knew?) that placing your floppy disks on top of magnetic signs might corrupt your data. I accepted responsibility, I told him, because I didn't anticipate he might do this, and I should have briefed him. On one hand it was politically expedient to end the quarrel quickly. On the other, perhaps true: never assume your users or clients have a single shred of intelligence beyond their field.

1970-01-01 4y ago

AM radio was (still is) one of the best tools for finding issues like this sparking motor. The real trick is to find an AM radio station that you enjoy listening to every day, and when you hear interference, drop what you are doing to find the source. Engines and motors close to your server room will create disk errors and absolutely wreck any signal in unshielded wiring.

Ensorceled 4y ago

We had a server that was failing intermittently at about the same time of day, around 3am. Looked like a power failure from the logs but there was no problems with other servers and this server was on a dedicated circuit that was power backed up and the server had a UPS attached. So one of my team volunteered to stay and watch the server. Around 3am, one of the custodians buzzes himself into the secure server room that has limited access, at the time only 3 of us on the IT team, our head of Regulatory and the CFO, were supposed to have access. The custodian walks over to the server, says hi to Chris and proceeds to unplug the server’s UPS from the bright orange socket panel and plug in his vacuum cleaner. He then starts to vacuum the raised floor in our air filtered server room. The failure was intermittent because sometimes he finished vacuuming before the UPS ran out. Apparently our facilities manager considered herself and her staff exempt from our corporate security policies. She also told the contract custodian to “just unplug something if they didn’t have the courtesy to leave you a free plug”.