4 ms·

My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't mo

by shrike 5y ago

My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't movable (I don't remember why) so I unplugged it from the PC, swept around it, and plugged it back in. About 2 minutes later half a dozen people run into the room. Apparently the SUN workstation I had unplugged the keyboard from was a critical component of the call manager and there was a bug that forced a reboot when the keyboard was plugged in while the system was running. I had hung up on ~36,000 people. Lesson learned and I only had to keep the data center clean for another year.

benjohnson 5y ago
That's....not your fault. There should have been a label on the computer near the keyboard jack written by the people who knew about the problem. If you didnt make a label afterwards yourself, the next keyboard-based outage would be your fault :)
- elwell 5y ago
  > keyboard-based outage I love it.
  - Ashanmaril 5y ago
    Reminds me of the "load-bearing poster" bit from The Simpsons
- Talanes 5y ago
  A cover on the service critical cable stretched across a walkway wouldn't be amiss either.
cjbprime 5y ago
It sounds like the real lesson that needed to be learned was taught to whoever decided that a system without any fault-tolerance that reboots whenever a keyboard is plugged in was fit for production.
- geoduck14 5y ago
  My work is doing planned DR testing right now. One key system had some problems failing over - so they delayed failing back, then they decided to change when they were going to fail back. Each time they changed their plans, there were war rooms discussing the "impact" of them changing plans. In each meeting, I'm scratching my head: this the closest thing to an actual disaster and we are all in a tizzy.
  - xvector 5y ago
    Ah, I love when people decide they need committees for plans for their committees.
    - deleted 5y ago
      [deleted]
  - bcrosby95 5y ago
    Reminds me, my friend recently had a "scoping" meeting. It was planned for 2 hours. Lasted 9. Obviously they needed a scoping meeting to scope their scoping meeting.
  - sharken 5y ago
    Doing regular fail overs is a healthy practice, but it's also a test of the company. A large outage is just about the only thing that can convince management that it is important. Reducing technical debt and staying current with technology ought to be a priority, but all too often it's not. Netflix learned it the hard way: https://opensource.com/article/18/4/how-netflix-does-failovers-7-minutes-flat https://opensource.com/article/18/4/how-netflix-does-failove...
    - jimmydorry 5y ago
      They had an outage which caused them to optimize the time to recover. However, this article didn't make any mention of regular testing of the new failover. If I'm reading it right, they designed their backup to not report on its health at all, and they instead have to just hope it works when they need it. Was this not the completely wrong lesson to learn? >Since our capacity injection is swift, we don't have to cautiously move the traffic by proxying to allow scaling policies to react. We can simply switch the DNS and open the floodgates, thus shaving even more precious minutes during an outage. >We added filters in the shadow cluster to prevent the dark instances from reporting metrics. Otherwise, they will pollute the metric space and confuse the normal operating behavior. >We also stopped the instances in the shadow clusters from registering themselves UP in discovery by modifying our discovery client. These instances will continue to remain in the dark (pun fully intended) until we trigger a failover.
anitil 5y ago
I mean, at least they could have put some superglue on the plug!
aspaceman 5y ago
I'm hilariously reminded of the episode of Rick and Morty where Morty flips the wrong light-switch, killing a room full of cryogencially frozen people.
adaml_623 5y ago
I did almost the same action and unplugged and replugged a Sun keyboard while rearranging cables. But in my case the machine just suspended to the boot mode. 10 minutes later my team lead came and found me, figured out the situation and typed 'go'. The server restarted and just continued happily doing its job.
urbandw311er 5y ago
I wonder if it was one of the earlier PS/2 keyboards. As I recall some of those weren’t plug and play, so it would have been less of a bug and more just that the hardware wasn’t designed to be hot swapped.
- toomanybeersies 5y ago
  > As I recall some of those weren’t plug and play Due to its how its implemented on the motherboard, standard PS/2 isn't hot swappable. Any support for hot swapping is the exception, rather than the norm. I always thought it was a bit ironic that VGA and RS-232 cables (with D-sub connectors) could be mechanically attached with screws, even though they were hot swappable. Yet PS/2, which used screwless mini-DIN connectors, wasn't hot swappable.
GuB-42 5y ago
So, you have an keyboard cable stretched across the walkway, maybe a tripping hazard, unlabeled, that crashes everything when it is unplugged... It looks a lot like this https://xkcd.com/908/ https://xkcd.com/908/ I have also seen instances of similar things, but usually there is a large sign saying "do not unplug/turn off" and they try to make it as unobtrusive as possible. The problem here is that it is you were the newbie. It should have been done by a higher up who could then properly chastise whoever was responsible of that mess.
- toomanybeersies 5y ago
  Fundamentally, a large "do not unplug" sign is just like any other failsafe system. Nuclear missile silos, probably the most "do not touch this switch" systems in existence, are only slightly more refined, and have red covers over all the important switches instead. The alternative is to epoxy the keyboard in. But then when you legitimately need to unplug it, you need to find a hammer. As an aside, that XKCD strip feels heavily inspired by an episode of the IT Crowd [1] [1] https://www.youtube.com/watch?v=iDbyYGrswtg https://www.youtube.com/watch?v=iDbyYGrswtg

by shrike 5y ago

benjohnson 5y ago

That's....not your fault. There should have been a label on the computer near the keyboard jack written by the people who knew about the problem. If you didnt make a label afterwards yourself, the next keyboard-based outage would be your fault :)

elwell 5y ago
> keyboard-based outage I love it.
- Ashanmaril 5y ago
  Reminds me of the "load-bearing poster" bit from The Simpsons
Talanes 5y ago
A cover on the service critical cable stretched across a walkway wouldn't be amiss either.

cjbprime 5y ago

It sounds like the real lesson that needed to be learned was taught to whoever decided that a system without any fault-tolerance that reboots whenever a keyboard is plugged in was fit for production.

geoduck14 5y ago
My work is doing planned DR testing right now. One key system had some problems failing over - so they delayed failing back, then they decided to change when they were going to fail back. Each time they changed their plans, there were war rooms discussing the "impact" of them changing plans. In each meeting, I'm scratching my head: this the closest thing to an actual disaster and we are all in a tizzy.
- xvector 5y ago
  Ah, I love when people decide they need committees for plans for their committees.
  - deleted 5y ago
    [deleted]
- bcrosby95 5y ago
  Reminds me, my friend recently had a "scoping" meeting. It was planned for 2 hours. Lasted 9. Obviously they needed a scoping meeting to scope their scoping meeting.
- sharken 5y ago
  Doing regular fail overs is a healthy practice, but it's also a test of the company. A large outage is just about the only thing that can convince management that it is important. Reducing technical debt and staying current with technology ought to be a priority, but all too often it's not. Netflix learned it the hard way: https://opensource.com/article/18/4/how-netflix-does-failovers-7-minutes-flat https://opensource.com/article/18/4/how-netflix-does-failove...
  - jimmydorry 5y ago
    They had an outage which caused them to optimize the time to recover. However, this article didn't make any mention of regular testing of the new failover. If I'm reading it right, they designed their backup to not report on its health at all, and they instead have to just hope it works when they need it. Was this not the completely wrong lesson to learn? >Since our capacity injection is swift, we don't have to cautiously move the traffic by proxying to allow scaling policies to react. We can simply switch the DNS and open the floodgates, thus shaving even more precious minutes during an outage. >We added filters in the shadow cluster to prevent the dark instances from reporting metrics. Otherwise, they will pollute the metric space and confuse the normal operating behavior. >We also stopped the instances in the shadow clusters from registering themselves UP in discovery by modifying our discovery client. These instances will continue to remain in the dark (pun fully intended) until we trigger a failover.

anitil 5y ago

I mean, at least they could have put some superglue on the plug!

aspaceman 5y ago

I'm hilariously reminded of the episode of Rick and Morty where Morty flips the wrong light-switch, killing a room full of cryogencially frozen people.

adaml_623 5y ago

I did almost the same action and unplugged and replugged a Sun keyboard while rearranging cables. But in my case the machine just suspended to the boot mode. 10 minutes later my team lead came and found me, figured out the situation and typed 'go'. The server restarted and just continued happily doing its job.

urbandw311er 5y ago

I wonder if it was one of the earlier PS/2 keyboards. As I recall some of those weren’t plug and play, so it would have been less of a bug and more just that the hardware wasn’t designed to be hot swapped.

toomanybeersies 5y ago
> As I recall some of those weren’t plug and play Due to its how its implemented on the motherboard, standard PS/2 isn't hot swappable. Any support for hot swapping is the exception, rather than the norm. I always thought it was a bit ironic that VGA and RS-232 cables (with D-sub connectors) could be mechanically attached with screws, even though they were hot swappable. Yet PS/2, which used screwless mini-DIN connectors, wasn't hot swappable.

GuB-42 5y ago

So, you have an keyboard cable stretched across the walkway, maybe a tripping hazard, unlabeled, that crashes everything when it is unplugged... It looks a lot like this https://xkcd.com/908/ https://xkcd.com/908/ I have also seen instances of similar things, but usually there is a large sign saying "do not unplug/turn off" and they try to make it as unobtrusive as possible. The problem here is that it is you were the newbie. It should have been done by a higher up who could then properly chastise whoever was responsible of that mess.

toomanybeersies 5y ago
Fundamentally, a large "do not unplug" sign is just like any other failsafe system. Nuclear missile silos, probably the most "do not touch this switch" systems in existence, are only slightly more refined, and have red covers over all the important switches instead. The alternative is to epoxy the keyboard in. But then when you legitimately need to unplug it, you need to find a hammer. As an aside, that XKCD strip feels heavily inspired by an episode of the IT Crowd [1] [1] https://www.youtube.com/watch?v=iDbyYGrswtg https://www.youtube.com/watch?v=iDbyYGrswtg