4 ms·

Outage Stories: The copy and paste outage

48 points by kevin_nisbet 4y ago

justinsaccount 4y ago
> And the way this system detected a process failure was to search the process list for a matching process. If we run the command super-ha-proxy..., and then it's in the process list, the process is working. If it's not in the process list, it must have crashed, exited, etc. and we need to launch a new instance. > The supervisor script just starts detecting the old process, and has no idea that the additional instance has been launched The number of times I've seen outages and weird flaky problems from some shitty script trying (and failing) to parse ps output... daemontools, runit, s6, systemd, etc all solve this problem properly.
mananaysiempre 4y ago
The way I read this, the redundancy mechanism depends on a process actually being dead when monitoring thinks it’s dead. But that’s really not the way you’re supposed to do this, is it? Maybe it’s just... resting. At the very least it seems like the allegedly dead process should be shot in the head before activating the backup one (although that’s likely quite difficult to do reliably even if the mental model is simple).
- kevin_nisbet 4y ago
  You're correct, it was a bad design. The vendor that built this system had a number of issues in this realm that demonstrated a lack of experience in building for the reliability targets of telecom. This was just one of the easier stories to tell. STONITH is a good design option, but I think this implementation would've still struggled. The way they were tracking processes was fragile, so the shoot the process lost track of which pid it should signal. And this was a sort of clustered system, so if you shoot the node you take out other services (think of murdering a kubernetes node to terminate one process, you're going to inflict some spillover.)
user00012-ab 4y ago
unrelated to whatever content was on that page, but for some reason I just don't trust websites that fade the text in; I also dislike pages like blogger.com that have a "loading page" intro to every site. Just feels dirty or something.
tpankaj 4y ago
This is why production services should have health checks that actually verify whether the service is doing its job in the system, not whether it's merely running the process. A process can be completely broken while still technically running, and vice versa (at least with janky ps-parsing scripts).
latchkey 4y ago
It is interesting how the article talks about having to provide excellent uptime and then goes on to describe how the system was designed to be brittle. This is less about a copy/paste error and more about system design.
bombcar 4y ago
Often "zombie" systems/processes/hardware are much harder to handle resiliently than actually exploded/dead. You need end-to-end testing of the system to detect that even though everything "seems" right, something is wrong (and potentially even to decide that "box X is testing correct but not working, kill it".
- GauntletWizard 4y ago
  My solution to this has been: Always have a dummy plug. I built this "on the fly" once by reconfiguring a service to use a blank copy of it's database, with permissions set to read-only. All the read queries succeeded, with zero results. All the write queries failed fast. It solved a hard problem fast.
hadlock 4y ago
We had a similar problem. Medium sized real-time trading firm HQd in the midwest. Had a bunch of dell stuff at our federally required DR datacenter. Dell tech (or whoever, I forget exactly, big company though) enters secure DR datacenter to put in a new blade in our blade server enclosure. I guess plugs a test router into our network via cat-5, then plugs in the power. Test router has the same static IP as some piece of Critical Infrastructure. Suddenly all of our real time trades are being routed to the Dell tech's test router, just into a black hole. All of our network config also gets routed to his test router in our DR datacenter. Everything stops. Even my personal workstation in our office, 10 miles away, my internet traffic gets routed to this guy's test router. Even our VOIP phones get routed into this black hole. 100s of thousands of day trades getting routed into the black hole that is his test router. Tech guy has no idea any of this is happening. Finally after a few minutes,our head of IT/networking/everything finds the tech's personal cell phone, and calls him from his own personal cell phone, because VOIP is dead in the water now. Tech unplugs network device from network, within 5 min everything self-heals. Untold amount of fines and fees for not being able to handle trades during the outage.
- kevin_nisbet 4y ago
  Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where: 1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat. I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.

48 points by kevin_nisbet 4y ago

> And the way this system detected a process failure was to search the process list for a matching process. If we run the command super-ha-proxy..., and then it's in the process list, the process is working. If it's not in the process list, it must have crashed, exited, etc. and we need to launch a new instance. > The supervisor script just starts detecting the old process, and has no idea that the additional instance has been launched The number of times I've seen outages and weird flaky problems from some shitty script trying (and failing) to parse ps output... daemontools, runit, s6, systemd, etc all solve this problem properly.

mananaysiempre 4y ago

The way I read this, the redundancy mechanism depends on a process actually being dead when monitoring thinks it’s dead. But that’s really not the way you’re supposed to do this, is it? Maybe it’s just... resting. At the very least it seems like the allegedly dead process should be shot in the head before activating the backup one (although that’s likely quite difficult to do reliably even if the mental model is simple).

kevin_nisbet 4y ago
You're correct, it was a bad design. The vendor that built this system had a number of issues in this realm that demonstrated a lack of experience in building for the reliability targets of telecom. This was just one of the easier stories to tell. STONITH is a good design option, but I think this implementation would've still struggled. The way they were tracking processes was fragile, so the shoot the process lost track of which pid it should signal. And this was a sort of clustered system, so if you shoot the node you take out other services (think of murdering a kubernetes node to terminate one process, you're going to inflict some spillover.)

user00012-ab 4y ago

unrelated to whatever content was on that page, but for some reason I just don't trust websites that fade the text in; I also dislike pages like blogger.com that have a "loading page" intro to every site. Just feels dirty or something.

tpankaj 4y ago

This is why production services should have health checks that actually verify whether the service is doing its job in the system, not whether it's merely running the process. A process can be completely broken while still technically running, and vice versa (at least with janky ps-parsing scripts).

latchkey 4y ago

It is interesting how the article talks about having to provide excellent uptime and then goes on to describe how the system was designed to be brittle. This is less about a copy/paste error and more about system design.

bombcar 4y ago

Often "zombie" systems/processes/hardware are much harder to handle resiliently than actually exploded/dead. You need end-to-end testing of the system to detect that even though everything "seems" right, something is wrong (and potentially even to decide that "box X is testing correct but not working, kill it".

GauntletWizard 4y ago
My solution to this has been: Always have a dummy plug. I built this "on the fly" once by reconfiguring a service to use a blank copy of it's database, with permissions set to read-only. All the read queries succeeded, with zero results. All the write queries failed fast. It solved a hard problem fast.

hadlock 4y ago

We had a similar problem. Medium sized real-time trading firm HQd in the midwest. Had a bunch of dell stuff at our federally required DR datacenter. Dell tech (or whoever, I forget exactly, big company though) enters secure DR datacenter to put in a new blade in our blade server enclosure. I guess plugs a test router into our network via cat-5, then plugs in the power. Test router has the same static IP as some piece of Critical Infrastructure. Suddenly all of our real time trades are being routed to the Dell tech's test router, just into a black hole. All of our network config also gets routed to his test router in our DR datacenter. Everything stops. Even my personal workstation in our office, 10 miles away, my internet traffic gets routed to this guy's test router. Even our VOIP phones get routed into this black hole. 100s of thousands of day trades getting routed into the black hole that is his test router. Tech guy has no idea any of this is happening. Finally after a few minutes,our head of IT/networking/everything finds the tech's personal cell phone, and calls him from his own personal cell phone, because VOIP is dead in the water now. Tech unplugs network device from network, within 5 min everything self-heals. Untold amount of fines and fees for not being able to handle trades during the outage.

kevin_nisbet 4y ago
Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where: 1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat. I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.