6 ms·

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all config

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to do anything to fix it failed. It is hard to imagine that this system has good test coverage.

mjibson 11y ago
I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?
- ikeboy 11y ago
  You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.
  - maxander 11y ago
    You may be able to build an elaborate system of dummy network operations to test with, but this system may wind up with bugs that mask what would be errors in the real system. And how to you test against that? A dummy network to test the dummy network operations on? What if the dummy network contains bugs that make it behave significantly different from the real network, in error cases? How do you test for that? Its turtles all the way down!
    - ikeboy 11y ago
      You can test it against the actual network; if something goes wrong, you'll have downtime, but you'll be prepared to get it all back up. Or, to test whether the "prevent errors from going to new places" works, temporarily configure the new places to ignore new configs; if the system works, no messages will be sent there; if the system doesn't work, they ignore the message and you learn about a bug.
  - dward 11y ago
    You mean an ICMP request? The IPs were anycast and did not become unreachable until all edge routers had stopped announcing BGP routes. At that point the failure was global. Check out the postmortem, it's a good read.
    - ikeboy 11y ago
      The servers became unreachable, but the IPs weren't unreachable until all the servers were reconfigured. My test should have caught this bug: > In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.
- manquer 11y ago
  While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.
  - mgw 11y ago
    They had a canary release mechanism in place. This is described in the post mortem. > These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. Taking no cofirmation of the canary testing process as a signal to go ahead though is not just a bug but a design flaw IMO.
  - magicalist 11y ago
    There was a canary release.
  - mattdeboard 11y ago
    If you read the actual report, it mentions that they did a canary step but its effectiveness was undermined. > In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.
- awinter-py 11y ago
  better test -- if you have seldom-exercised edge-case functionality that you can't figure out how to test, remove those features.
- Rapzid 11y ago
  Yes, in a manner of speaking; physical or virtual lab. At googles scale it wouldn't be unreasonable to have a completely parallel, but scaled back, network where they test their automation and code for happy and sad path. That doesn't mean that bugs can't creep in. Who knows, maybe these were all extremely unlikely bugs and Google hit an astronomically unlikely bad-luck streak. Happens.
- pmarreck 11y ago
  You fake out the connection with a faker object and give that to the code that wants to communicate to the network, and it returns streamed, deterministic data that would have been expected from the actual network, given deterministic inputs. The test uses the fake; the production code gets given the real object.
- deleted 11y ago
  [deleted]
drstewart 11y ago
Seriously. This is a good postmortem, but these are hardly edge case bugs. In this case, major critical functionality just plain didn't work. Kind of shocking.
- kevan 11y ago
  They explained the issues in laymans terms that most likely mask the true complexity of what happened. It's easy to read the final result: "tried to reject but then deleted everything" and think "Well duh that's bad, who would build a system that does that?", but I think you're fooling yourself if you think that edge cases couldn't cause that.
  - brlewis 11y ago
    Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. It seems obvious to me that the push system should not proceed without confirmation from the management software, and the management software should not confirm the change is OK if it detects failure. I see a straightforward defect here, not a confluence of edge cases.
specialist 11y ago
Spit balling here... For progressive rollouts, what if config changes where pulled instead of pushed? Each system would be responsible for itself updating, verifying (canary, smoketest, make sure other systems successfully updated, etc), bouncing, and then rolling back as needed.
- nostrademons 11y ago
  A bunch of that's in-place already, eg. all Google servers have health checks that run basic smoke tests on a configuration, and if a large number of replicas become unhealthy after a config change, the rollout process automatically aborts and rolls back to the last known good conversion. The problem here was that there was a bug in the health check that masked the problem by assigning the last-good configuration, and then there was a bug in that code that had saved "nothing" as the last-good configuration. So rather than failing and having the error caught at the top level, it failed and buggy failure-recovery code made the problem worse.
agnivade 11y ago
I feel 1 is a timing related issue. Which are extremely hard to get right, and might happen. 2 and 3 shouldn't have happened. But since they aren't releasing any further details. It would be unfair to rate the system.
pmarreck 11y ago
The take-home here is: Unit-test your failure states as well, people. Not just your happy paths! I mean, this problem was a result of MULTIPLE untested failure states. And yes, it IS possible to unit-test this sort of thing. You can fake out network connections and responses. I haven't yet found something that's impossible to unit-test, if you just think about how to do it properly, actually. EDIT: Why downvotes without a typewritten rebuttal? That's just not what I expect from HN (as opposed to, say, Reddit)

mjibson 11y ago

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

ikeboy 11y ago
You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.
- maxander 11y ago
  You may be able to build an elaborate system of dummy network operations to test with, but this system may wind up with bugs that mask what would be errors in the real system. And how to you test against that? A dummy network to test the dummy network operations on? What if the dummy network contains bugs that make it behave significantly different from the real network, in error cases? How do you test for that? Its turtles all the way down!
  - ikeboy 11y ago
    You can test it against the actual network; if something goes wrong, you'll have downtime, but you'll be prepared to get it all back up. Or, to test whether the "prevent errors from going to new places" works, temporarily configure the new places to ignore new configs; if the system works, no messages will be sent there; if the system doesn't work, they ignore the message and you learn about a bug.
- dward 11y ago
  You mean an ICMP request? The IPs were anycast and did not become unreachable until all edge routers had stopped announcing BGP routes. At that point the failure was global. Check out the postmortem, it's a good read.
  - ikeboy 11y ago
    The servers became unreachable, but the IPs weren't unreachable until all the servers were reconfigured. My test should have caught this bug: > In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.
manquer 11y ago
While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.
- mgw 11y ago
  They had a canary release mechanism in place. This is described in the post mortem. > These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. Taking no cofirmation of the canary testing process as a signal to go ahead though is not just a bug but a design flaw IMO.
- magicalist 11y ago
  There was a canary release.
- mattdeboard 11y ago
  If you read the actual report, it mentions that they did a canary step but its effectiveness was undermined. > In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.
awinter-py 11y ago
better test -- if you have seldom-exercised edge-case functionality that you can't figure out how to test, remove those features.
Rapzid 11y ago
Yes, in a manner of speaking; physical or virtual lab. At googles scale it wouldn't be unreasonable to have a completely parallel, but scaled back, network where they test their automation and code for happy and sad path. That doesn't mean that bugs can't creep in. Who knows, maybe these were all extremely unlikely bugs and Google hit an astronomically unlikely bad-luck streak. Happens.
pmarreck 11y ago
You fake out the connection with a faker object and give that to the code that wants to communicate to the network, and it returns streamed, deterministic data that would have been expected from the actual network, given deterministic inputs. The test uses the fake; the production code gets given the real object.
deleted 11y ago
[deleted]

drstewart 11y ago

Seriously. This is a good postmortem, but these are hardly edge case bugs. In this case, major critical functionality just plain didn't work. Kind of shocking.

kevan 11y ago
They explained the issues in laymans terms that most likely mask the true complexity of what happened. It's easy to read the final result: "tried to reject but then deleted everything" and think "Well duh that's bad, who would build a system that does that?", but I think you're fooling yourself if you think that edge cases couldn't cause that.
- brlewis 11y ago
  Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. It seems obvious to me that the push system should not proceed without confirmation from the management software, and the management software should not confirm the change is OK if it detects failure. I see a straightforward defect here, not a confluence of edge cases.

specialist 11y ago

Spit balling here... For progressive rollouts, what if config changes where pulled instead of pushed? Each system would be responsible for itself updating, verifying (canary, smoketest, make sure other systems successfully updated, etc), bouncing, and then rolling back as needed.

nostrademons 11y ago
A bunch of that's in-place already, eg. all Google servers have health checks that run basic smoke tests on a configuration, and if a large number of replicas become unhealthy after a config change, the rollout process automatically aborts and rolls back to the last known good conversion. The problem here was that there was a bug in the health check that masked the problem by assigning the last-good configuration, and then there was a bug in that code that had saved "nothing" as the last-good configuration. So rather than failing and having the error caught at the top level, it failed and buggy failure-recovery code made the problem worse.

agnivade 11y ago

I feel 1 is a timing related issue. Which are extremely hard to get right, and might happen. 2 and 3 shouldn't have happened. But since they aren't releasing any further details. It would be unfair to rate the system.

pmarreck 11y ago

The take-home here is: Unit-test your failure states as well, people. Not just your happy paths! I mean, this problem was a result of MULTIPLE untested failure states. And yes, it IS possible to unit-test this sort of thing. You can fake out network connections and responses. I haven't yet found something that's impossible to unit-test, if you just think about how to do it properly, actually. EDIT: Why downvotes without a typewritten rebuttal? That's just not what I expect from HN (as opposed to, say, Reddit)