8 ms·

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causi

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

jgrahamc 4y ago
Well, the "we can't reach these data centers at all and need to go through the break glass procedure" was pretty "ouch" also.
- exikyut 4y ago
  I can('t) imagine, yikes. Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/ https://rachelbythebay.com/w/2019/01/20/quiet/ Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha. I would be very interested to hear more about how the break-glass process worked.
michaelmior 4y ago
This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.
- scottlamb 4y ago
  I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.
  - michaelmior 4y ago
    You're probably right, but if they're going to mention it as part of the problem, I would want to see it as part of the solution. However, I agree that they certainly shouldn't name names.
dangrossman 4y ago
I think I experienced first-hand the moment those network engineers were reverting their own reverts, breaking the web again. For example, DoorDash.com had come back online, then went back to serving only HTTP 500 errors from Cloudflare, then came back online again. I raised it in the HN discussion and @jgrahamc responded minutes later. https://news.ycombinator.com/item?id=31821290 https://news.ycombinator.com/item?id=31821290
gouggoug 4y ago
I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?
- yuliyp 4y ago
  Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.
  - wstuartcl 4y ago
    also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version). Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits). Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue. I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
  - crvdgc 4y ago
    Sounds like a racing condition. A lock (algorithmical or just through communication) should have been used.
    - trhway 4y ago
      Lock is another huge failure mode.

jgrahamc 4y ago

Well, the "we can't reach these data centers at all and need to go through the break glass procedure" was pretty "ouch" also.

exikyut 4y ago
I can('t) imagine, yikes. Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/ https://rachelbythebay.com/w/2019/01/20/quiet/ Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha. I would be very interested to hear more about how the break-glass process worked.

michaelmior 4y ago

This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.

scottlamb 4y ago
I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.
- michaelmior 4y ago
  You're probably right, but if they're going to mention it as part of the problem, I would want to see it as part of the solution. However, I agree that they certainly shouldn't name names.

dangrossman 4y ago

I think I experienced first-hand the moment those network engineers were reverting their own reverts, breaking the web again. For example, DoorDash.com had come back online, then went back to serving only HTTP 500 errors from Cloudflare, then came back online again. I raised it in the HN discussion and @jgrahamc responded minutes later. https://news.ycombinator.com/item?id=31821290 https://news.ycombinator.com/item?id=31821290

gouggoug 4y ago

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?

yuliyp 4y ago
Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.
- wstuartcl 4y ago
  also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version). Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits). Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue. I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
- crvdgc 4y ago
  Sounds like a racing condition. A lock (algorithmical or just through communication) should have been used.
  - trhway 4y ago
    Lock is another huge failure mode.