6 ms·

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. Those logical clusters also included network control jobs in other physical locations." So the automation equivalent of a human driven command that says "deschedule these core jobs in another region". Maybe someone needs to write a paper on Fault tolerance in the presence of Byzantine Automations (Joke. There was a satirical note on this subject posted here yesterday.)

tpaschalis 7y ago
Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense. [0] https://scholar.harvard.edu/files/mickens/files/thesaddestmoment.pdf https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
- carlsborg 7y ago
  My view on this: System engineering is as important as the algorithms. And the system engineering team should have at least some grey hair.
  - hguant 7y ago
    Working in a systems engineering position for a year and a half now: the grey hair comes to you.
- mxuribe 7y ago
  Hadn't read this before; very funny...and as noted, true!
dnautics 7y ago
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.
- londons_explore 7y ago
  It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
  - karlding 7y ago
    I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
    - log_n 7y ago
      You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance. Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
  - smileypete 7y ago
    Use a 28k modem? :)
    - _n_b_ 7y ago
      An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.
mslot 7y ago
At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability. At some point people realized multi-server systems within one AZ are prone to failure. They then started deploying their system redundantly to multiple AZs within the same region to increase availability. This helped, but created consistency issues. To fix this people started building multi-AZ software systems, creating dependencies across AZs that weakened overall availability. At some point people realized multi-AZ systems within one region are prone to failure. They then started deploying their system redundantly to multiple regions of the same cloud platform to increase availability. This helped, but created consistency issues. To fix this people started building multi-region software systems, creating dependencies across regions that weakened overall availability. At some point people realized multi-region systems within one cloud platform are prone to failure. They then started deploying their system redundantly to multiple cloud platforms to increase availability. This helped, but created consistency issues. To fix this people started building multi-cloud software systems, creating dependencies across cloud platforms that weakened overall availability.
- log_n 7y ago
  Humanity's colonization of the stars will eventually be spurred on by the desire improve the resiliency of SaaS.
- asark 7y ago
  I honestly think a one- or two-server setup with scripted server re-creation, proven in Vagrant or whatever, and maybe using Docker but only to isolate services on those one or two servers and make it easier to re-create, tested frequently by spinning up local dev copies, and (obviously) with backups, is probably a stabler, cheaper, and higher-availability set-up for the vast majority of use cases. Even if predicted, scheduled downtime is somewhat higher, it's probably worth it for the many benefits. But no, cloud everything.
- ohum 7y ago

tpaschalis 7y ago

Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense. [0] https://scholar.harvard.edu/files/mickens/files/thesaddestmoment.pdf https://scholar.harvard.edu/files/mickens/files/thesaddestmo...

carlsborg 7y ago
My view on this: System engineering is as important as the algorithms. And the system engineering team should have at least some grey hair.
- hguant 7y ago
  Working in a systems engineering position for a year and a half now: the grey hair comes to you.
mxuribe 7y ago
Hadn't read this before; very funny...and as noted, true!

dnautics 7y ago

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

londons_explore 7y ago
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
- karlding 7y ago
  I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
  - log_n 7y ago
    You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance. Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
- smileypete 7y ago
  Use a 28k modem? :)
  - _n_b_ 7y ago
    An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.

mslot 7y ago

At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability. At some point people realized multi-server systems within one AZ are prone to failure. They then started deploying their system redundantly to multiple AZs within the same region to increase availability. This helped, but created consistency issues. To fix this people started building multi-AZ software systems, creating dependencies across AZs that weakened overall availability. At some point people realized multi-AZ systems within one region are prone to failure. They then started deploying their system redundantly to multiple regions of the same cloud platform to increase availability. This helped, but created consistency issues. To fix this people started building multi-region software systems, creating dependencies across regions that weakened overall availability. At some point people realized multi-region systems within one cloud platform are prone to failure. They then started deploying their system redundantly to multiple cloud platforms to increase availability. This helped, but created consistency issues. To fix this people started building multi-cloud software systems, creating dependencies across cloud platforms that weakened overall availability.

log_n 7y ago
Humanity's colonization of the stars will eventually be spurred on by the desire improve the resiliency of SaaS.
asark 7y ago
I honestly think a one- or two-server setup with scripted server re-creation, proven in Vagrant or whatever, and maybe using Docker but only to isolate services on those one or two servers and make it easier to re-create, tested frequently by spinning up local dev copies, and (obviously) with backups, is probably a stabler, cheaper, and higher-availability set-up for the vast majority of use cases. Even if predicted, scheduled downtime is somewhat higher, it's probably worth it for the many benefits. But no, cloud everything.
ohum 7y ago