5 ms·

We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling th

We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling the savior regions).

mkl 7y ago
That sounds fascinating! How often does your team have to leap into action?
- aaronblohowiak 7y ago
  We don’t usually discuss the frequency of unplanned failovers, but I will tell you that we do a planned failover at least every two weeks. The team also uses traffic shaping to perform whole system load tests with production traffic, which happens quarterly.
  - justinator 7y ago
    Do you do any chaos testing? Seems like it would slot right in, there.
    - Zobat 7y ago
      I'd say yes. I heard about this tool just a week ago at a developer conference. https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
    - arainwater 7y ago
      they have invented the term, so probably yes :)
    - a_t48 7y ago
      Netflix was a pioneer of chaos testing, right? https://en.m.wikipedia.org/wiki/Chaos_engineering https://en.m.wikipedia.org/wiki/Chaos_engineering
    - aaronblohowiak 7y ago
      https://www.oreilly.com/library/view/chaos-engineering/9781491988459/ https://www.oreilly.com/library/view/chaos-engineering/97814... ;)
- azimuth11 7y ago
  I think some Google engineers published a free Meap book on service relatability and uptime guarantees. Seemingly counterintuitive, scheduling downtime, without other teams’ prior knowledge, encourages teams to handle outages properly and reduce single points of failure, among other things.
  - fnord123 7y ago
    Service Reliability Engineering is on OReilly press. It's a good book. Up there with ZeroMQ and Data Intensive Applications as maybe the best three books from OReilly in the past ten years.
    - fnord123 7y ago
      Derp, Site Reliability Engineering. https://landing.google.com/sre/books/ https://landing.google.com/sre/books/

We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling the savior regions).

mkl 7y ago

That sounds fascinating! How often does your team have to leap into action?

aaronblohowiak 7y ago
We don’t usually discuss the frequency of unplanned failovers, but I will tell you that we do a planned failover at least every two weeks. The team also uses traffic shaping to perform whole system load tests with production traffic, which happens quarterly.
- justinator 7y ago
  Do you do any chaos testing? Seems like it would slot right in, there.
  - Zobat 7y ago
    I'd say yes. I heard about this tool just a week ago at a developer conference. https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
  - arainwater 7y ago
    they have invented the term, so probably yes :)
  - a_t48 7y ago
    Netflix was a pioneer of chaos testing, right? https://en.m.wikipedia.org/wiki/Chaos_engineering https://en.m.wikipedia.org/wiki/Chaos_engineering
  - aaronblohowiak 7y ago
    https://www.oreilly.com/library/view/chaos-engineering/9781491988459/ https://www.oreilly.com/library/view/chaos-engineering/97814... ;)
azimuth11 7y ago
I think some Google engineers published a free Meap book on service relatability and uptime guarantees. Seemingly counterintuitive, scheduling downtime, without other teams’ prior knowledge, encourages teams to handle outages properly and reduce single points of failure, among other things.
- fnord123 7y ago
  Service Reliability Engineering is on OReilly press. It's a good book. Up there with ZeroMQ and Data Intensive Applications as maybe the best three books from OReilly in the past ten years.
  - fnord123 7y ago
    Derp, Site Reliability Engineering. https://landing.google.com/sre/books/ https://landing.google.com/sre/books/