4 ms·

The main mistake IMO is that, the day before the outage, they made a significant Consul-related infra change. Then they have this massive outage, where Consul i

by yashap 5y ago

The main mistake IMO is that, the day before the outage, they made a significant Consul-related infra change. Then they have this massive outage, where Consul is clearly the root cause, but nobody ever tries rolling that recent change back? That’s weird. I went into more detail here: https://news.ycombinator.com/item?id=30015826 https://news.ycombinator.com/item?id=30015826 The outage occurring could certainly happen to anyone, but it taking 72 hours to resolve seems like a pretty fundamental SRE mistake. It’s also strange that “try rollbacks of changes related to the affected system” isn’t even acknowledged as a learning in their learnings/action items section.

tptacek 5y ago
That doesn't sound accurate. Wasn't the major change they ended up rolling back Consul streaming, which they'd enabled months before, and had been slowly rolling out?
- twblalock 5y ago
  Right, but the day before the outage, they enabled streaming for a service that didn't have it turned on. That's a discrete config change, the day before the outage.
- yashap 5y ago
  > Several months ago, we enabled a new Consul streaming feature on a subset of our services. This feature, designed to lower the CPU usage and network bandwidth of the Consul cluster, worked as expected, so over the next few months we incrementally enabled the feature on more of our backend services. On October 27th at 14:00, one day before the outage, we enabled this feature on a backend service that is responsible for traffic routing. As part of this rollout, in order to prepare for the increased traffic we typically see at the end of the year, we also increased the number of nodes supporting traffic routing by 50% So they rolled out a pretty significant Consul related change the day before their massive Consul outage began. They’d been doing a slow rollout, but ramping it up a bunch is a significant change.
faitswulff 5y ago
It's possible they deal with so much load that they considered a day's worth of traffic to be sufficient load testing: > The system had worked well with streaming at this level for a day before the incident started, so it wasn’t initially clear why it’s performance had changed. And a short note later on how much load their caching system sees: > These databases were unaffected by the outage, but the caching system, which regularly handles 1B requests-per-second across its multiple layers during regular system operation, was unhealthy.

by yashap 5y ago

tptacek 5y ago

That doesn't sound accurate. Wasn't the major change they ended up rolling back Consul streaming, which they'd enabled months before, and had been slowly rolling out?

twblalock 5y ago
Right, but the day before the outage, they enabled streaming for a service that didn't have it turned on. That's a discrete config change, the day before the outage.
yashap 5y ago
> Several months ago, we enabled a new Consul streaming feature on a subset of our services. This feature, designed to lower the CPU usage and network bandwidth of the Consul cluster, worked as expected, so over the next few months we incrementally enabled the feature on more of our backend services. On October 27th at 14:00, one day before the outage, we enabled this feature on a backend service that is responsible for traffic routing. As part of this rollout, in order to prepare for the increased traffic we typically see at the end of the year, we also increased the number of nodes supporting traffic routing by 50% So they rolled out a pretty significant Consul related change the day before their massive Consul outage began. They’d been doing a slow rollout, but ramping it up a bunch is a significant change.

faitswulff 5y ago

It's possible they deal with so much load that they considered a day's worth of traffic to be sufficient load testing: > The system had worked well with streaming at this level for a day before the incident started, so it wasn’t initially clear why it’s performance had changed. And a short note later on how much load their caching system sees: > These databases were unaffected by the outage, but the caching system, which regularly handles 1B requests-per-second across its multiple layers during regular system operation, was unhealthy.