6 ms·

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up

by m12k 6y ago

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances in the list were the few oldest ones, so when they then scaled down the number of instances again, these old instances were the first to be shut down, causing the outage, because the instances that should have taken over were not listed in the stale load balancer host list.

cuu508 6y ago
That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.
- jacksonpollock 6y ago
  agreed!
- kergonath 6y ago
  This. The linked post is unreadable.
- yawaramin 6y ago
  BLUF style–Bottom Line Up Front. Often used by the military for field action reports.
- pradn 6y ago
  The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.
  - antoinealb 6y ago
    Here is an example of such a postmortem format if people wonder: https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/
vondur 6y ago
It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.
- staticassertion 6y ago
  Feels like you've just described every single bug ever.
  - m463 6y ago
    Even at the start I'm sure Grace Hopper didn't anticipate a computer with a moth in it. https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k.jpg https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k....
  - deleted 6y ago
    [deleted]
- Nextgrid 6y ago
  Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.
  - deleted 6y ago
    [deleted]
  - bstar77 6y ago
    I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes. The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.
kinow 6y ago
Thanks! They should include this TL;DR in their post!
devit 6y ago
Seems the root cause is that instead of using an established system like Kubernetes, they rolled their own and didn't engineer and test it as well as such an endeavor would have required.
debt 6y ago
TL;DR meta point is: "The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy for all of our ingress load-balancing" The legacy system supporting Slack in production was heavily resource-constrained as they were moving to a new fancy system. Slack admits here that the legacy system likely wasn't getting the attention it needed and lo-and-behold it started failing in mysterious ways. Organizational failure by not properly calculating all the risks caused by rotating out part of their load-balancing system. They probably should've asked for more budget here to keep their existing system functional as they slowly transitioned to their new system. They admit that COVID caused all their systems to become stressed, they probably had appropriately budgeted for the transition to Envoy whenever they asked management(probably pre-covid). The team likely was never meant to support both the load they're now seeing during COVID while transitioning to a new system. Either way during any transition, there's a period where you must support both systems at full capacity until the legacy system can be gracefully decommissioned.
- sailfast 6y ago
  I really feel this comment. Especially during transition it’s hard to see the “to-be legacy” system as worthy of the effort because the new stuff will be here so soon! Until it isn’t because life happens and you’re left with a really shaky platform. These are tough investment decisions especially when resource constrained. As a rule, perhaps ensuring that current state platforms are secure, as you say, before attempting a migration is the best way to go, but one person’s critical work is another person’s “too much insurance.”
aagha 6y ago
I used to work for an Independent System Operator (ISO)[0] and we used to have a sometimes painful process for rolling out any changes to production. It's been a while and I don't remember it all, but it went something like this: 1. Fill out out a change request (CR) form and print it. 2. Have it signed by your manager and the managers of every system it touched, including business owners. 3. Attend the 2x a week meeting and explain your CR. In this meeting, explain what was happening, why, who authorized it, what to do if it failed, what to do if it initially worked but failed later (e.g, on a weekend). 4. Hope your CR passes the vote. 5. Implement your roll-out plan. This is a robust process. Where it breaks down--I felt--is when you need to fix a typo on the public-facing website that's managed by a CMS. 0 - https://www.eesi.org/files/070913_Jay_Caspary.pdf https://www.eesi.org/files/070913_Jay_Caspary.pdf

by m12k 6y ago

cuu508 6y ago

That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.

jacksonpollock 6y ago
agreed!
kergonath 6y ago
This. The linked post is unreadable.
yawaramin 6y ago
BLUF style–Bottom Line Up Front. Often used by the military for field action reports.
pradn 6y ago
The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.
- antoinealb 6y ago
  Here is an example of such a postmortem format if people wonder: https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/

vondur 6y ago

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

staticassertion 6y ago
Feels like you've just described every single bug ever.
- m463 6y ago
  Even at the start I'm sure Grace Hopper didn't anticipate a computer with a moth in it. https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k.jpg https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k....
- deleted 6y ago
  [deleted]
Nextgrid 6y ago
Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.
- deleted 6y ago
  [deleted]
- bstar77 6y ago
  I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes. The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.

kinow 6y ago

Thanks! They should include this TL;DR in their post!

devit 6y ago

Seems the root cause is that instead of using an established system like Kubernetes, they rolled their own and didn't engineer and test it as well as such an endeavor would have required.

debt 6y ago

TL;DR meta point is: "The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy for all of our ingress load-balancing" The legacy system supporting Slack in production was heavily resource-constrained as they were moving to a new fancy system. Slack admits here that the legacy system likely wasn't getting the attention it needed and lo-and-behold it started failing in mysterious ways. Organizational failure by not properly calculating all the risks caused by rotating out part of their load-balancing system. They probably should've asked for more budget here to keep their existing system functional as they slowly transitioned to their new system. They admit that COVID caused all their systems to become stressed, they probably had appropriately budgeted for the transition to Envoy whenever they asked management(probably pre-covid). The team likely was never meant to support both the load they're now seeing during COVID while transitioning to a new system. Either way during any transition, there's a period where you must support both systems at full capacity until the legacy system can be gracefully decommissioned.

sailfast 6y ago
I really feel this comment. Especially during transition it’s hard to see the “to-be legacy” system as worthy of the effort because the new stuff will be here so soon! Until it isn’t because life happens and you’re left with a really shaky platform. These are tough investment decisions especially when resource constrained. As a rule, perhaps ensuring that current state platforms are secure, as you say, before attempting a migration is the best way to go, but one person’s critical work is another person’s “too much insurance.”

aagha 6y ago

I used to work for an Independent System Operator (ISO)[0] and we used to have a sometimes painful process for rolling out any changes to production. It's been a while and I don't remember it all, but it went something like this: 1. Fill out out a change request (CR) form and print it. 2. Have it signed by your manager and the managers of every system it touched, including business owners. 3. Attend the 2x a week meeting and explain your CR. In this meeting, explain what was happening, why, who authorized it, what to do if it failed, what to do if it initially worked but failed later (e.g, on a weekend). 4. Hope your CR passes the vote. 5. Implement your roll-out plan. This is a robust process. Where it breaks down--I felt--is when you need to fix a typo on the public-facing website that's managed by a CMS. 0 - https://www.eesi.org/files/070913_Jay_Caspary.pdf https://www.eesi.org/files/070913_Jay_Caspary.pdf