2 ms·
Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer. Deployment broke. Yeah, that probably should have bee
by xorcist 6y ago
Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer.
Deployment broke. Yeah, that probably should have been caught. Even if it didn't, monitoring should have caught the stale load balancer config. It didn't, for some reason unknown to us.
These things happen. Things break. The autoscaler then proceeded to kill customer traffic. That was the part that worked as designed, so another design would have avoided escalating the situation (if you forgive some armchair engineering here).
- cutemonster 6y agoMaybe what's missing was to test high load and autoscaling to 5x the traffic on, say, a holiday (when Slacks customers don't work)