3 ms·
Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my
by jedberg 6y ago
Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered.
If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made.
The second question I would ask is what kind of Chaos Engineering they are doing. Are they doing major traffic failover tests? Rapid upscaling tests? Random terminations?
Those are probably the first two things I'd want to solve.
- pcnix 6y agoIn my experience, ALB scaling is opaque, and constantly lags behind herds of traffic. This is very noticeable if you're running services that experience sudden spikes, there's a noticeable lag of ALB scaling. That said though, it does do it's job for the most part.
- rhizome 6y agoI hope all of these questions would be asked only after everything was working again!
- jedberg 6y agoOf course. :) These are things I would ask during a post-mortem.