3 ms·
> a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure Most of it hard earned, after having to deal with thousands of ou
by bostik 3y ago
> a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure
Most of it hard earned, after having to deal with thousands of outlier events.
An old friend used to run the Finnish election results services' public-facing backend, and was an early AWS customer. He broke their load-balancers - twice.
Both times he got in touch with AWS support well in advance, asking to pre-scale and pre-warm the load balancers because he was expecting a major traffic surge once the results started to come in. First time around, he was confidently told that he shouldn't worry, since AWS can take on any amount of traffic.
On the election night, AWS load balancers failed to scale in time. The traffic estimates my friend had provided were indeed accurate within an order of magnitude, but AWS hadn't believed his numbers. NOBODY could have a service that legitimately required scaling from a few hundred requests per second to 2M requests per second in about one minute. Apparently their on-call engineers got burned badly that night.
Couple of years later, he approached AWS support again, with the same request. That time he was taken more seriously, and was assured that the autoscaling algorithms were now much smarter and their relevant engineering teams had prepared the ground for more rapid autoscaling needs. They were confident they didn't need to pre-scale the fleet.
The scaling wasn't still fast enough and their on-call team had to _again_ manually force their balancers to stay just ahead of the demand with the massive surges coming in waves. So much for the fine-tuned scaling algorithms.
Fast-forward to today, and AWS have a specialist "hot launch" service offering where they work with the customers up front to make sure their launches have sufficient capacity available and their perimeter load balancers properly pre-warmed to absorb these kinds of supposedly one-off situations.