6 ms·
I work in the entertainment / ticketing industry and we've been burned badly before by relying on AWS' Elastic Load Balancer due to sudden & unexpected traffic
by napkindrawing 11y ago
I work in the entertainment / ticketing industry and we've been burned badly before by relying on AWS' Elastic Load Balancer due to sudden & unexpected traffic spikes.
From the article: "Elastic Load Balancer (ELB): [...] It scales without your doing anything. If it sees additional traffic it scales behind the scenes both horizontally and vertically. You don’t have to manage it. As your applications scales so is the ELB."
From Amazon's ELB documentation: "Pre-Warming the Load Balancer: [...] In certain scenarios, such as when flash traffic is expected [...] we recommend that you contact us to have your load balancer "pre-warmed". We will then configure the load balancer to have the appropriate level of capacity based on the traffic that you expect. We will need to know the start and end dates of your tests or expected flash traffic, the expected request rate per second and the total size of the typical request/response that you will be testing."
- CoffeeDregs 11y agoAnother gotcha is that ELB appears to load balance based on the IP addresses of the requests... We had private VPC/IP workers talking hundreds of requests per second to a non-sticky-session, public ELB fronted service (... don't ask why ...) and experienced really strange performance problems. Latency. Errors. What? Deployed a second private ELB fronting the same service and pointed the workers at it. No more latency. No more errors. The issue appeared to have been that the private IP workers all would transit the NAT box to get to the public service and the ELB seemed to act strangely when 99.99% of the traffic was coming from one IP address. The private ELB saw requests from each of the individual IP addresses of the workers and acted a lot better. Or something.
- samstave 11y agoElbs are one of the known biggest weaknesses of aws... Their whole position on them is super opaque and prewarming is still an issue. I'll write more about this later, but so many people have had outages due to aws' inability to properly size these things.
- deleted 11y ago[deleted]
- devNoise 11y agoI went to a meetup about 2 years ago and one of the engineers from CloudMine gave a talk about load balancing on AWS. CloudMine ended up dumping ELB for HAProxy to handle their scaling needs.
- morenoh149 11y agohow does HAProxy compare to OpsWorks? the HAProxy wikipedia page mentions OpsWorks is based on it
- no1youknowz 11y agoYou'd be surprised about how many people don't know this. I had an expectation to scale past 1B users. I was trialling AWS when I realised through testing that it was this way. It could not deal with sudden spikes of traffic. Suffice to say, I went elsewhere.
- CodyReichert 11y agoWhere did you go, if you don't mind expanding?
- no1youknowz 11y agoI don't mind. I went with dedicated hosting. I found a supplier which had their own scalable infrastructure. They already had clients which had ad server type applications that scaled into the Billions and could handle traffic spikes. With that type of setup, it was a no brainer. I'm a sysadmin with over 10 years with Linux. So for me to setup and support servers is pretty trivial. The agreement I had with the supplier. They managed the network and hardware 24/7. I managed the setup and support of the servers from the OS up. This arrangement worked well and I had zero downtime.
- sbierwagen 11y agoI don't mind. I went with self hosting. I found a supplier which had their own scalable infrastructure. That's a little vague. By "self-hosting" you mean Linux VMs, like EC2, right, or something more abstracted than that? What supplier?
- no1youknowz 11y agoSorry, I just updated the post. I meant dedicated hosting. So bare-metal machines. If you want to know the supplier. They are called Mojohost. http://www.mojohost.com/ http://www.mojohost.com/
- 11y ago
- a-priori 11y agoI wish Amazon would switch to a 'provisioned throughput' model for ELB like they have for DynamoDB, where you say what level of throughput you want to support and you're billed on that rather than actual traffic. Then they keep sufficient capacity available to support that service level. So if you expect flash traffic, you just bump up your provisioned throughput. Simple and transparent.
- ceejayoz 11y agoThat would be a very cool offering.
- toomuchtodo 11y agoYou can contact AWS support if needed, and they'll warm up the ELB ahead of time. http://serverfault.com/a/321371 http://serverfault.com/a/321371 https://forums.aws.amazon.com/thread.jspa?threadID=76834 https://forums.aws.amazon.com/thread.jspa?threadID=76834 It's not perfect, but works in a pinch.
- toby 11y agoThis might be of interest, Netflix pre-scales based on anticipated demand: http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scaling.html http://techblog.netflix.com/2013/11/scryer-netflixs-predicti...
- garyrichardson 11y agoLink to the documentation? I thought this was changed over a year ago to not requiring pre warming?
- manigandham 11y agoNginx running on a tiny instance can load balance 100k connections at thousands of requests per second. The network bandwidth for the instance will probably be saturated way before the CPU/RAM becomes a problem. ELB (and most other managed service load balancers) are overpriced and not great at what they do. The advantage with them is easier setup and lack of maintenance. If you're running a service with hundreds of millions or billions of requests, it's just far more effective in every way to use some small load balancing instances instead. Their Route53 service makes the DNS part easy enough with health checks.
- matt_wulfeck 11y agoWhy do you say they're overpriced? I would say for most apps their downright cheap. Especially since you spend so little time tinkering/monitoring/worrying about them. Most people just want to work on their app not manage Nginx configs.
- manigandham 11y agoThere is absolutely a tradeoff (as with everything in life) but in the context of this thread talking about scale with 100s of millions of requests, gigabytes of bandwidth and large spikes - it's far better to just host your own load balancers. Most people (and apps) likely won't hit this scale so ELB is just fine. If you do though, ELB is just pricey and not really that great.
- mnutt 11y agoAfter testing ELB and seeing the scaling issues, we ended up going to a pool of HAProxies + weighted Route53 entries. Route53 does a moderately good job of balancing between the HAProxies, and the health checks will remove an HAProxy if it goes bad. HAProxy itself is rock solid. The first bottleneck we came across was HAProxy bandwidth, so make sure the instance type you select has enough for how much bandwidth you expect to use.
- toomuchtodo 11y agoDo health checks work within a VPC? My understanding was they don't, so this only works for externally facing services. I agree Haproxy is solid, but ELBs are wonderful for internal microservices. If you do decide to use Haproxy for microservices internally, I highly recommend Synapse from AirBnB: https://github.com/airbnb/synapse https://github.com/airbnb/synapse
- frugalmail 11y agoRuby, High Availablity and High Scalability? Despite idempotency, I'm not sure how comfortable I am with that.
- mryan 11y agoSynapse is a service discovery framework. Essentially, it just writes HAProxy config files based on discovered upstreams - it does not receive any requests itself. The scalability is handled by HAProxy.
- frugalmail 11y agoI was under the impression that HAProxy is what it is powering Amazon's ELB service.