14 ms·
A Beginner's Guide to Scaling to 11M+ Users on Amazon's AWS
- edvinasbartkus 11y agoIt would be cool if they would show the range of costs ($$$) for each step of growth. My fear is that if you do everything by the book the costs correlate with growth.
- boothead 11y agoIt would also be interesting to see that as a rough $$$/user. It would be very interesting to see how much you need to be making from each user to cover hosting.
- chucky_z 11y agoI did this migration recently and we're spending about 1.75 cents per user. We could do it for cheaper, but we've recently had some issues that were absolutely trivial to resolve with AWS, that would have been very difficult with our previous hosting provider.
- Roodgorf 11y agoThis hits on something in the calculation that I feel is very hard to factor in, the cost of development time. Sure, there are plenty of ways to do these things cheaper on a hardware/software cost per user basis, but more often than not I've found that we can get changes out so much faster in AWS that you're easily saving thousands in developer time, which would seem to more than cover the extra cost to me.
- snaily 11y agoPer month, I take it?
- chucky_z 11y agoCorrect.
- beachstartup 11y agoi run an infrastructure startup. the rule of thumb is once you hit $20-99k/month, you can cut your AWS bill in half somewhere else. sites in this phase generally only use about 20% of the features of aws. the other rule of thumb is once you hit six figures/month, you're probably spending someone else's money, are locked in to their stack, or just don't really care to begin with, so there's no point in telling/selling you otherwise.
- miseg 11y agoI've configured my web application to deploy to S3/Cloudfront for asset deployment. It's a PHP app. In the end, I might just pay a little more for a faster server. Keep things simple, everthing on the one app. It's a "normal" app (in the grand scheme of the Internet), so 10 users at a time would be high traffic already.
- falcolas 11y agoIf the PHP responses are relatively static, adding a bit of caching in front of it will improve the responsiveness and decrease the load dramatically. Simply adding a 5 minute cache let us scale one PHP application from 100 concurrent to "SSL & gzip require more CPU than PHP". We figured that was sufficient. More dynamic applications (like a commenting system) might feel better at 10-30 seconds of caching with expiration commands, but it will still help scale up significantly.
- miseg 11y agoBy caching, you mean like script execution caching that PHP accelerators give? https://en.wikipedia.org/wiki/List_of_PHP_accelerators https://en.wikipedia.org/wiki/List_of_PHP_accelerators Am I right in thinking that such caching comes built-in with PHP 5.5+ ?
- semerda 11y agoLook at Varnish Cache https://www.varnish-cache.org/ https://www.varnish-cache.org/ and Google's PageSpeed module on the server. https://developers.google.com/speed/pagespeed/?hl=en https://developers.google.com/speed/pagespeed/?hl=en
- miseg 11y agoThanks. That's quite added complexity in my scenario, which I think I would avoid.
- 11y ago
- napkindrawing 11y agoI work in the entertainment / ticketing industry and we've been burned badly before by relying on AWS' Elastic Load Balancer due to sudden & unexpected traffic spikes. From the article: "Elastic Load Balancer (ELB): [...] It scales without your doing anything. If it sees additional traffic it scales behind the scenes both horizontally and vertically. You don’t have to manage it. As your applications scales so is the ELB." From Amazon's ELB documentation: "Pre-Warming the Load Balancer: [...] In certain scenarios, such as when flash traffic is expected [...] we recommend that you contact us to have your load balancer "pre-warmed". We will then configure the load balancer to have the appropriate level of capacity based on the traffic that you expect. We will need to know the start and end dates of your tests or expected flash traffic, the expected request rate per second and the total size of the typical request/response that you will be testing."
- CoffeeDregs 11y agoAnother gotcha is that ELB appears to load balance based on the IP addresses of the requests... We had private VPC/IP workers talking hundreds of requests per second to a non-sticky-session, public ELB fronted service (... don't ask why ...) and experienced really strange performance problems. Latency. Errors. What? Deployed a second private ELB fronting the same service and pointed the workers at it. No more latency. No more errors. The issue appeared to have been that the private IP workers all would transit the NAT box to get to the public service and the ELB seemed to act strangely when 99.99% of the traffic was coming from one IP address. The private ELB saw requests from each of the individual IP addresses of the workers and acted a lot better. Or something.
- samstave 11y agoElbs are one of the known biggest weaknesses of aws... Their whole position on them is super opaque and prewarming is still an issue. I'll write more about this later, but so many people have had outages due to aws' inability to properly size these things.
- deleted 11y ago[deleted]
- zdw 11y agoAWS is great and all (especially if you need a lot of CPU cycles), but this should come with the caveat that if you're under 1K users AWS probably isn't the best solution - conventional VPS hosting is usually more cost effective.
- otterley 11y agoYou might not be plugging in all inputs into your cost calculus -- namely, the amount of labor you spend reconfiguring your datacenter to accommodate change.
- griffordson 11y agoThis seems to be an unpopular opinion on HN, but you are correct. It is possible to generate millions in revenue with 1 or 2 devs. If you manage to do that, paying a higher than average price for AWS is a no brainer.
- vidarh 11y agoHow much revenue you can generate per developer is totally irrelevant. If you generate millions in revenue but server costs eats it all up, paying a 3x+ premium to run on AWS can easily bankrupt you. By all means, if your server costs are inconsequential to your bottom line, go nuts. I've just moved a client off EC2 because the premium they were paying would have been a massive problem. The 85% reduction in hosting cost has bought them months of extra runway. Their operational costs related to their hosting also dropped - there's simply been fewer issues to deal with. I'm sure there are instances where AWS is fine. But there are also plenty of cases where it is a matter of survival to cut those costs.
- griffordson 11y agoAll good points. I should have been more specific. You can generate > $1M in profit with 1 or 2 devs, and in that case, AWS is a no brainer. In my experience, it is much more difficult to manage dedicated hardware in multiple data centers for high availability with only 1 or 2 devs. The opportunity costs alone in that case can kill you. But I don't live in a world where runway is a consideration so YMMV. At the time I commented, the parent post was getting downvoted. I've seen that knee jerk reaction on HN multiple times, and that is what prompted my comment.
- falcolas 11y agoHoo boy. Here we go. The problem with AWS reps is that they only see everything as working perfectly, with no possibility for downtime of their services. RDS is great, but only to a certain level. You'll still need to pull it off RDS once you reach that service's capacity (much sooner than their 10m user mark). They also keep pushing Aurora, but without telling us what the tradeoffs are for the high availability. Based on the responses so far (MySQL backed by InnoDB), it appears to be based on a technology similar to Galara, which has a lot of caveats for its use, especially with multiple writers. Don't depend on Elastic Scaling for high availability - when an AZ is having issues, the AWS API will either be down or swamped, so you want to have at least 50% extra capacity at all times, if you want high availability. Using their scaling numbers, your costs start spiking at 10 users. Realistically, with intelligent caching (even something as simple as Nginx caching), you can easily support several thousand users just fine with a t2 style instance, either a small or micro. Splitting services onto different hosts not only increases your hosting costs, it increases the workload on your developers/admins and likeliness of failure. DR: Don't wait until you have over a thousand users to have multiple instances in different AZs. The cost of duplicating a t2.small across an AZ is small compared to lost users or sales. Automation: Be prepared for vendor lockin if you use Amazon's solutions. Also be prepared for their APIs being unavailable during times of high load or during AZ failures. > Lambda [...] We’ve done away with EC2. It scales out for you and there’s no OS to manage. The biggest problem with Lambda right now are the huge latency costs with cold lambda instances. You'll get a pretty good 95% percentile response times, but that other 5% will be off-the-chart bad. In summary, AWS has a lot of great toys, and can absolutely be used for scaling up to silly levels. However, most who have done this degree of scaling do not do so using AWS tools.
- late2part 11y agoAgreed - they are a great solution for small teams that are growing fast and don't have predictability. But, once you have some level of predictability and scale, it makes sense to move off to something much higher performance and lower cost. Until you become a decrepit Fortune 50 company and can't manage an IT department due to bloat, and it's cheaper to outsource.
- 11y ago
- frik 11y agoStart with SQL and only move to NoSQL when necessary. Users > 10.000.000+: Moving some functionality to other types of DBs (NoSQL, graph, etc) Interesting insights from Amazon. While not everyone will agree, there is apparently some truth in it.
- collyw 11y agoThe isn't usually a good reason to start with a NoSQL solution, except for buzzwords on your CV.
- UK-AL 11y agoOr the fact there data sets that fit nosql databases seem to work far better. Patient records is one I can think off.
- billmalarky 11y agoThese data sets can be easily handled by Postgresql's JSONB data type.
- collyw 11y agoOr a normal table.
- deleted 11y ago[deleted]
- UK-AL 11y agoCan but its not the best. This is obvious since most patient record systems these days do not use SQL. They use things like MUMPS.
- billmalarky 11y agoNormal tables don't elegantly handle certain types of data. I'm not saying you can't make it work, but there's a valid reason why people choose to use document stores over traditional tables in certain cases.
- aganders3 11y agoThere is a lot of pessimism about AWS in here. Does anyone have a link to a similar article from the roll-your-own perspective? I am comfortable writing small Python web apps (i.e. running on a single instance with SQL server on the same box), but scaling on my own is a mystery to me at this point.
- greenleafjacob 11y agoEtsy's blog has some good posts [1]. [1] https://codeascraft.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ https://codeascraft.com/2012/03/13/making-it-virtually-easy-...
- krat0sprakhar 11y agoThere's actually an account on Medium - AWSActivate which publishes a lot of useful stuff like this. Check it out - https://medium.com/@awsactivate https://medium.com/@awsactivate
- chinathrow 11y ago> Users > 1,000,000+ [...] > Put caching in front of the DB Isn't that a little late?
- bpicolo 11y agoNot really. SQL DBs can handle a crapload of traffic. Maybe not a million all at once by default, but generally with a million users you're looking at << 50k on site at any given time, and if you split reads off to replicas you can handle a lot of scale. In my experience, 50-100k qps (writes) is where SQL starts to get especially hard
- ufmace 11y agoI gotta wonder why they want to start splitting things up at only 10 users. Unless your uses are really active all day and you have a lot of very processor-intensive stuff going on, I wouldn't think you need that until well over 1000s of users.
- chillydawg 11y agoAs with almost everything like this, "users" is a completely undefined term and the service could be anything. If all you want to do is serve wordpress or whatever, then sure this kind of cookie cutter approach is no problem, but for most bespoke web services or business infrastructures you pretty much just have to analyse all thise stuff yourself and figure out the most cost effective way to do it all.
- vitoc 11y agoComing from an environment that uses lots of AWS resources to handle scaling requirements across different kinds of workloads on different linked accounts, one of the challenges we faced was to communicate and collaborate efforts and its impacts on cost efficiency. Typically our best environment isn’t the product of a singular design effort at the individual level, but many times emergent based on differing opinions and trials to assert assumptions in practice. We built a tool, https://liquidsky.singtel-labs.com https://liquidsky.singtel-labs.com, to help with this.
- novaleaf 11y agoI went with Google Cloud, and my 1 to 10 user infrastructure is the same as 1million+ users: 1) Use Load Balancer + Autoscaler for all service layers. This effectively makes each layer a cloud of on-demand microservices. 2) Use Cloud Datastore: (NoSql) Maybe I lucked out that I don't have complex relational data to store, but Cloud Datastore abstracts out the entire DB layer, so I don't have to worry about scaling/reliability ever. ... aside from random devops stuff, that's pretty much it. The key point is to "cloudify" each layer of the infrastructure.
- chillydawg 11y agoWonderful problem if you can get it :)
- vgt 11y agoThis story doesn't get told enough. Most of Google Cloud is built to operate the same way with 1 user or 1m users. And in many cases, Google doesn't charge you for the "scaling vector", whereas AWS will, and will sometimes even require a separate product (see Firehose). Things like Load Balancer not requiring pre-warming, PubSub seamlessly scaling, Datastore and AppEngine seamlessly scaling. This is especially obvious on the product I work on, BigQuery: - We had a customer who did not do anything special, did not configure anything, didn't tell us, and ingested 4.5 million rows per second using our Streaming API for a few hours. - We frequently find customers who scale up to 1PB-size without ever talking to us. I can be their first point of contact at Google.. after they're at that scale. - Unlike traditional Databases, BigQuery lets you use thousands of cores for the few seconds your query needs them, and you only pay for the job. If I were to translate this to VM pricing, BigQuery gives you ability to near-instantly fire up thousands VMs, shut them down in 10 seconds, and only pay per-second. Customers like that kind of thing :) Disclosure: Shamelessly biased
- lenkite 11y agoWholeheartedly agree! Google Cloud is so severely underrated as a platform for scalable web-apps. If you use the cloud data store and web-app common sense, there is no re-architecting required for users in the range of 100->million+. And _much_ cheaper and lesser operational overhead compared to EC2/AWS. The disadvatange is that you have to use the Google stack and API's, but for new apps this is worth it.
- clentaminator 11y agoA lot of comments in this thread are voicing concerns over the marketed cost/performance benefits of AWS and the reliability of their services in the case of region failure e.g. the API services goes down. But are there benefits to using Amazon's more high-level services such as SQS and SNS which, supposedly, replicate their configuration state and data across multiple regions, in terms of reliability? For instance, on a per-instance basis AWS might be more expensive than a bare-metal provider, and there's nothing to stop you running your own RabbitMQ instance. But SQS messages are replicated across three regions, so if you were building an equivalent service you'd need three instances in different regions and a reliable distributed message queue. So does that additional complexity/cost make SQS at all worthwhile? Or does it come down to the fact that, while your own hand-rolled service would require more management, your potential message throughput at a given cost would be much higher than with SQS?
- VOYD 11y ago11m+ isn't scale. 111m+ is scale.
- frik 11y agoHow much would it cost Amazon to run Amazon.com on AWS? (Amazon.com retail website runs on EC2 and AWS since 2010)
- blahshaw 11y agoI'd be surprised if Amazon didn't run on AWS.
- grepory 11y agoI would argue that you need monitoring significantly sooner than 500,000 users. I guess, until then, you just use Twitter noise for monitoring? Seems like pretty bad customer experience. If I have something in an environment that I would start to consider "production" (i.e. someone relies on my product to do something regularly), then I'd have monitoring regardless of the number of users. Even something as simple as, "Am I returning valid data from GET /"?
- nzoschke 11y agoThis is a great article! I see a lot of pessimism about AWS in this thread but its unfounded. The sheer number of success stories on AWS at every scale is amazing. This guide demonstrates the diverse set of services AWS offers for customers from zero to Netflix. AWS is world-class engineering and operations that can be summoned by a single API call. There might be ways to cut monthly costs on other providers, but many people forget to factor in your time to research, design stand up and operate software. I'd go all in on SQS, with all it's design quirks and potential costs, over rolling my own RabbitMQ cluster on Digital Ocean any day. I'm biased, working full time on open source tools to help beginners on AWS at Convox (https://github.com/convox/rack https://github.com/convox/rack), but frankly there's not a better time to build and scale your business on AWS. The platform is pure productivity with very little operational overhead.
- zAy0LfpBZLC8mAC 11y ago> AWS is world-class engineering and operations that can be summoned by a single API call. Are they still doing world-class ICMP filtering, breaking PMTUD?
- morenoh149 11y agovideo of this material here https://www.youtube.com/watch?v=vg5onp8TU6Q https://www.youtube.com/watch?v=vg5onp8TU6Q
- meirelles 11y agoIMHO many companies save time, money or both using AWS. Others fail miserably trying to do so. I like very much the Amazon's AWS. I use them extensively. But apparently some folks goes a little crazy to adopt cloud services as final solution for every use case. They have no idea how much traffic a real high-end server fully loaded with memory and SSD disks should handle these days.