8 ms·
The Amazon Prime Day 2023 AWS Bill
- mokarma 3y agoNaive question: What are they using EBS for? It seems unnecessary given all the Databases. Is that just local caching for EC2's?
- cj 3y agoEBS is just a networked hard drive, so they could really be using it for anything storage related. Is Amazon’s general architecture for their retail site publicly described anywhere?
- yowlingcat 3y agoWell, not to answer your question with a question, but what would you imagine backs all of those database services? Or, said another way, I'm not sure Corey Quinn is mapping the cost dependency graph correctly by giving this breakdown as mutually exclusive (from the standpoint of AWS internally).
- OJFord 3y agoWell, without precluding other use, not even specifically caching but just disk for EC2 instances.
- ripper1138 3y agoIt’s disk storage for EC2
- nonameiguess 3y agoAt minimum, root volumes for the VMs. Theoretically, you could load immutable machine images from the network and run entirely off of in-memory filesystems if you persist nothing past instance shutdown (similar to how extremely cautious people might run Tails booted off USB on a laptop with no hard drive), but that won't actually save cost since memory is more expensive than disk anyway.
- thayne 3y agoI don't think you can even technically do that in AWS. I don't think there is any way to detach the root volume from a running instance, or use an immutable network image to boot from. However, for many server workloads, operating entirely would be reasonable. Often you just need the operating system kernel and your server software, and maybe a monitoring agent. And all of that will be loaded in memory anyway.
- fragmede 3y agoDid any AWS customers experience unavailability during prime day, eg capacity issues launching instances, due to prime day taking precedence over other customers? If there are, they're under NDA so we'd never know.
- other_herbert 3y agoYou’ve talked me into running some load tests around and before these times… around thanksgiving I’ll give it a shot too… I wonder though if it’s just a redirection of traffic… if regular business sites are less busy because people are shopping it would just slightly shift the load from one “side” to the other Hmmmm….
- vineyardmike 3y agoAmazon surely allocates their resources in advance of prime day, so they can preemptively change prices to account for demand or deny requests. That said, why would capacity issues be behind NDA? Anyone can grab their API and attempt to allocate a VM (or 100k)
- LazyMans 3y agoYou can query the spot pricing api and see what’s going on with that. I have a feeling Amazon purposely tries not to hang their customers out to dry by consuming large amounts of spot instances, or on-demand tanking spot availability.
- fragmede 3y agoJust a chilling effects from general paranoia over breaking NDA. What is and isn't actually covered by the NDA isn't something I had the time to look up for my comment for. You can't spin up 100k instances on a virgin account, but it's an interesting idea!
- spencerchubb 3y agoI am not saying that this is true AT ALL, but it would be kind of ironic if AWS slowed down competing ecommerce stores to try and get an advantage.
- rurp 3y ago> $102 million in infrastructure spend for an event that brought in over $12.7 billion in sales isn’t the worst return on investment that companies could make — by a landslide! Well it's not amazing if your margin's are tiny, as they are in many industries (such as retail). Plus this was almost certainly architected by some of the foremost AWS experts in the world. It's verrrry easy to spend vastly more than was strictly necessary in AWS. I don't mean to be too negative though, it was a really interesting article. Pretty wild to think about spending $100m on infrastructure over two days and still making a bunch of profit.
- madrox 3y agoImportant to remember that, before you could burst your infrastructure in the cloud, sites simply went offline in events like this. You took actively lost revenue in those cases.
- Uvix 3y agoDepending on the margins that could be preferable.
- ndriscoll 3y agoOr you could just design your architecture to not perform trillions of database requests for hundreds of millions of sales. The listing data is almost static and should almost fit in RAM (the hot set probably does. Apparently Amazon has ~350M listings. A 24TB RAM server could give ~68kB/listing, and probably only a small fraction is hot). Since you'll need multiple servers anyway, you could shard on products and definitely fit things in RAM. 375 million sales even if condensed into 1 hour would only be 104k/second. A single db server should be able to handle the cart/checkout. Assuming ~10M page views/second, a couple racks of servers should be able to handle it. The ad/tracking infrastructure surely can't account for the 1000x disparity in resource usage.
- turtlebits 3y agoI think you're forgetting that Amazon doesn't have a 100% conversion rate...
- jayzalowitz 3y agoCorey is probably right, but id chunk an extra 10-20% of overprovisioning/undercounting on actual bill here and considering they OWN the fleet, they probably went out of there way to have disaster recovery ready to go in a bunch more contexts.
- version_five 3y agoAmazon Prime Day event resulted in an incremental 163 petabytes of EBS storage capacity allocated – generating a peak of 15.35 trillion requests and 764 petabytes of data transfer per day. The main thing that strikes me is how (seemingly) inefficient everything is. What do they possibly need this amount of data for in selling stuff? Are they taking high-def video of every customer as they browse for something to buy? I get that it's a huge company and this is (I guess) their business time, but how can the y need so much storage. Ditto for much of the other stuff.
- CamperBob2 3y agoHot take: Amazon's search UX is so terrible that it not only wastes near-endless amounts of customer time and patience, but their own bandwidth as well.
- greatpostman 3y agoThey’ve a/b tested it to death
- fiddlerwoaroof 3y agoI wonder if Amazon has overfitted and/or a/b tester itself into a bad local optima. It’s pretty hard for me to believe that their current website really is as good as their data indicates.
- thayne 3y agoIME a/b tests are often run by people with little to no knowledge of statistics or experimental procedure. It is pretty easy to end up backing bad decisions with data when you don't completely understand the data.
- CamperBob2 3y ago"Hey, check this out! User engagement as a function of time spent on amazon.com is up 125% with the new build!" Once a metric becomes a target for optimization, it often loses its value as an indicator of a larger goal. People who obsess over A/B tests rarely understand that.
- Seanambers 3y agoIsn't the real clue here that the prices in the article are cost + margin. AMZN gets a steal.
- ovao 3y agoAnd notably, AWS can, and likely does, allocate whatever unused or unpartitioned infra to themselves (or, more pedantically, to Amazon). A perpetual ‘savings plan’.
- ckdarby 3y agoEven if AWS treats Amazon like any customer the article is off by a factor of 30-60%. RIs for their RDS instances. Saving Plan for their EC2s. 1 or 3 year commit, no upfront vs all upfront, etc. A customer at the size of Amazon using AWS would have private pricing arrangement and an EDP.
- simpsond 3y agoYou wouldn’t commit for 3 years for increased resources of a single day.
- jayzalowitz 3y agoHonestly, their EDP is probably effectively cost, set in stone to make sure that if the government breaks them up or something like that both systems are good.
- mrbonner 3y agoIt’s not a surprise for me to hear that Amazon is still a heavy user of RDBMS all these years even after the so-called Rolling Stone project to get rid of Oracle DB in 2015. If Amazon can use RDBMS for their scale, I’m just furious when folks jumping up and down screaming in top of their lungs “Why do we use Postgres and not (insert some random NoSQL engine here)?” My response so far is calmly ask another question “Why not?” And let them try to find a justification to suite our scale requirements.
- endisneigh 3y agoIt’s fascinating that this is your conclusion from the article. Mine would be that if you can make it work and believe these estimates then dynamodb is clearly more cost effective. And given that every project inevitably settles in access patterns and thus is a perfect fit for something like dynamodb, why bother with rdbms as the hot path? Just use dynamo and stream to a columnar database for analytics once your product is “finished”.
- bognition 3y agoIt all depends on your workload, access patterns, and data model. You can absolutely spend an arm and a leg making a system work using a RDBMS that would be simpler and cheaper using a NoSQL store. The opposite is also true. When picking a database you should always consider the trade offs of the different technologies and weigh those against your goals and budgets. Sometimes is okay to spend more for a system that is just simpler to manage and use. Sometimes it’s not.
- orochimaaru 3y agoYour application use cases should dictate the database choice - eg consistency needed, access patterns, data normalization, reliability, etc.
- RcouF1uZ4gsC 3y agoSometime back IIRC, some hackers were upset about something Amazon did and tried to DDOS them. When they realized their entire attack was just a fraction of what Amazon handled during the Holiday shopping season, they realized the futility and called it off.
- benjaminwootton 3y agoThe real cost would come in the months after whilst trying to decipher the bill adequately to track down everything you used and get it turned off. (Half a joke.) I imagine there would be a ton of Lambda and the like in there too.
- jeffbee 3y agoThe amount of mail alone is bonkers. If we assume that half of this traffic went to the big operators, Google and Microsoft, each of them would have observed a noticeable traffic bump, 10s of 1000s of requests per second on average all day. It is fun to think about how these systems are interconnected and how they affect each other.
- gumby 3y ago> There’s the internal chargeback costs that AWS charges Amazon for services that would be subject to Do they do this? I have asked some friends who are developers at AWS and both told me that they don't worry or even know what their usage costs. But that's just anecdote; perhaps their boss knows.
- donavanm 3y agoI cant comment on individual teams or the business and accounting practices of Amazon. I would ABSOLUTELY say that, at a minimum, every director or principal engineer needs to be familiar with costs and _should_ understand their P&L. Senior engineers and line managers probably/should have a passing familiarity or consideration. Individual random SDEs may not as its not their primary business function or deliverable and someone else is ultimately responsible. Disclosure: Principal at AWS, opinions are my own.
- deleted 3y ago[deleted]
- infinitedata 3y agoFunny how folks here and from the article are fixated in comparing the $102M vs the $12.7b. They somehow forget there are product, advertising, warehouses, transportation, shipping, labor, operation and other labor cost involved. You didn’t spend $102 to earn $12,700…
- deleted 3y ago[deleted]