3 ms·

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. H

I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. But I never see anyone talk about real-world numbers. Just synthetic poopoo. I think I read the article correctly, but I think it only talks about how they introduce "chaos engineering," I didn't recall them talking about how they actually handle a volume of traffic like 80k req/s.

baby_souffle 3y ago
> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probably changes all the time based on load. They'll never release these details because it's a competitive advantage to have the "how popular are they in $place at $time-of-day" data private. When they do share numbers, it'll always be the most flattering and devoid of any context beyond the "wow" factor.
- andrewmcwatters 3y ago
  I'm not sure it's the biggest deal in the world, plus real-world market presence data is regularly detailed in shareholder reports for various companies. You could take low-end instance specifications and standard industry stacks and extrapolate forward how many instances they might need to maintain at a maximum, but those numbers are going to be off. Are they running 400 low-end front-end instances across the globe? Probably, (plus the 40 or so other services they claim to need, multiplied by region count at a minimum) and that would actually be well below realistic and reasonable for a company like Amazon. You can take a bunch of regional instances that handle roughly 200 req/s and make that work.
- tacozilla 3y ago
  While I can understand the cynicism, the real answer is a lot closer to something much more boring, which is most people just don't care about the actual numbers, and if they were to release them, while interesting to a small few, generally no one would actually care. There's also a common misnomer that Amazon.com is somehow just this one giant app running on a set of servers, which isn't remotely how it's actually deployed, and that's before we spend time arguing whether a team's instances even count as "primary e-commerce front-end stack" or not. :P
  - hughesjj 3y ago
    Also their "stack" includes various degrees of AWS (maws, naws, and I'm sure a bunch of snowflake situations innumerable here, I mean do you count corpinfra? Controls?) Not to mention all the data center logistics
tacozilla 3y ago
Amazon doesn't really have a "primary e-commerce front-end stack" in any concrete sense. They have hundreds/thousands of teams that deploy bits and pieces to a massive pipeline that ultimately makes up what you see on Amazon.com, but each team can have their own infrastructure backing things. Some teams might run everything off a dozen low-end EC2 instances while another sibling team has 3k+ instances; it's really all over the place, and that's ignoring specific events like Black Friday or Prime Day, etc. where teams need to prescale things in advance. Source: Worked at Amazon/AWS for almost a decade.
- andrewmcwatters 3y ago
  Thanks for the neat insight!
superfrank 3y ago
I used to work on a system that did about 55k/sec at peak. The service was internal it was only handling grpc calls which were coming from inside our VPC and it was written in Go. It's main job was was reading and writing to an SQL db that was sharded across 3 or 4 of the biggest instances AWS offered at the time (2017ish). Everything was Dockerized and I think we were using Docker Swarm for container orchestration. I don't remember the specs for each box, but we had auto scaling set up so at peak we'd hit a little over 200 containers. Looking back now, I'm sure we could have gotten much better performance out of that service, but the team was young and inexperienced and throwing money at the problem was an easier solution.
- andrewmcwatters 3y ago
  Sounds about in the ballpark I would expect, though!
yazaddaruvala 3y ago
> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of this as ElasticSearch clusters - but its not ES), with different product groupings in each cluster. Each cluster is made up of 1000s of nodes running Lucene (think similar to ES shards). This is just for the "match set", i.e. which products to return. Then there are services to re-sort those matched products based on popularity, likelihood of purchase, etc. Think giant ML models. Then there are product lookups. Before all of this, there is Query analysis to simplify/improve the query (think giant ML models) to classify "Apple" into electronics vs groceries based on the other keywords and your current context. Meanwhile, Bun.js is talking about 65k "hello world" type req/s. The compute per req is magnitudes different.
- Tomis02 3y ago
  > Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.
  - dmoy 3y ago
    > That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services. Well, Jeff Bezos circa 2002 or so did: https://news.ycombinator.com/item?id=3102800 https://news.ycombinator.com/item?id=3102800 It's hard to remove that ethos now
    - hughesjj 3y ago
      And thank God he did, amazon would have direct without the SOA enforcement. Scale issues are emergent issues. It's a phase transition.
Karrot_Kream 3y ago
Having led API teams at a big tech where we handled similar (slightly lower) request numbers at the edge, I can tell you that the entire stack was engineered to be defensive and handle a certain amount of load. As other commenters have said, fan-out means that 80k qps at the edge means you're getting probably 10-100x that on some of the most heavily hit backend systems. A lot of the work we did was very aggressive caching and sharding. Autoscaling to handle load spikes. Observability was our secret sauce. We would monitor everything. Our caches, NICs, our load balancers, etc. Cache hotspotting and DB problems were the problems that kept us up at night, though my teams didn't deal with much stateful data.
deleted 3y ago
[deleted]
ilyt 3y ago
Well, to get actual impact you'd need infrastructure-wide tracing and that's hard. Like, you could hit a cache and serve 95% of page from it. Or hit some long path that will burn half a second on 20 servers in the backend to serve some big query But yeah title is useless
- hughesjj 3y ago
  I mean, coralmetrics+pmet has been doing that infrastructure wide tracing for decades (albeit being slowly replaced in spots now). Back in 2014 they were still sharing a few years old (at that time) detail-level service call graph that had so many nodes and lines it looked like string art.

baby_souffle 3y ago

> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probably changes all the time based on load. They'll never release these details because it's a competitive advantage to have the "how popular are they in $place at $time-of-day" data private. When they do share numbers, it'll always be the most flattering and devoid of any context beyond the "wow" factor.

andrewmcwatters 3y ago
I'm not sure it's the biggest deal in the world, plus real-world market presence data is regularly detailed in shareholder reports for various companies. You could take low-end instance specifications and standard industry stacks and extrapolate forward how many instances they might need to maintain at a maximum, but those numbers are going to be off. Are they running 400 low-end front-end instances across the globe? Probably, (plus the 40 or so other services they claim to need, multiplied by region count at a minimum) and that would actually be well below realistic and reasonable for a company like Amazon. You can take a bunch of regional instances that handle roughly 200 req/s and make that work.
tacozilla 3y ago
While I can understand the cynicism, the real answer is a lot closer to something much more boring, which is most people just don't care about the actual numbers, and if they were to release them, while interesting to a small few, generally no one would actually care. There's also a common misnomer that Amazon.com is somehow just this one giant app running on a set of servers, which isn't remotely how it's actually deployed, and that's before we spend time arguing whether a team's instances even count as "primary e-commerce front-end stack" or not. :P
- hughesjj 3y ago
  Also their "stack" includes various degrees of AWS (maws, naws, and I'm sure a bunch of snowflake situations innumerable here, I mean do you count corpinfra? Controls?) Not to mention all the data center logistics

tacozilla 3y ago

Amazon doesn't really have a "primary e-commerce front-end stack" in any concrete sense. They have hundreds/thousands of teams that deploy bits and pieces to a massive pipeline that ultimately makes up what you see on Amazon.com, but each team can have their own infrastructure backing things. Some teams might run everything off a dozen low-end EC2 instances while another sibling team has 3k+ instances; it's really all over the place, and that's ignoring specific events like Black Friday or Prime Day, etc. where teams need to prescale things in advance. Source: Worked at Amazon/AWS for almost a decade.

andrewmcwatters 3y ago
Thanks for the neat insight!

superfrank 3y ago

I used to work on a system that did about 55k/sec at peak. The service was internal it was only handling grpc calls which were coming from inside our VPC and it was written in Go. It's main job was was reading and writing to an SQL db that was sharded across 3 or 4 of the biggest instances AWS offered at the time (2017ish). Everything was Dockerized and I think we were using Docker Swarm for container orchestration. I don't remember the specs for each box, but we had auto scaling set up so at peak we'd hit a little over 200 containers. Looking back now, I'm sure we could have gotten much better performance out of that service, but the team was young and inexperienced and throwing money at the problem was an easier solution.

andrewmcwatters 3y ago
Sounds about in the ballpark I would expect, though!

yazaddaruvala 3y ago

> If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. Not all req/s are made the same. Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. FWIW a search request at Amazon hits roughly 100 unique search clusters (think of this as ElasticSearch clusters - but its not ES), with different product groupings in each cluster. Each cluster is made up of 1000s of nodes running Lucene (think similar to ES shards). This is just for the "match set", i.e. which products to return. Then there are services to re-sort those matched products based on popularity, likelihood of purchase, etc. Think giant ML models. Then there are product lookups. Before all of this, there is Query analysis to simplify/improve the query (think giant ML models) to classify "Apple" into electronics vs groceries based on the other keywords and your current context. Meanwhile, Bun.js is talking about 65k "hello world" type req/s. The compute per req is magnitudes different.

Tomis02 3y ago
> Not all req/s are made the same. > Amazon search is made of 100s of services, and Amazon's search page loads 20 products per page, that means 80k search req/s translates to 1.6 MM product API req/s for example. That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services.
- dmoy 3y ago
  > That's their problem, no? Nobody's forcing them to have an architecture where a request propagates to hundreds of services. Well, Jeff Bezos circa 2002 or so did: https://news.ycombinator.com/item?id=3102800 https://news.ycombinator.com/item?id=3102800 It's hard to remove that ethos now
  - hughesjj 3y ago
    And thank God he did, amazon would have direct without the SOA enforcement. Scale issues are emergent issues. It's a phase transition.

Karrot_Kream 3y ago

Having led API teams at a big tech where we handled similar (slightly lower) request numbers at the edge, I can tell you that the entire stack was engineered to be defensive and handle a certain amount of load. As other commenters have said, fan-out means that 80k qps at the edge means you're getting probably 10-100x that on some of the most heavily hit backend systems. A lot of the work we did was very aggressive caching and sharding. Autoscaling to handle load spikes. Observability was our secret sauce. We would monitor everything. Our caches, NICs, our load balancers, etc. Cache hotspotting and DB problems were the problems that kept us up at night, though my teams didn't deal with much stateful data.

deleted 3y ago

[deleted]

ilyt 3y ago

Well, to get actual impact you'd need infrastructure-wide tracing and that's hard. Like, you could hit a cache and serve 95% of page from it. Or hit some long path that will burn half a second on 20 servers in the backend to serve some big query But yeah title is useless

hughesjj 3y ago
I mean, coralmetrics+pmet has been doing that infrastructure wide tracing for decades (albeit being slowly replaced in spots now). Back in 2014 they were still sharing a few years old (at that time) detail-level service call graph that had so many nodes and lines it looked like string art.