11 ms·
How Amazon uses chaos engineering to handle 80k requests per second
- latchkey 3y agoI feel like Amazon search is one of the worst products I've ever used. It is a clusterf/ck of paid advertisements and obviously gamed results. I don't care how many requests/sec you get. If the results are horrible, what does it matter?
- wombat-man 3y ago'chaos' is a great way to describe the results, to be fair.
- latchkey 3y agotechnically they said 'chaos engineering'... which obviously means they use monkeys slapping their hands on keyboards to write the code that returns the results.
- beebmam 3y agoI'm sick of it being impossible to identify cheaply made products from high quality, durable products on Amazon. The rating system is flat out broken and there's an entire industry built around gaming those ratings. I'm at the point that I rarely ever buy products on Amazon anymore. It's a total disgrace. On an ethical level, I wish I had the ability to say "I only want to be presented with results that weren't made in China or other slave societies".
- digging 3y agoThese days it should be assumed that the quality of anything bought on Amazon is dogshit. Which, even though I canceled Prime years ago for ethical reasons, is honestly the biggest reason I won't even click an Amazon link.
- baz00 3y agoThe trick is to find your products somewhere else and then look them up on Amazon for a price comparison.
- amzthrow 3y agoI understand how you feel. It can be frustrating. Contrary to popular belief Amazon actually does put energy into making sure products are responsibly sourced. Products are de-listed if they’re found to come from unethical sources. To take that even further take a look at Climate Pledge Friendly. Those are products with (at least one) third party certification. These certifications don’t just further climate goals. Social responsibility is also considered. Including worker conditions and product durability. You can filter search results by this attribute. Admittedly it can be hard to filter for specific certifications. https://www.amazon.com/b?node=21221607011 https://www.amazon.com/b?node=21221607011
- NoPicklez 3y agoBuying from ethical sources is great, but that doesn't mean those products aren't crap either
- amzthrow 3y agoTake a look at the certifications. It’s a signal.
- nijave 3y agoAmazon can simultaneously do due diligence on a subset of products while completely ignoring the rampant fraud and knockoffs of many other products.
- beebmam 3y agoA giant portion of Amazon's products are made in China by a populace that's enslaved by their totalitarian dictator. Please allow me to identify which products are made in China, and other communist/totalitarian states, so that I can exclude them from my searches.
- 0cf8612b2e1e 3y agoIs that Amazon’s fault? So many once reputable brands have been MBAed to death and are now indistinguishable from the bottom tier garbage. It is near impossible to do real product research on anything, anywhere.
- nijave 3y agoSurely a company with the resources of Amazon can determine fake reviews from real ones. Especially those bait-and-switch where the product is changed and half the reviews aren't even for the current listing.
- xnx 3y agoIs there any alternative front end for Amazon? AI with some image-similarity smarts could do a much better job grouping/de-duping similar products.
- tkahnoski 3y agoThis would be awesome... When I'm in a rabbit hole trying to find a product and I see three or four of roughly the same design I know not to bother any further unless I can find a reliable manufacturer website.
- RockRobotRock 3y agoWorking as intended.
- waynesonfire 3y agosource: https://www.wsj.com/articles/amazon-changed-search-algorithm-in-ways-that-boost-its-own-products-11568645345 https://www.wsj.com/articles/amazon-changed-search-algorithm...
- yazaddaruvala 3y agoDisclaimer: Used to work at Amazon. Funny story, internally Amazon Search doesn't consider the ads products to be part of the "search results". It is tracked and accounted to Ads. The way Ads are handled on Amazon is really poorly done. The Ads teams claim to make a lot of money (and based on the internal accounting tricks they do), and as such have been pushing Amazon's leadership to go more into Ads, even tho every person I've met that worked at Amazon also hated the prevalence of Ads. Literally, Directors and VPs at Amazon are afraid to step on the toes of Ads' leadership team because of how well they have told the story about "Ads is excessively profitable". Meanwhile, all of us in the thread can easily say, even if it is short term profitable, it most certainly is not long term profitable for Amazon. Both from internally and externally it has been very disappointing to watch actually.
- acchow 3y agoSure the ads aren’t great. But that’s almost barely a problem compared to the gamed reviews
- SteveNuts 3y ago> But that’s almost barely a problem compared to the gamed reviews Which pales in comparison to the problem of counterfeit goods, IMO. I can at least somewhat comb through the reviews to look for outliers of well written reviews. Getting something that's obviously a fake (has happened to me multiple times) is completely unacceptable. Newegg has this issue too, I got a knock-off Intel CPU there once, I was furious.
- wincy 3y agoWhat the heck, what does a knock off Intel CPU even look like? Who is making them and more importantly, how?
- NovemberWhiskey 3y agoIt's more like: old being sold as new, engineering samples sold as production parts, relabelling lower spec parts as higher spec on the packages etc.
- deleted 3y ago[deleted]
- truetraveller 3y agoSurprised that 80K/second is called "massive" for Amazon.com's main search feature.
- Scalene2 3y agoDid the math, that's about 7 billion searches a day. That doesn't sound like a lot.
- WrongAssumption 3y agoI mean, google does 8.5 billion per day. What would be a lot to you?
- marginalia_nu 3y agoTo be fair, a lot of various URL bars and input fields turn other activities into implicit Google queries.
- umpalumpaaa 3y agoThere are 8 billion people on the planet.
- andrewmcwatters 3y agoIt dawned on me that in web software, people talk about req/s from two entirely different perspectives and it's borderline fraud: req/s from localhost to localhost, and req/s from the Internet to any user. The latter is actually interesting. People saying you can get 10k req/s from Node.js is stupid. You're not actually getting that on say, a single low-end instance over the Internet, which is what most developers are actually going to do. Instead, you'll get two orders of magnitude fewer requests per second. What Amazon is talking about here is most likely non-synthetic, real-world 80k requests per second. Which is actually a decent job.
- adamckay 3y ago> People saying you can get 10k req/s from Node.js is stupid. No, it's not, for exactly the reason you state: > You're not actually getting that on say, a single low-end instance over the Internet Some languages are, of course, more efficient, but it doesn't matter - you can get very good performance out of any language/runtime - it's all about your architecture and infrastructure.
- dr-detroit 3y ago[dead]
- andrewmcwatters 3y agoI would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. If it's the real-deal, and not like people saying "Bun.js can serve 65k req/s+ (cough cough to localhost,)" that's impressive. But I never see anyone talk about real-world numbers. Just synthetic poopoo. I think I read the article correctly, but I think it only talks about how they introduce "chaos engineering," I didn't recall them talking about how they actually handle a volume of traffic like 80k req/s.
- baby_souffle 3y ago> I would love to know what software stack, hardware, and uplink connections in total they utilize to accomplish a real-world 80k request per second throughput. How many instances do you guys think Amazon runs for its primary e-commerce front-end stack? In total, and per region? Assuming they have a multi-region rollout. > But I never see anyone talk about real-world numbers. Just synthetic poopoo. The number probably changes all the time based on load. They'll never release these details because it's a competitive advantage to have the "how popular are they in $place at $time-of-day" data private. When they do share numbers, it'll always be the most flattering and devoid of any context beyond the "wow" factor.
- andrewmcwatters 3y agoI'm not sure it's the biggest deal in the world, plus real-world market presence data is regularly detailed in shareholder reports for various companies. You could take low-end instance specifications and standard industry stacks and extrapolate forward how many instances they might need to maintain at a maximum, but those numbers are going to be off. Are they running 400 low-end front-end instances across the globe? Probably, (plus the 40 or so other services they claim to need, multiplied by region count at a minimum) and that would actually be well below realistic and reasonable for a company like Amazon. You can take a bunch of regional instances that handle roughly 200 req/s and make that work.
- 3y ago
- aeturnum 3y agoI'm pretty unclear on the "how" here - but from what I can understand in the article the search resilience team injected properly tagged synthetic traffic into their system to do testing? That does seem like the kind of practice that could be part of healthy holistic approach - but the article elides a ton of details. I suppose the idea is that it promotes AWS services (with the idea of suggesting that this kind of resiliency comes easier on their platform) - but this is a great example of how good writing strips things down to the barest details. I would love to take lessons from it but I think the details actually aren't here.
- _a_a_a_ 3y ago> I suppose the idea is that it promotes AWS services an advert in disguise then?
- catchnear4321 3y agoanything that uses the template “How X did Y to improve Z by METRIC” and also hosted on company x domain is an ad. for that matter, any blog post by a company is an ad. maybe not to sell product, but to at least build exposure and familiarity with the brand.
- vrosas 3y agoTo add to that, I’ve read enough of my own companies blog posts to know that Y was an incorrect solution to Z, was half-implemented at best, barely improved METRIC, and has already made a lot of peoples’ lives harder for simply existing. So now I take every single one of these posts with a massive grain of salt.
- 1vuio0pswjnm7 3y agoRecruiting ad
- ljm 3y agoThe solution to any problem in a cloud provider is another service the same provider offers, as far as they’re concerned.
- hexo 3y agoonly 80k?
- bdamm 3y agoCan anyone calculate the dollars-per-request revenue and profit? Would be interesting to see how much it costs Amazon to make money and make some connection to the request rate.
- NBJack 3y agoBack in The Day, rumor had it the detail page hosting/rendering would easily max out a single machine after only a handful of queries per second. I regrettably can't verify the truth of that, or if it still holds, but there is a LOT going on with a given /dp . "How many of those requests translate to actual purchases?" is the next question.
- timeagain 3y agoIIRC there was some memory leak issue in the custom rendering engine/language that was being used and for a time the solution was to reboot any VM that has taken more than $x requests. $x was a very small number
- slipshady 3y agoMy professor at Cornell whose lab Werner Vogel (Amazon's current CTO) worked loved telling this story. Maybe there's some veracity to it.
- dastbe 3y agoi do recall the simplestack version of dpx had some obscene issues with garbage generation, followed by some obscene issues with throughput due to aggressive locking (to avoid the obscene amount of garbage generated).
- NBJack 3y agoI am not surprised. That platform was a neat concept, but wow, the nesting and resource consumption was atrocious. There was potential there, yet the implementation in Java made IMHO some fundamentaly flawed assertions about small objects. I unfortunately know of a great director who had to find new opportunities outside of the company over it.
- bananapub 3y agowhy would they need to do anything - including blogging - to handle 80kqps, that's ~one machine nowadays
- boredtofears 3y agoDid someone mean to post this about ten years and accidentally set the publish date too far out?
- deleted 3y ago[deleted]
- farhanhubble 3y agoThis is just plain old testing, no?
- deleted 3y ago[deleted]
- tree666 3y agoYour article shows you put in a lot of work, and was written well. But... A few thoughts... Service Owners usually tend to design and build their systems in a way which allows them to understand and know how the system will behave under the worst conditions. It is not difficult to test these conditions either. A bash script utilizing curl will suffice. If you have to use 'Chaos Engineering' to experiment your way into innovation, this is a sign you built your service wrong. What will Amazon Re-Invent next!? I am guess the wheel. Well written article though.
- NBJack 3y agoI might be misunderstanding what you consider one, but in my experience a bash script driving curl is great for some types of API or front-end load testing. However, it won't necessarily help you know how your system will behave if S3 kicks the bucket in us-east-1 (again), your image host for that super-cool Kubernetes cluster suddenly throttles you during a critical restart, or your other service of choice went down due to an expired certificate. If you however mean to use it to perform a denial of service on an endpoint you don't own, you're more hard-core than I thought.
- deleted 3y ago[deleted]
- tracer4201 3y ago[dead]
- yathaid 3y agoIs your bash script open or closed? [1] How does a bash script simulate network failures, connectivity drops or a gray-failure in one of your dependencies? [2] A lot of thought has been put into this domain. Dismissing that without understanding any of the complexities is just showing your ignorance. [1] - https://brooker.co.za/blog/2023/05/10/open-closed.html https://brooker.co.za/blog/2023/05/10/open-closed.html [2] - https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference https://docs.aws.amazon.com/fis/latest/userguide/fis-actions...
- hughesjj 3y ago
- DevKoala 3y agoSome background on the challenges around e-commerce search/product discovery/matching. https://aws.amazon.com/blogs/industries/building-blocks-for-modern-retail-ecommerce-and-media-search-with-aws/ https://aws.amazon.com/blogs/industries/building-blocks-for-... 80k RPS is not just for a single database lookup, don’t be silly.
- pelorat 3y agoM80K requests per seconds were doable in the 90's and you didn't need a super computer for it.
- deleted 3y ago[deleted]
- AtlasBarfed 3y agoAmazon's ultimate chaos engineering is a working culture that turns over its entire staff every two years and stays afloat.
- scrum-treats 3y agoAgreed. The only thing chaotic is the turnover and the level of managerial incompetence.
- balls187 3y agoSeconded by Search on Amazon.com returns almost entirely sponsored results.
- offmycloud 3y agoAmazon product search results are the chaos.
- moribvndvs 3y ago1000 sponsored links that are exactly not what you searched and filtered for.
- baja_blast 3y agoAnd a hundred different identical products at vastly different prices from brand new sellers like `z-qq-yadonk-8771` that some how has 4.7k reviews at 4.5 stars.
- efitz 3y agoI feel like my eyes were just assaulted trying to read that white-on-gold mobile page. I was unable to read the article due to the accessibility-hostile theme that disables reader mode on mobile Safari.
- bluedevilzn 3y agoWhen I was at Amazon, I thought this was an insane scale. Then, I joined Google to learn how they serve couple orders of magnitudes higher.
- slipshady 3y agoDynamoDB served upwards of 100MM tps during Prime Day '23 - https://aws.amazon.com/blogs/aws/prime-day-2023-powered-by-aws-all-the-numbers/ https://aws.amazon.com/blogs/aws/prime-day-2023-powered-by-a...
- hughesjj 3y agothis is low acale for any given public api at amazon. You should see how much kinesis/ddb/iam/sqs/cloudwatch handles (in total per region)
- akdor1154 3y agoWait, your Search feature uses 40 microservices?
- yathaid 3y agoOkay, describe your architecture and let's see how many holes it has.
- Weryj 3y agoCan confirm, working there was Chaos, not sure that's what they meant by chaos engineering though.
- hughesjj 3y agoIt's where they delete entire teams and remove funding for critical infrastructure and see how well the system self-repairs It's just in this case the system is the company itself instead of just a suite of software services