6 ms·
Amazon.com product pages are down
- ghshephard 5y ago20+ years of using them - I don't think I've seen a universal outage on all of their product pages.
- rapfaria 5y agoCoincidentally happens on the new CEO's first week.
- tehlike 5y agoA new effort to increase stability will be started by the VP of Engineering in light of this issue.
- jaredsohn 5y agoObviously it is a coincidence but I have a funny vision in my head of Amazon systems relying on Bezos running a script or similar for the past 27 years and forgetting to have someone else take it over for him. Could probably be an Onion article.
- paranoidrobot 5y agoOther possible funny (but probably happened somewhere else) scenarios: It doesn't rely on him running it, it just runs under his UID. A widely distributed authentication/security process does an internal healthcheck by looking up 'jbezos' on the datastore. A few hours ago someone disabled the jbezos user.
- nodesocket 5y agoHonestly this explanation is not that far fetched at all, in fact plausible.
- grogenaut 5y agonah a long time ago they moved everything to run as 'nobody' and that was before the move off of those systems and onto things like ECS. fun fact jeff's employee id is like 72 or something while still being the lowest as they didn't reset the oracle sequence once they got that system working after testing but before making his user.
- paranoidrobot 5y agoI know it's unlikely, but I've grown (even more) sceptical over the years. I've stopped being surprised when I encounter some configuration file somewhere with a comment: "TODO: Fix this later" and it turns out more than half the business relies on that thing being configured that way. Even at companies who do have the money and technical resources to do things right, things get forgotten, or pushed back on. So, while these are problems that are unlikely, I'm sure I'm not the only one wondering... "Did they have something dumb like this, too?"
- paxys 5y agoWould be a fun story but Bezos is still the executive chairman of Amazon, so his user account will be left very much intact.
- paranoidrobot 5y agoOk, so the account still exists, but it's now in a different OU and so the healthcheck that looks for it in the Office of the CEO ou fails... Again, a joke, but I've seen stuff fail in other dumb ways because of some assumptions that were made about things that never change (until the day they do).
- dylan604 5y agoI envision the bunker from Lost where every 30 mins a code must be typed in to keep things running. Now you know why Bezos dind't want to do it any longer.
- xyst 5y agoLoading up on weekly AMZN puts this week sounds like a pretty good idea
- BoorishBears 5y agoIf you want to lose money, sure.
- fridif 5y agoAh yes, the old "literally can't go wrong trade", FAANG calls
- nodesocket 5y agoNot even a grain of sand for AMZN, foolish if you serious.
- kortilla 5y agoThis event already happened. Any puts you buy tomorrow will have taken this outage into account.
- deleted 5y ago[deleted]
- Flanagain 5y agoI remember the first or second Amazon Prime day the entire site was down for hours and hours (and showing puppies as an error message). Probably the closest I've seen.
- nodesocket 5y agoIt was 2018. See this CNBC article on how they scrambled to try and fix it. https://www.google.com/amp/s/www.cnbc.com/amp/2018/07/19/amazon-internal-documents-what-caused-prime-day-crash-company-scramble.html https://www.google.com/amp/s/www.cnbc.com/amp/2018/07/19/ama...
- ImpressiveWebs 5y agoI was just coming to Hacker News to see if this was just me. Every browser I tried and every Amazon I tried (.com, .ca, co.uk) had the same result.
- ping_pong 5y agoWow, my kid was complaining that he couldn't access a page a couple of hours ago and I dismissed it as him making a mistake somehow. I guess this is must be a pretty huge outage for it to be across all the products!
- varenc 5y agoIt’s interesting seeing how their mobile app handles the problem. Instead of showing an error page, you're just mysteriously returned to the Amazon homepage when trying to view a product. For a huge outage like this, the app’s behavior is a confusing experience, with users likely blaming themselves. The desktop web’s error message at least makes it pretty clear it’s an Amazon issue.
- arthurcolle 5y agoIs that interesting? Seems pretty standard and could even be handled relatively trivially in response to specific HTTP codes of any given individual request... Pretty sure nginx /whatever reverse proxy/load balancer they are using could do this natively without any special logic
- BoorishBears 5y agoIt's interesting because of how awful the UX is there for a multi-trillion dollar company. This would be acceptable if it showed an error notification after redirecting you, but it doesn't. It just very cheerfully dumps you to their homepage. It's bad enough the mobile app is just the website bundled up in a less functional form, the least they could do is add small things like this to it.
- atatatat 5y agoYou'll find what you're trying to pay them for eventually if they put you back on their starting page — and they know it.
- BoorishBears 5y agoHow does adding a simple toast-style message saying "We had some trouble finding that" alongside the homepage stop that?
- hattmall 5y agoYou should try hitting one of the potentially non-profitable customer filters. Like buy 5 things with a gift card that all ship to a different address on the same day. No message, no warning, no email, just your password no longer works and then an endless loop of resetting and verification emails.
- deleted 5y ago[deleted]
- magoghm 5y agoIt isn't only Amazon.com, they are also down for amazon.com.mx and amazon.co.jp (the other two Amazon stores I use frequently).
- anirudhgarg 5y agoEven amazon.in (India)
- spollo 5y ago.ca as well.
- tkgally 5y agoamazon.co.jp seems to be working fine for me. I just accessed it through the iPad app and was able to view product pages and my purchase history with no problem. I’m located in Japan, in case that makes a difference.
- magoghm 5y agoamazon.co.jp is now working again for me also.
- Izkata 5y agoI was just able to load some random pages on amazon.co.uk
- magoghm 5y agoIndeed, amazon.co.uk product pages seem to be working.
- ghshephard 5y agoAccording to:https://www.digitalcommerce360.com/article/amazon-sales/ https://www.digitalcommerce360.com/article/amazon-sales/ "The online retailer’s net income shot up 219.8% to $8.107 billion, exceeding the company’s previous record of $7.222 billion set in the fourth quarter of 2020" 8 Billion * 4 / 365 /24 = $3,652,968/hour.
- Sebb767 5y agoTo be honest, I expected it to be larger. 4 million is an insanely big pile of money, but it's nothing that will show up in Amazons budget, even if we're talking about an hour or more of downtime.
- djrogers 5y agoThat’s net income - gross income is likely an order of magnitude higher.
- dreyfan 5y agoAmazon net sales were $386B in 2020.
- ghshephard 5y agoThat's total gross revenue for all of Amazon. From: https://www.statista.com/statistics/266288/annual-et-income-of-amazoncom/#:~:text=In%202020%2C%20online%20retail%20platform,than%20386%20billion%20U.S.%20dollars https://www.statista.com/statistics/266288/annual-et-income-.... "In 2020, online retail platform Amazon reported a net income of 21.33 billion U.S. dollars, up from a 11.6 billion U.S. dollar net income in the previous year. During the same fiscal period, the company's revenue amounted to more than 386 billion U.S. dollars."
- dreyfan 5y agoNo shit. Nobody cares about net income. They care about how much consumer spend Amazon is losing during the outage. The total number above does indeed include AWS, which should be subtracted since it's not offline, leaving us with ~$340B in 2020.
- axaxs 5y agoMy wife asked me to buy some things, and after getting frustrated by Amz doing this I just bought from Walmart online. I wonder how many others did something similar, or how many millions of dollars such an outage cost them.
- deleted 5y ago[deleted]
- fmajid 5y agoAlso many people just default to looking on Amazon because of Prime and force of (conditioned) habit. In fact, that searches start on Amazon (despite its deliberately crippled search functionality) rather than Google is probably the single biggest heartache for Google as they lose that sweet, sweet time-of-purchase-intent opportunity to influence the purchase decision for advertisers. If you break that conditioning and shoppers realize prices are actually cheaper on Target, Wal-mart or whatever, they will start routinely looking elsewhere. Now the outage didn’t last long enough to impact a huge proportion of their user base, but if it had it could durably drag on their growth.
- axaxs 5y agoI do this now. Mainly because Google shopping results ALWAYS have scam sites in them. My workflow was Google search, click shopping, see one store you've never heard of coming in $50 cheaper...Google store to see no real helpful info, or if you're lucky, one or two links to someone asking if it's legit and being told no. I wish they had better vetting.
- Sebb767 5y agoThis seems like a major global outage. Good luck to those poor sysadmins over at Amazon, this must easily be in the millions lost per minute. Let's see whether this outage takes out HN with it once again ;-)
- leddt 5y agoMore likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.
- xwolfi 5y agoI d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on
- ROARosen 5y agoIt's back
- lamontcg 5y agoI think peak Christmas time shopping was a couple million per hour back in 2006. I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).
- absolutemoose 5y agoWhoever caused this has got to be really stressed out right now.
- cghendrix 5y agoI would legitimately probably cry.
- absolutemoose 5y agoI mean and it's Amazon of all places, the least forgiving of the tech companies.
- rantwasp 5y agothe question is: would you be at your desk?
- MattGaiser 5y agoAmazon seems like the kind of culture that would fire them anyway.
- adt2bt 5y agoHonestly, given the headlines of the past few weeks this is a fair joke, but in my experience they do not fire people for outages like this. Anything that can take down retail or AWS at large is considered a design problem, not a human error problem that deserves termination.
- pookeh 5y agoDesign problems become human problems at performance review time
- MattGaiser 5y ago1. Even before the past few weeks of articles, anecdotes from people I know have always indicated that Amazon is a pretty brutal company so I am surprised by that. 2. Is it just a matter of delayed impact on you? Would it come up in reviews later?
- dannyw 5y agoI started experiencing gradual, random errors on .com a while ago. Now it's down globally? .au app works.
- freedomfriez 5y agoSomeone working in Retail is having a no good, very bad Sunday.
- anirudhgarg 5y agoHavent heard from Amazon itself ? I dont seem them respond - not even on their offical twitter channel - https://twitter.com/amazon https://twitter.com/amazon
- MattGaiser 5y agoDid they fire too many engineers?
- elif 5y agoPerhaps they got tired of the stinky waterbottle smell...
- MattGaiser 5y agoThey will eventually have a standard issue catheter.
- Tabular-Iceberg 5y agoI’m sure many consider not having to get up from their desks and go to the loo a perk.
- nanoscopic 5y agoHaving worked on the systems that make Amazon.com function previously, all I can say is: Awesome! Burn baby burn.
- cellis 5y ago"Buy it again" seems to still work...
- henning 5y agoClearly if they had written more unit tests, this wouldn't have happened.
- johnwheeler 5y agoIf only they had Uncle Bob Martin on their payroll.
- kburman 5y agoCan someone explain how does WW outage happen? My assumption is anysite at Amazon scale is atleast designed to be regional partitioned.
- xwolfi 5y agoThere must be a silly single source of failure, like a lambda giving the time of day or a marketing tracker pixel. Yourkit, a java profiler, stopped working after a certain date in 2018 and had to be patched on thousands of users in my company alone. Could be some such time based constraint on a dependency loaded by all components for instance. You can regionally partition but if the poison pill was inserted years ago and rolled out everywhere with a time trigger...
- paxys 5y agoThere are always bottlenecks and single points of failure no matter how resilient the design in. And even if there are multiple regions, the same bit of offending code could have been deployed to all of them.
- KeepFlying 5y agoIt could be that all global systems happen to rely on some kind of single point of failure that the team never really realized before. Or if they did realize it they didn't realize the potential impact. Or it was a code change that was able to be validated and rolled out to all regions but then some totally independent interaction caused it to fail afterward in a unique way. I'm banking on surprising singe point of failure though. I really hope we get a detailed post mortem here.
- dehrmann 5y ago> Or it was a code change Probably not; I don't know specifically for Amazon, but most places avoid weekend code changes.
- alisonkisk 5y ago> some totally independent interaction caused it to fail afterward
- cronix 5y agoGraph on downdetector.com shows AWS (likely) experiencing the same issue, which I guess is no surprise. So, a lot more than just amazon.com affected by this.
- AnssiH 5y agoI think that is just downdetector users incorrectly reporting Amazon retail website failures against "Amazon Web Services". They should probably name it "AWS" to avoid that.
- pcbro141 5y agofrom a friend at AMZN: still no root cause identified as of 8:55 PM PST
- peculiar_throw 5y agoI’m sure someone’s already asking “Is the COE done yet?”
- 1970-01-01 5y agoWeird how only the product pages are down. Items on saved lists can still be bought. Its global. Someone out there is updating their resume...
- kgdinesh 5y agoits back up for amazon.in
- owlbynight 5y agoIt works for me if I reload the page of any product after I land on it. Seems to only happen on the first click.
- anirudhgarg 5y agoSeems to back up now
- deleted 5y ago[deleted]
- abeppu 5y agoNormally the fact that amazon pages have _so many things_ on them means that there are a lot of backend services that could fall over and you'd be mostly ok (e.g. not showing "frequently bought together" items or whatever). But of course, some things are really essential.
- rottc0dd 5y agoWow. It started 2 mins before and again stopped working. Link that worked. https://www.amazon.com/Perdido-Street-Station-Crobuzon-Book-ebook/dp/B000FBFO8C https://www.amazon.com/Perdido-Street-Station-Crobuzon-Book-...
- rottc0dd 5y agoNow, it is working... again.
- natas 5y agoCome back Jeff!
- jeffrallen 5y agoAnd the earth breathed a sigh of relief.
- ghshephard 5y ago... And its back!
- xyst 5y agoLooks like it’s fixed now.
- rz2k 5y agoItems can be put into your cart as long as you don't want to see the product page and they were already in some list like saved items. From there you can add additional items to your cart from "Compare with similar items". With those items are in your cart, repeat. Turning a saved book you were thinking about buying into socks with the least number of intermediate related items surely constitutes a new kind of Amazon Golf.
- fridif 5y agoPlease, stop, before they make this an OA question for next year.@
- beefjerkins 5y agoHopefully we'll see some sort of writeup regarding what caused the global amazon network to start chucking errors. I'm curious as to how this wasn't isolated to one individual region
- koreanguy 5y agoanother fastly problems ..
- busymom0 5y agoAccessing the .COM page from Canada seems to work fine for me.
- herpderperator 5y agoWhat was the error? It's working now and nobody in the comments so far seems to have mentioned what happened when they visited the product page.
- herpderperator 5y agoThere was another HN post[0] on the front page which showed the error: https://twitter.com/DRLynam/status/1414420344144728064/photo/1 https://twitter.com/DRLynam/status/1414420344144728064/photo... [0] https://news.ycombinator.com/item?id=27806471 https://news.ycombinator.com/item?id=27806471
- imvetri 5y agoHow does amazon.com build the product pages ?
- gonzo41 5y agoJeff leaves and sh-t hits the fan!. Good luck to em.
- deleted 5y ago[deleted]