21 ms·
Amazon search was down
- ohitsdom 10y agoWorking for me right now. Hard to imagine search being down for 4 hours...
- jdimov10 10y agoThey should consider moving to AWS ;)
- jawilson2 10y agoNah, they should just host it themselves. Bandwidth is much cheaper than AWS.
- 0xmohit 10y agoNot only bandwidth, but also disk. And don't forget the CPU throttling, noisy neighbors, IOPS and what not.
- imtringued 10y agoIs that a problem with AWS? I thought people accept paying a premium because they don't have to deal with these issues anymore. If bandwidth is relatively expensive less people are going to abuse it, etc...
- dijit 10y agoEC2 was designed around the idea of having highly parallel and intensive workloads for short periods of time. For people who use it as a VPS provider, they are going to be hard hit by other tenants who are applying the intended purpose.
- 0xmohit 10y agoYou pay a premium because you expect things to work well. When things don't work, AWS doesn't tell you a reason why it didn't work. Instead they teach you lessons about how to architect your applications for failure by saying: The EC2 team also recommend that you architect for failure using the white paper linked below: https://d0.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf See my earlier comment as well: https://news.ycombinator.com/item?id=11822298 https://news.ycombinator.com/item?id=11822298
- 0xmohit 10y agoIn that event, AWS would respond by saying: The EC2 has responded, and informed me that they do not provide analysis of individual issues in AWS infrastructure. They referred to the AWS SLA that states that "AWS will use commercially reasonable efforts to make Amazon EC2 and Amazon EBS each available with a Monthly Uptime Percentage (defined below) of at least 99.95%". The AWS SLA is available here: https://aws.amazon.com/ec2/sla/ (The above is a part of the response I received when I attempted to ask AWS about sporadic reboots and outages on one of my instances.)
- 0xmohit 10y agoAnother gem from AWS in response to a query that mentioned that every major DNS server around the world had picked up a change made to the A record of a domain except for Amazon DNS (even after 2 hours of the change): Thanks for that information. It really help us understand the issue faced. As you know changing a A record for a domain can take time to replicate through all the Root DNS servers and then onto the none authoritative servers from there on. When you are editing your DNS Zone file it is highly related to the TTL settings for quicker updates on changes to them. However it really isn't uncommon to see full DNS replication when making changes to an A record. .. Also for even more control and perhaps faster DNS resolving times within AWS look into our Route53 service. https://aws.amazon.com/route53/ I hope this has addressed your questions and please feel free to ask if there is anything else. Tools like http://mxtoolbox.com/dnspropagation.aspx http://mxtoolbox.com/dnspropagation.aspx and http://www.viewdns.info/propagation/ http://www.viewdns.info/propagation/ indicated that the change had been picked up by every listed server but for AWS. Solution proposed by AWS: Use Route53.
- maguowei 10y agohaha
- umanwizard 10y agoThey do use AWS.
- CameronBanga 10y agoThat's the joke.
- saosebastiao 10y agoIt's only partially true though. There are tons of teams that Amazon that do not use AWS. They use legacy internal data centers that do not have an ounce of AWS infrastructure. And for some reason it is nearly impossible for them to transition.
- jonhohle 10y agoIt may have been a while since you worked there. Very few teams are not using AWS (primarily EC2). For all of retail, marketplace, and digital, it was increasingly difficult to get legacy or bare metal hosts. The only teams that I interacted with still on legacy hosts were either operationally deficient or still running their own databases.
- saosebastiao 10y ago> The only teams that I interacted with still on legacy hosts were either operationally deficient or still running their own databases. There are a ton of these teams. AFT (several hundred devs), for example, is practically on their 10th year of a 3 year mandate to get off of their monolithic Oracle database backends, a constraint which forces them to run in legacy datacenters. You'll find similar issues with plenty of Supply Chain, Operations, and Transportation teams as well. The primary data warehouse clusters (non-redshift) and management interfaces are also in legacy data centers, last I recall.
- rdrey 10y agoA lot of AWS teams don't use AWS because of the circular dependencies that would create during a new region buildout.
- alecbaldwinlol 10y agoReminds me of Larry Ellison's old joke, "So when building our cloud, we decided to go with Oracle... we got a good price..."
- rwmj 10y agoUnlike any other customer ever.
- alecbaldwinlol 10y agoEven with 80% discounts for large "friends of the business", yes, I agree that better AND cheaper alternatives should be pursued ;)
- _nickwhite 10y agoIt just came back up it seems (9AM EST). I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.
- amarka 10y agoCan confirm that doing a search from the homepage search bar (before 9AM EST) threw a 500, seems to be back up.
- 3pt14159 10y agoSorry to be pedantic, but you mean 9AM EDT or just 9AM Eastern Time if you want to say "9AM in New York" which is what most people mean. There are still areas of Canada (Southhampton Island) and Central America (Panama IIRC) that are on EST year round.
- _quick_q 10y agoThey were responding to the comment where "9AM EST" was explicitly used. I don't understand the value of your comment as it doesn't serve to clarify - I think most if not all readers would understand the reference.
- GFK_of_xmaspast 10y agoThere's an hour difference between "9am est" and "9am edt".
- _quick_q 10y agoI understand, but that was not the point of my comment.
- 3pt14159 10y agoYeah, I saw that but I figured both people would see the response if I put it on the child comment since people often only look at comment replies and their children. Conceptually I was replying to both.
- Drakim 10y agoDid it take so long to get it up again because nobody noticed for the first 3 hours? (Actually what is Amazon search used for?)
- b34r 10y agoNot everyone arrives from Google.
- randlet 10y agoI use Amazon search every time I'm on their site (which is very frequently).
- ajoy39 10y agoto search for things that you want to purchase....
- mille562 10y agoIt's used to search their site for products. If you want to buy a "571B Banana Slicer" the easiest way to find it on amazon's site is to use their search. The search feature being broken translates to sales being close to zero for that time period.
- arbitrage 10y ago> "571B Banana Slicer" Wow, turns out that's actually a thing.
- SparkyMcUnicorn 10y agoLook at it for the novelty, and buy it for the reviews.
- mwambua 10y ago'For decades I have been trying to come up with an ideal way to slice a banana. "Use a knife!" they say. Well...my parole officer won't allow me to be around knives....' You made my day! :-)
- yomly 10y agoI can only imagine the hell some team must be experiencing right now. Godspeed, Amazon site reliability team...
- deleted 10y ago[deleted]
- aiabgold 10y agoWorking fine for me (~6:30am Pacific Time).
- lwhalen 10y agoWould love to see the outage writeup and RCA for this, if/when it's published.
- ikeboy 10y agoNot the first time this year https://archive.is/mvVUo https://archive.is/mvVUo http://www.usatoday.com/story/tech/news/2016/03/10/amazons-website-down-quickly-up/81596092/ http://www.usatoday.com/story/tech/news/2016/03/10/amazons-w... Although this was longer and more severe.
- ionwake 10y agoDo people get fired over things like this in the bigger companies?
- stevewilhelm 10y agoNot for a single incident. If production outages become a frequent occurrence, the VP of engineering or Ops may not be around much longer.
- hmate9 10y agoSomeone probably will get fired. Last quarter amazon earned $20.58bn from product sales. So 4 hours of that is $38,000,000.
- BinaryIdiot 10y agoI highly doubt search equates to all of their sales during a period. That would seem odd and would indicate no one uses their navigation at all.
- jessriedel 10y agoAlso, Amazon is established enough, and has few enough comparable competitors, that anyone whose search breaks is liable to just try again tomorrow. Still, some people who do go somewhere else, or are less likely to try Amazon in the future. I'd be surprised if this wasn't at least a multi-million dollar mistake.
- cwilkes 10y agoI never do. Going to their homepage is like seeing the dog's breakfast. It is a complete mess of whatever passed an a/b test of increasing sales. It is like they reached a local maxima but it is on St Crappy's Peak. If I go to Amazon.com I just go to the search bar and refine from there. So I guess I use their "navigation" in the form of their groupings of features on the left side.
- deleted 10y ago[deleted]
- granos 10y agoDoes something like this actually cost them a huge amount in sales or do people know enough about the internet today to just come back in a few hours and try again? I know that personally, if I'm buying from Amazon, I would just come back later and try again.
- Grue3 10y agoConsidering case studies like "300 million dollar button" [1], they are likely losing insane amounts of money. [1] https://articles.uie.com/three_hund_million_button/ https://articles.uie.com/three_hund_million_button/
- hullo 10y agoI think it comes down to how you define "a huge amount in sales". Let's optimistically say they lose 1% of sales over a 4 hour period although it could certainly be much higher. Is 1% of Amazon's sales a huge amount for them? Not in the short term. Is that absolute amount a huge amount for me, personally, or most of the other companies in the world? It sure is.
- tlrobinson 10y agoI'd guess they're probably losing more like 90% of sales while search is down. Basically every purchase I make on Amazon starts with a search.
- 10y ago
- misiti3780 10y agoIs it back up now because I just searched from Croatia and it worked fine ?
- thomnottom 10y agoI read that as "searched FOR Croatia" and thought, "Man, they really do sell everything these days..."
- iagooar 10y agoWould have been funnier if he searched for "Greece".
- BinaryIdiot 10y agoI noticed this while I was up very late working but it only was an issue for some items. For instance when I searched for Item1 and Item2 it failed but Item3 just kept working. Not sure how they have search divided up but found that interesting. Would love to see something come out about how it affected their users.
- umanwizard 10y agoWas one of the searches in a category, while the others were in all product search? That could account for the issue.
- BinaryIdiot 10y agoI tried them all from the homepage I just don't remember the exact products I searched for at the time.
- breakingcups 10y agoSearch caches maybe?
- BinaryIdiot 10y agoYeah maybe. They would have to be server-side as I was picking random things I have never searched before to see if search was broken for everything or not (I was initially getting different error messages for different products).
- umanwizard 10y agoYeah loads of stuff is cached server-side; this is a pretty realistic guess.
- deleted 10y ago[deleted]
- artursapek 10y agoDoes Amazon do public post-mortems for things like this, the way they do for AWS outages?
- sabujp 10y agoI expect to see a post mortem.
- el_duderino 10y agoDoes Amazon disclose such things as this when it affects a primary site functions?
- vamur 10y agoCould be due to HTTP/2 switch - e.g in the morning it was working in HTTPS and now it's HTTP only again. BTW, it's interesting how much bloat there is in their HTML code.
- TazeTSchnitzel 10y agoWas broken the other day for Amazon UK, too.
- dtdbdesign 10y agoI found an article about their last downtime and either way they are losing a lot of money https://www.upguard.com/blog/the-cost-of-downtime-at-the-worlds-biggest-online-retailer https://www.upguard.com/blog/the-cost-of-downtime-at-the-wor...
- rdoto454 10y agoNice!! ^^
- known 10y agoThat hurts AMZN credibility
- snambi 10y agoAmazon has search?
- khedoros 10y agoYou're surprised that they do?
- evansekeful 10y agoIt's kind of interesting how being the purchasing agent for an Amazon seller account gives you visibility on Amazon's health. After getting my 9am sales report, I panicked and had our tech team check if our seller software was working properly. Conversely, I knew Bezos was laughing all the way to the bank when people ragged on Prime Day last year before the official numbers came out. We didn't even have a featured deal and could see the insane difference.