8 ms·
Amazon was Down
- seanp2k2 14y agoCan't get into AWS management console either.
- ritchiea 14y agoI can't get to the management console or status.aws.amazon.com
- itomatik 14y agoaws console seems fine to me. the amazon.com is down for sure.
- andrewryno 14y agoYeah same here. Amazon is throwing a "Http/1.1 Service Unavailable" but AWS console is fine.
- magnacartic 14y agoJust successfully launched an instance, but I can't get my Prime video!
- davorak 14y agoDoing fine here, both amazon.com and the AWS console.
- rattray 14y agohere=where?
- michaelburk 14y agoI got a 500 error on amazon.com. Now just a timeout.
- rdl 14y agoWow. Something this big means I'll bet it's a networking issue. I wonder if they lose money for a brief outage, or if people just delay their purchases. I seem to remember them graphing this somewhere.
- level09 14y agohttp://i.imgur.com/E8vAzKp.jpg http://i.imgur.com/E8vAzKp.jpg
- simonster 14y agoI doubt that works when the site is up either. Many servers these days reject pings.
- gsibble 14y agoBrowsing the site is nearly impossible at the moment.
- argsv 14y agohttp://www.amazon.ca/ http://www.amazon.ca/ is OK. I get Http/1.1 Service Unavailable on first two requests. I got 500 with the message "We're very sorry, but we're having trouble doing what you just asked us to do. Please give us another chance--click the Back button on your browser and try your request again. Or start from the beginning on our homepage. "
- deleted 14y ago[deleted]
- knowtheory 14y agoaws != amazon. It's perfectly possible for AWS to keep running just fine, while Amazon the website bursts into flames.
- Osiris 14y agoDoes Amazon not run their website on AWS? I assumed (incorrectly, apparently) that AWS was originally built to allow Amazon to scale their own services. Is it really a separate product that they don't use themselves?
- mathrawka 14y agoJust because a site is on AWS does not mean it cannot go down for its own reasons. There are more failures possible than infrastructure.
- simonster 14y agoJust because the servers themselves are up doesn't mean that the software running on the servers is up (or that it's capable of handling the load). I just got my shopping cart to load, but it took quite a long time. Maybe they're getting DOSed.
- knowtheory 14y agoOriginally, that's true, Amazon didn't run on AWS afaik. But i believe they do now. Nevertheless, they still have application architecture which sits above the aws substrate. It's perfectly feasible for them to have seriously fucked up a deployment that runs on top of AWS, which may be functioning just fine (and at least all of my services running out of us-east seem to be up and running).
- deleted 14y ago[deleted]
- 14y ago
- ck2 14y agoComes up for me http://www.amazon.com/gp/cart/view.html http://www.amazon.com/gp/cart/view.html
- austenallred 14y agoAt an estimated loss of $31,000 per minute http://news.cnet.com/8301-10784_3-9962010-7.html?tag=nefd.top http://news.cnet.com/8301-10784_3-9962010-7.html?tag=nefd.to... I'm blown away that I see Amazon goes down so often. That certainly, in my mind, doesn't bode well for the brand of AWS.
- joet3ch 14y agoand that was for 2008, probably $100,000+/minute now
- gry 14y agoHeroku is up: http://cl.ly/image/0B0U1K3Z342R http://cl.ly/image/0B0U1K3Z342R. Conversely, when AWS had issues, Amazon.com was not impacted. Amazon.com != AWS. I'm curious to know when AWS or Amazon.com innovations impact each other, or which one leads. I'd rather it be Amazon.com.
- pkfrank 14y agoEven if Amazon.com != AWS, it is still bad for the Amazon Brand which encompasses AWS. If they can't keep their own server up, how can you trust them with yours? An unfair argument, perhaps, but one that impacts them all the same.
- ceol 14y agoI think the gp was saying that the brand— as in, the perception of AWS— will suffer, not the actual services. Anyone with an ounce of server knowledge would know it's impossible to keep a website up for 100% of the time, so downtime at Amazon is understandable, but maybe the average Joe Manager is deciding between Rackspace and AWS and happens to visit amazon.com during this downtime. "If Amazon can't even keep their bread-and-butter running, how can I trust them with something like AWS?" he might say.
- scottbruin 14y ago> Anyone with an ounce of server knowledge would know it's impossible to keep a website up for 100% of the time As far as I know Google has 100% uptime, so it's not impossible. May not be 100% for every geographical location but that's partly because of things Google cannot control nor make redundant.
- begurken 14y agoWow, almost 40 minutes of total outage now. They're going to be having one helluva '5-whys' tomorrow (http://en.wikipedia.org/wiki/5_Whys http://en.wikipedia.org/wiki/5_Whys).
- monksy 14y agoSounds like someone that is oncall is going to have a bad night. Amazon.de and .co.uk are up.
- tquai 14y agoIt's a lesson in overengineering. At this point my $5 Pentium 3 server has a greater uptime than Amazon.
- vineel 14y agoExcept your Pentium 3 server doesn't have to handle over 100 million unique visitors per quarter. :P
- setrofim_ 14y agoUnless your server needs to handle comparable amounts of traffic, it's not the same thing.
- tquai 14y agoIt doesn't. My server is appropriately engineered to its task.
- potatolicious 14y agoYour $5 Pentium 3 server isn't the largest retail website on the internet making $61 billion a year. Having seen a lot of the code that Amazon runs on, and having seen first-hand the scale that it runs on, I'll say this: it's not perfect, but it's remarkably well-engineered, and a hell of a lot better than most snarky HNers could do.
- hhw 14y agoBut that's the point. Most people don't need anything that well-engineered. Compared to more traditional hosting solutions from quality providers, AWS has terrible uptime and at a much higher cost for the same amount of resources. Two VPS'es from two different providers in a simple failover configuration with an anycast DNS solution would be simpler, cheaper, and much more reliable.
- hhw 14y agoWow, apparently that last comment really hit a nerve, as several people decided to downvote it, but not a single person actually refuted any of what I said. I was under the impression that downvotes were more to be used against trolling or flamebaiting, and not just opinions that people disagreed with. Considering everything I said is quite easy to verify as being true, this downvoting just strikes me as kind of intellectually dishonest. I expected better from HN.
- ivabz 14y agoSeems like Only US market got goosebumps. UK looks fine and up.
- podperson 14y agoSomewhat off-topic: my (limited) experience with Amazon Prime video suggests it's significantly less reliable than Netflix or iTunes (neither of which are stupendously reliable, but I'd say Netflix is by far the most reliable of the three). Hulu might actually be worse than Amazon Prime.
- notimetorelax 14y agoI don't know if you saw this posted on HN, but Netflix test their system really well. They use so-called chaos monkey [1] that shuts down random servers on a whim. This allows them to detect and get rid of dependencies, i.e. tolerate failures in other parts of the system. [1] http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html
- mistermumble 14y agoIsn't Netflix hosted on AWS?
- svedlin 14y agoNetflix is indeed running on AWS. Some details about their back-end here: http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome.html http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome...
- jdrobins2000 14y agoWish I had thought of chaos monkey, much cooler than whatever I called my version of it. A few years ago I built an automated test system in perl, complete with message bus and message listener container for running tasks on various servers. One of the automated tests I wrote had a component that would periodically (at random intervals) kill processes, unmount shared filesystems, offline interfaces, etc. to cause failovers, to verify that all processes and resources were failed over, and all tasks were reassigned to other nodes and no jobs were dropped or stalled. It is really the only way to ensure you've covered your bases - beating the shit out of your system repetitively. It uncovered a bunch of big holes and some very obscure ones too, and once we got those fixed it ran pretty much flawlessly.
- rapcal 14y agoBack online
- begurken 14y agoStill dead as a dodo here in the Bay Area. Edit: ... and it's back as of 23:00 PDT.
- gkoberger 14y agoA while ago, someone claiming to have worked at Amazon said that downtime doesn't really affect things as much as you'd think. He said most people simply just come back later. https://news.ycombinator.com/item?id=5147461 https://news.ycombinator.com/item?id=5147461 [Edit: That being said, there's also the statistic that every 100ms of latency costs Amazon 1%. Imagine what 20+ minutes of "latency" would do. https://news.ycombinator.com/item?id=273900 https://news.ycombinator.com/item?id=273900]
- ankitml 14y agothe latency vs cost curve will not be linear. After some latency, increase in latency wont affect cost much.
- troels 14y agoAlso, latency (a constant lack of resources) and down time (an extraordinary lack of resources) are two very different things. I wouldn't be surprised if some down time had little impact on sales, whereas latency has a lot.
- ankitml 14y agotrue! latency is also location dependent. so in latency network and health of other external 'resources' also matter. Whereas downtime is only a server issue. So they are two very different things.
- jonny_eh 14y agoI imagine the difference between latency and downtime is that latency tends to occur every time you visit, while by definition, downtime is more rare. In other words, latency provides a bad experience, while downtime provides no experience.
- SatvikBeri 14y agoThe Oatmeal actually has a great comparison of how people react to latency vs how they react to downtime: http://theoatmeal.com/comics/no_internet http://theoatmeal.com/comics/no_internet
- michaelrbock 14y agoAnd it's back up for me.
- DallaRosa 14y agoAmazon.com is back up
- danielovichdk 14y agoUse Windows Azure!
- lucb1e 14y agoHad downtime yesterdayevening (12 hours ago) as well in the Netherlands. People from Germany were able to load the website (the .com version; .de worked at all times), and after two hours I was able to as well. Upon trying to add something to my cart it returned the same error 500 though, so that was still down the last time I checked (about 10 hours ago). I'm not sure if or when this was resolved. I didn't submit this as story because I didn't think anyone would care, given the recent call not to post downtimes. Given the #1 spot the story has now, it seems I should have. So do people care or not?
- ketralnis 14y agoDo we really need a front-page post every time a well-known site has a hiccup? It's bad enough getting it every time github does. What are you hoping for here? A thread full of me-toos?
- mattbillenstein 14y agoThere are two types of websites, those that have suffered downtime, and those that will.
- geuis 14y agoThis is just a small complaint coming many hours after this link was posted. You linked to perhaps the most top-level URL amazon has available for a temporary outage. This means a couple things. 1) Hours later, the outage is over and I'm just hitting the home page. No specific information about what you were reporting. 2) This specific link is, as far as I know, now no linger available for other stories. That may not matter I the long run but it bares mentioning.