7 ms·
Amazon was Down
- seanp2k2 13y agoCan't get into AWS management console either.
- ritchiea 13y agoI can't get to the management console or status.aws.amazon.com
- itomatik 13y agoaws console seems fine to me. the amazon.com is down for sure.
- andrewryno 13y agoYeah same here. Amazon is throwing a "Http/1.1 Service Unavailable" but AWS console is fine.
- magnacartic 13y agoJust successfully launched an instance, but I can't get my Prime video!
- davorak 13y agoDoing fine here, both amazon.com and the AWS console.
- rattray 13y agohere=where?
- michaelburk 13y agoI got a 500 error on amazon.com. Now just a timeout.
- rdl 13y agoWow. Something this big means I'll bet it's a networking issue. I wonder if they lose money for a brief outage, or if people just delay their purchases. I seem to remember them graphing this somewhere.
- level09 13y agohttp://i.imgur.com/E8vAzKp.jpg http://i.imgur.com/E8vAzKp.jpg
- simonster 13y agoI doubt that works when the site is up either. Many servers these days reject pings.
- gsibble 13y agoBrowsing the site is nearly impossible at the moment.
- argsv 13y agohttp://www.amazon.ca/ http://www.amazon.ca/ is OK. I get Http/1.1 Service Unavailable on first two requests. I got 500 with the message "We're very sorry, but we're having trouble doing what you just asked us to do. Please give us another chance--click the Back button on your browser and try your request again. Or start from the beginning on our homepage. "
- deleted 13y ago[deleted]
- knowtheory 13y agoaws != amazon. It's perfectly possible for AWS to keep running just fine, while Amazon the website bursts into flames.
- Osiris 13y agoDoes Amazon not run their website on AWS? I assumed (incorrectly, apparently) that AWS was originally built to allow Amazon to scale their own services. Is it really a separate product that they don't use themselves?
- mathrawka 13y agoJust because a site is on AWS does not mean it cannot go down for its own reasons. There are more failures possible than infrastructure.
- simonster 13y agoJust because the servers themselves are up doesn't mean that the software running on the servers is up (or that it's capable of handling the load). I just got my shopping cart to load, but it took quite a long time. Maybe they're getting DOSed.
- knowtheory 13y agoOriginally, that's true, Amazon didn't run on AWS afaik. But i believe they do now. Nevertheless, they still have application architecture which sits above the aws substrate. It's perfectly feasible for them to have seriously fucked up a deployment that runs on top of AWS, which may be functioning just fine (and at least all of my services running out of us-east seem to be up and running).
- deleted 13y ago[deleted]
- 13y ago
- ck2 13y agoComes up for me http://www.amazon.com/gp/cart/view.html http://www.amazon.com/gp/cart/view.html
- austenallred 13y agoAt an estimated loss of $31,000 per minute http://news.cnet.com/8301-10784_3-9962010-7.html?tag=nefd.top http://news.cnet.com/8301-10784_3-9962010-7.html?tag=nefd.to... I'm blown away that I see Amazon goes down so often. That certainly, in my mind, doesn't bode well for the brand of AWS.
- joet3ch 13y agoand that was for 2008, probably $100,000+/minute now
- gry 13y agoHeroku is up: http://cl.ly/image/0B0U1K3Z342R http://cl.ly/image/0B0U1K3Z342R. Conversely, when AWS had issues, Amazon.com was not impacted. Amazon.com != AWS. I'm curious to know when AWS or Amazon.com innovations impact each other, or which one leads. I'd rather it be Amazon.com.
- pkfrank 13y agoEven if Amazon.com != AWS, it is still bad for the Amazon Brand which encompasses AWS. If they can't keep their own server up, how can you trust them with yours? An unfair argument, perhaps, but one that impacts them all the same.
- ceol 13y agoI think the gp was saying that the brand— as in, the perception of AWS— will suffer, not the actual services. Anyone with an ounce of server knowledge would know it's impossible to keep a website up for 100% of the time, so downtime at Amazon is understandable, but maybe the average Joe Manager is deciding between Rackspace and AWS and happens to visit amazon.com during this downtime. "If Amazon can't even keep their bread-and-butter running, how can I trust them with something like AWS?" he might say.
- scottbruin 13y ago> Anyone with an ounce of server knowledge would know it's impossible to keep a website up for 100% of the time As far as I know Google has 100% uptime, so it's not impossible. May not be 100% for every geographical location but that's partly because of things Google cannot control nor make redundant.
- begurken 13y agoWow, almost 40 minutes of total outage now. They're going to be having one helluva '5-whys' tomorrow (http://en.wikipedia.org/wiki/5_Whys http://en.wikipedia.org/wiki/5_Whys).
- monksy 13y agoSounds like someone that is oncall is going to have a bad night. Amazon.de and .co.uk are up.
- tquai 13y agoIt's a lesson in overengineering. At this point my $5 Pentium 3 server has a greater uptime than Amazon.
- vineel 13y agoExcept your Pentium 3 server doesn't have to handle over 100 million unique visitors per quarter. :P
- setrofim_ 13y agoUnless your server needs to handle comparable amounts of traffic, it's not the same thing.
- tquai 13y agoIt doesn't. My server is appropriately engineered to its task.
- potatolicious 13y agoYour $5 Pentium 3 server isn't the largest retail website on the internet making $61 billion a year. Having seen a lot of the code that Amazon runs on, and having seen first-hand the scale that it runs on, I'll say this: it's not perfect, but it's remarkably well-engineered, and a hell of a lot better than most snarky HNers could do.
- hhw 13y agoBut that's the point. Most people don't need anything that well-engineered. Compared to more traditional hosting solutions from quality providers, AWS has terrible uptime and at a much higher cost for the same amount of resources. Two VPS'es from two different providers in a simple failover configuration with an anycast DNS solution would be simpler, cheaper, and much more reliable.
- hhw 13y agoWow, apparently that last comment really hit a nerve, as several people decided to downvote it, but not a single person actually refuted any of what I said. I was under the impression that downvotes were more to be used against trolling or flamebaiting, and not just opinions that people disagreed with. Considering everything I said is quite easy to verify as being true, this downvoting just strikes me as kind of intellectually dishonest. I expected better from HN.
- ivabz 13y agoSeems like Only US market got goosebumps. UK looks fine and up.
- podperson 13y agoSomewhat off-topic: my (limited) experience with Amazon Prime video suggests it's significantly less reliable than Netflix or iTunes (neither of which are stupendously reliable, but I'd say Netflix is by far the most reliable of the three). Hulu might actually be worse than Amazon Prime.
- notimetorelax 13y agoI don't know if you saw this posted on HN, but Netflix test their system really well. They use so-called chaos monkey [1] that shuts down random servers on a whim. This allows them to detect and get rid of dependencies, i.e. tolerate failures in other parts of the system. [1] http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html
- mistermumble 13y agoIsn't Netflix hosted on AWS?
- svedlin 13y agoNetflix is indeed running on AWS. Some details about their back-end here: http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome.html http://techblog.netflix.com/2012/12/aws-reinvent-was-awesome...
- jdrobins2000 13y agoWish I had thought of chaos monkey, much cooler than whatever I called my version of it. A few years ago I built an automated test system in perl, complete with message bus and message listener container for running tasks on various servers. One of the automated tests I wrote had a component that would periodically (at random intervals) kill processes, unmount shared filesystems, offline interfaces, etc. to cause failovers, to verify that all processes and resources were failed over, and all tasks were reassigned to other nodes and no jobs were dropped or stalled. It is really the only way to ensure you've covered your bases - beating the shit out of your system repetitively. It uncovered a bunch of big holes and some very obscure ones too, and once we got those fixed it ran pretty much flawlessly.
- rapcal 13y agoBack online
- begurken 13y agoStill dead as a dodo here in the Bay Area. Edit: ... and it's back as of 23:00 PDT.
- gkoberger 13y agoA while ago, someone claiming to have worked at Amazon said that downtime doesn't really affect things as much as you'd think. He said most people simply just come back later. https://news.ycombinator.com/item?id=5147461 https://news.ycombinator.com/item?id=5147461 [Edit: That being said, there's also the statistic that every 100ms of latency costs Amazon 1%. Imagine what 20+ minutes of "latency" would do. https://news.ycombinator.com/item?id=273900 https://news.ycombinator.com/item?id=273900]
- ankitml 13y agothe latency vs cost curve will not be linear. After some latency, increase in latency wont affect cost much.
- troels 13y agoAlso, latency (a constant lack of resources) and down time (an extraordinary lack of resources) are two very different things. I wouldn't be surprised if some down time had little impact on sales, whereas latency has a lot.
- ankitml 13y agotrue! latency is also location dependent. so in latency network and health of other external 'resources' also matter. Whereas downtime is only a server issue. So they are two very different things.
- jonny_eh 13y agoI imagine the difference between latency and downtime is that latency tends to occur every time you visit, while by definition, downtime is more rare. In other words, latency provides a bad experience, while downtime provides no experience.
- SatvikBeri 13y agoThe Oatmeal actually has a great comparison of how people react to latency vs how they react to downtime: http://theoatmeal.com/comics/no_internet http://theoatmeal.com/comics/no_internet
- michaelrbock 13y agoAnd it's back up for me.
- DallaRosa 13y agoAmazon.com is back up
- danielovichdk 13y agoUse Windows Azure!
- lucb1e 13y agoHad downtime yesterdayevening (12 hours ago) as well in the Netherlands. People from Germany were able to load the website (the .com version; .de worked at all times), and after two hours I was able to as well. Upon trying to add something to my cart it returned the same error 500 though, so that was still down the last time I checked (about 10 hours ago). I'm not sure if or when this was resolved. I didn't submit this as story because I didn't think anyone would care, given the recent call not to post downtimes. Given the #1 spot the story has now, it seems I should have. So do people care or not?
- ketralnis 13y agoDo we really need a front-page post every time a well-known site has a hiccup? It's bad enough getting it every time github does. What are you hoping for here? A thread full of me-toos?
- mattbillenstein 13y agoThere are two types of websites, those that have suffered downtime, and those that will.
- geuis 13y agoThis is just a small complaint coming many hours after this link was posted. You linked to perhaps the most top-level URL amazon has available for a temporary outage. This means a couple things. 1) Hours later, the outage is over and I'm just hitting the home page. No specific information about what you were reporting. 2) This specific link is, as far as I know, now no linger available for other stories. That may not matter I the long run but it bares mentioning.