10 ms·
AWS is down due to an electrical storm in the US
- jaequery 14y agowith storms getting worse each year, i'll likely be choosing either central or westcoast datacenters ... from now on.
- deleted 14y ago[deleted]
- freditup 14y agoTornadoes in the midwest and earthquakes in the west. You're doomed. Seriously though, as horrible as downtime is, I think most internet users aren't terribly surprised when they can't go to a specific website for a short period of time.
- jser 14y agoAny press is good press? Can't count the number of times I read about the Twitter fail whale.
- MichaelApproved 14y agoI'd be interested in seeing someone polish this turd. How would you spin amazon being down into good press when people are looking for reliability. Only thing I can think of is people will be writing about you and that should increase your page rank but I doubt amazon is concerned about higher page rank. Anyone else care to speculate how this could be good press?
- eropple 14y agoIt could be an opportunity to explain what reliability actually is (it's not "pack one availability zone with all your stuff") and how AWS helps you achieve that.
- MichaelApproved 14y agoThat's awesome spin. So Amazon gives you the Chaos Monkey, whether you like it or not. http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html http://www.codinghorror.com/blog/2011/04/working-with-the-ch...
- bbq 14y ago"I can't run my business/research/whatever" vs "This site is so popular it barely works"
- robryan 14y agoFor ecommerce, people will just got buy somewhere else. So not to big a deal for them but a really big deal to your bottom line. Especially when you are still paying for traffic acquisition, sure you can turn Adwords off in minutes but other traffic sources may take 24 hours.
- SomeCallMeTim 14y agoIf you're willing to go outside AWS...Denver. We occasionally see tornadoes out here, but Denver is outside "Tornado Alley" [1], so they're pretty rare. On the WikiPedia tornado page they set the odds of a tornado hitting the downtown area of a big city at around one in a thousand per year. If you want to be extra paranoid, you could always pick a secondary data center location where it would be safer in the summer, since Denver would be safe from tornadoes in the winter. [2] Or you could just put one on the West Coast and hope that there's not a 1-in-a-thousand tornado within a week of a big earthquake. [1] https://en.wikipedia.org/wiki/File:Tornado_Alley_Diagram.svg https://en.wikipedia.org/wiki/File:Tornado_Alley_Diagram.svg [2] http://www.weather.com/outlook/weather-news/severe-weather/articles/tornadoes-by-month-denver_2010-03-29 http://www.weather.com/outlook/weather-news/severe-weather/a...
- sliverstorm 14y agoTornadoes in the midwest and earthquakes in the west. You're doomed. Yup, everywhere has got its natural disasters.
- jl6 14y agoThe Europe (Ireland) location is pretty high up the "nothing interesting ever happens here" scale.
- kolleykibber 14y agoWhat about the revolution?
- codex 14y agoPardon my rant, but I am frustrated. It seems there is always an excuse with Amazon cloud. Is Google similarly disabled?
- kroo 14y agoNope.
- jordanthoms 14y agoI wonder how much of Google's reliability comes from service failover and redundancy and how much from very reliable datacenters etc. I'd find it hard to believe that their platform and DCs aren't better than what we have seen from AWS, which could make compute engine a very attractive product.
- Strom 14y agoA lot of it comes from automatic failover. Individual data centers have issues frequently, but if you're using High Replication Datastore then you won't notice it much, apart from occasionally seeing all your instances getting killed and restarted in a new datacenter, which also results in memcache getting reset.
- hythloday 14y agoWhy is this being downvoted? Is it inaccurate?
- batista 14y agoIf you mean GAE, its even worse...
- jonny_eh 14y agoI thought App Engine was much more reliable when they switched over from the Master/Slave Datastore to the High Replication Datastore.
- RegEx 14y agoThe status page seems to really underplay the severity of the situation. Netflix and Heroku are down, yet these are just side effects of 'performance issues' instead of a 'service disruption'. I wonder what it would take to cross that threshold.
- adrianpike 14y agoAWS has historically been both slow to update and heavily optimistic with their status page. When I got the frantic texts when EC2 first dropped offline, sure enough, the AWS status page was all green, but twitter was alight with people talking about it. I suspect a service disruption would have to be Godzilla.
- RegEx 14y agoAccording to other HNers, their RSS feed seems to be fairly accurate (someone please verify this), which makes the whole thing that much weirder.
- ehsanu1 14y agoNot weird at all. Companies the size of Amazon are practically bound to be schizophrenic.
- deleted 14y ago[deleted]
- joezydeco 14y agoYou called it. I had my money on the volcano. http://www.youtube.com/watch?v=tVEPvXBEOSE http://www.youtube.com/watch?v=tVEPvXBEOSE
- rmc 14y agoWhen our EC2 hosted website seems offline and AWS status page is all green, the first thing I do is check twitter. Usually you can see people asking about outage before amazon acknoledge it.
- zacharyvoase 14y agoBy what stretch of the imagination is this icon suitable for representing a total loss of availability due to a power outage?: http://status.aws.amazon.com/images/status2.gif http://status.aws.amazon.com/images/status2.gif Is this not a 'service disruption' situation? At the bottom of the page, the yellow icon is associated with 'performance issues'. If there's one thing that's shocked me about AWS, it's the total failure to acknowledge the severity of service disruptions. Like the above case, or the fact that a 3-hour loss of connectivity is displayed on the service history as a green tick with a small 'i' box: http://oi46.tinypic.com/x5qtch.jpg http://oi46.tinypic.com/x5qtch.jpg
- hristov 14y agoIt is obvious we are dealing with the imagination of a marketing exec here. And that is a sick cynical place.
- flyt 14y agoOr that there is absolutely no way to deep link to an ongoing outage, and users must reload, then expand the link every single time, or subscribe to an RSS feed. AWS needs to blantantly copy Heroku's status system, which is worlds better for people needing fast updates on their infrastructure. https://status.heroku.com/ https://status.heroku.com/ vs http://status.aws.amazon.com/ http://status.aws.amazon.com/
- droithomme 14y agoWow, that is so beautiful! I am in awe.
- kooshball 14y agoThe whole Heroku site is phenomenal. I am constantly amazed every time i see it.
- phillmv 14y agoWhat's the deal with the % measuring? I know going below three nines is seen as "bad service", but three hours of outage in a month gives you 99.5%, not 99.97%
- adrianpike 14y agoThere's another comment thread going over at (http://news.ycombinator.com/item?id=4180339 http://news.ycombinator.com/item?id=4180339), if, like me, you got extremely lucky and picked today's lucky availability zones, and have time to read HN instead of scramble to get things back up. Good luck, friends.
- molecule 14y agoreading the linked page, "AWS is down" means "some N. Virginia AWS services are down"
- jamespcole2 14y agounfuddle is down also, can't access any of my repos. I was going to spend the day working too - oh well time for a long lunch. This isn't the first time this has happened to AWS - we moved our app to linode last year after this happened to us and it seems to affect AWS more than any other hosting i've ever used, i'd be interested to know how their infrastructure is set up because it doesn't seem particularly robust.
- lsb 14y agoInterestingly, this is a great time to see which of your favorite websites are rock-solid and which are kind of shaky. I've been thinking about building a site with a Parse backend, and they're up, which is good to discover.
- sehugg 14y agoBe careful. Nothing is as it seems right now. Do not trust any API output, nor should you do any API operations that are non-recoverable. Things are up that are reported down and vice-versa. Wait for the dust to settle. We're all just going to be a bunch of Fonzies here. EDIT: Looks like API access has been restored, so I'm cautiously optimistic about things working. Note though that some instances may have rebooted or be otherwise impacted so check your error logs. EDIT2: Nope, ELB is still hosed. Continue to be skeptical.
- somesaba 14y agoWe're soooo screwed
- deleted 14y ago[deleted]
- sliverstorm 14y agoSite failover is one of the exercises that are left to the user. Considering how relatively bare-metal an experience AWS provides, this has never surprised me.
- deleted 14y ago[deleted]
- inopinatus 14y agoAre you suggesting that AWS should operate out of more than one location? It's a wonderful idea! I even have a list of possible locations they should look into. Beyond the Virginia site, they should be looking at DCs in Oregon, California, Ireland, Singapore, Tokyo and even Sao Paulo. What do you think?
- deleted 14y ago[deleted]
- 0-_-_-_-_-_-_-0 14y agoI got a crazy idea, what if I build my own cloud on top of AWS to abstract away all this extra complexity of building redundant sites across geographically distributed datacenters? That way developers don't have to think about hardware at all!
- Aloisius 14y agoJust pay the extra money and get off US-East people.
- danryan 14y agous-west-2 is the same price as us-east-1. Price is no excuse anymore.
- rdl 14y agoLatency to Europe kind of is, if you want to be in a single site with acceptable ping times to Europe, US East, and US West. But I'd rather just solve replication and be in each.
- lucian1900 14y agoIf latency to Europe is an issue, why not just have a few machines in their Ireland datacenter?
- rdl 14y agoBecause distributing sites across multiple regions is hard -- high latency between your front end and database sucks, and replicating a database over high latency is also hard. If you can do it, great, but if you can't, then pick the best single region. US East or US West plus Direct Connect to your own colo space, with AWS for the burst capacity, and your own redundancy for the database servers, might be the best plan if you can't do wide area "over the Internet" database replication. (I might get an extra 10G DC (since I need 1G DC myself) and then have some colo for sale with it in US East/US West later this year.)
- maybird 14y agoActs of God can happen anywhere.
- nakkiel 14y ago
- spullara 14y agoOk startup people. It is worth it to host in a different zone than US-EAST.
- k-mcgrady 14y agoI host in Europe (Dublin I think). I'm from here and so are my customers so there's no latency worry for me.
- philip1209 14y agoI commented along the same lines during the last AWS/Heroku outage, but Rackspace still is giving me amazing value and uptime, and every time I try to move away (as I did this week with my lastest project, on Heroku) I get hit with a massive service disruption that pushes me back to Rackspace.
- shawnps 14y agoHi, I work at Rackspace. If you don't mind me asking, what makes you initially want to move away?
- philip1209 14y agoComplete curiosity. I love Rackspace, but am always looking to try new things - new languages, new databases, new methods of hosting, new DNS, new foods, new music - I'm just a curious person. It's not problem with Rackspace; all of my mission-critical stuff is still hosted with you. But I use random projects of mine to to explore in multiple ways (including Black Sands by Bonobo, which I discovered last week - possibly my favorite coding album now).
- shawnps 14y agoGotcha! I'll have to check out Black Sands, too.
- RegEx 14y agoBlack sands by Bonobo is amazing. The song 'Black Sands' on the album is wonderful[0]. http://www.youtube.com/watch?v=Ua2loiGHZ38 http://www.youtube.com/watch?v=Ua2loiGHZ38
- jentulman 14y agoNot an album for coding per se, but if you like Bonobo you might like Pepe Deluxe's new album Queen of the Wave. It's my favourite new album of 2012. Retro psych rock and as mad as cheese.
- jordanthoms 14y agoGoogle Compute Engine anyone?
- msie 14y agoWindows Azure anyone?
- locusm 14y agoTheir current VM performance is incredibly bad. To be fair they are in some preview mode.
- jc4p 14y agoHP I-forget-their-catchy-name-for-their-service anyone?
- raverbashing 14y agoRedHat OpenShift? Or they're just like Heroku and sit on top of AWS?
- rplnt 14y agoThey don't (RH has its own datacenters) but I think it's still in some kind of beta. Great thing is that you can rip your app out of there and maintain the "service" yourself. Also, alternative providers for OpenShift will probably pop up over time.
- suninwinter 14y agoIt looks like this is affecting iTunes Match, possibly. I have two tracks just sitting there, waiting to upload and running lsof -i shows iTunes with a connection to an AWS machine.
- gee_totes 14y agoDo we know this is due to an electrical storm? Today had a leap second as well (The minute of midnight, June 30th lasted a second longer than normal).
- dfc 14y agoThe leap second has not happened yet[1]: UTC TIME STEP on the 1st of July 2012 A positive leap second will be introduced at the end of June 2012. The sequence of dates of the UTC second markers will be: 2012 June 30, 23h 59m 59s 2012 June 30, 23h 59m 60s 2012 July 1, 0h 0m 0s [1] http://hpiers.obspm.fr/iers/bul/bulc/bulletinc.43 http://hpiers.obspm.fr/iers/bul/bulc/bulletinc.43
- jaredbeck 14y ago8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. Amazon Elastic Compute Cloud (N. Virginia)
- dangrossman 14y agoThis is the kind of weather conditions that spawn very electrically active storms. I don't doubt they could cause the issues. Last night was probably the most electrically active storm I've ever seen up here -- virtually non-stop lightning strikes for an hour or two, and there's another just like it over Virginia right now. http://i.imgur.com/d5pEP.png http://i.imgur.com/d5pEP.png
- hendler 14y agoIf you have a load balancer you may have balanced across availability zones (Not regions) you'd still be up. So US-EAST didn't all go down, just one AZ.
- genwin 14y agoBut many people are saying that despite paying for multi-AZ for RDS, they were still down. Do you think they didn't also load-balance across AZs for their webservers?
- paulsutter 14y agoAWS is not down. Only US-East. If your app is down, it's only because you don't care about the availability of your service. It's pointless to complain. We've all seen before that Amazon can't keep whole regions up. If you rely on a region being up, you will have downtime and it's your fault.
- oconnore 14y agoRobust systems aren't hip. Get back to work and ship, ship, ship.
- narsil 14y agoThat's how I feel about most hackathons. There isn't enough work done on existing projects, but quick hacks the demonstrate concepts are the norm. It would be great to see a hackathon where the primary purpose was to contribute to an existing open source project or something similar.
- haberman 14y agoAccording to the AWS status page, only one availability zone within US-East is down, not the whole US-East region. Running a highly-available service exclusively from US-East is a reasonable strategy as long as you're spread across multiple availability zones. I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.
- dragonstyle 14y agoWe're across availability zones and definitely ran into an outage across zones tonight.
- sseveran 14y agoThe idea that a single data center can host a highly available service is just wrong.
- beedogs 14y ago> If your app is down, it's only because you don't care about the availability of your service. That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?
- marcamillion 14y agoI am getting tired of all of these outages. I know outages happen all the time at hosts, and maybe as a result of either a) news is more accessible now, or b) Amazon is bigger than most other hosts....I feel like Amazon & Heroku are going down WAYYYYY too much. I am starting to wonder if this "tell all" policy is really best.
- dakrisht 14y agoIs this the same EC2 zone that went out just 3-4 days ago?? Second or I believe third power outage/loss of service for AWS in the past 10-days if I'm not mistaken. This is wild. I wonder what's going on at Amazon and if they're capable of handling this much usage in addition to having power issues, etc. Instagram and Netflix servers are down from what I hear and have been down for a few hours. Now it makes sense that they're being hosted on AWS.
- jack7890 14y agoWould it be expected that Amazon will issue substantial refunds (e.g. no charges for June hosting for impacted users) due to the problems today?
- 0-_-_-_-_-_-_-0 14y agoThat has been the policy in the past.
- manishm 14y agoSimple solution to this is to have a backup or failover to a non-AWS Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS Azure/Google/Rackspace This not only spreads your risks but keeps your customers happy.
- marcuspovey 14y agoCloud taken out by a cloud.
- tathagatadg 14y agoSo cloud is not ready for storms? (troll face)
- tathagatadg 14y agoSo cloud is not ready for storm? (troll face)
- bconway 14y agoIf you're interested in information on the storms themselves and the destruction they caused in West Virginia, there's good coverage here: http://www.foxnews.com/weather/2012/06/30/state-emergency-declared-in-west-virginia-after-powerful-storms/ http://www.foxnews.com/weather/2012/06/30/state-emergency-de...
- morsch 14y agoGov. Earl Ray Tomblin in a statement: With temperatures near 100 degrees expected this weekend, it's critical that we get people's power back on as soon as possible. So let me get this straight: the critical issue with not having electricity after a huge storm is that the A/C isn't working? And 100F/38C isn't even that hot, right?
- vidarh 14y agoHigh temperatures pretty much always lead to increased mortality for the old and sick, so yes, that is one of the critical issues. Most other places where power is critical (e.g. hospitals) have backup generators and diesel supply contracts.
- ckrailo 14y agoYes it's that hot. Google about people dying in homes at temps < 100, even.
- cowkingdeluxe 14y ago100F in a non-dry climate is very hot as it feels like 110F. Heat sickness is a very real concern, along with food spoilage and pets. Additionally, sick people without access to A/C in 100F weather can become even more sick very quickly.
- smithian 14y ago100F with high humidity will result in a number of preventable deaths due to heat exhaustion, heat stroke, dehydration, etc. So yes, it is the most critical thing, along with power to hospitals and emergency services.
- kryptiskt 14y agoAccording to Colin Percival on Twitter[1][2], the US East-1 AZ has more IP addresses, and thus probably other resources, than the rest of AWS put together. It casts comments about "limited to one availability zone" into some relief. [1] https://twitter.com/cperciva/status/219067641023840257 https://twitter.com/cperciva/status/219067641023840257 [2] https://twitter.com/cperciva/status/219067963356098561 https://twitter.com/cperciva/status/219067963356098561
- rabbitfang 14y ago> the US East-1 AZ us-east-1 is a region, containing multiple AZ's.
- streeter 14y agoYou can use the EC2 API and ec2-describe-availability-zones to find out which availability zone is having issues: http://alestic.com/2012/06/ec2-outage-availability-zone http://alestic.com/2012/06/ec2-outage-availability-zone
- dustingetz 14y agohow is it that amazon.com itself is never, ever, impacted? edit: so basically, the businesses suffering outages (heroku, netflix, etc) don't value uptime to the same extent that amazon does. they got what they paid for.
- dkulchenko 14y agoThis outage only affected us-east-1. Considering an Amazon.com outage would cost them $51k in lost sales every minute, I seriously doubt they put all their eggs (servers, that is) in one basket.
- sofuture 14y agoAmazon.com does not run on the same EC2 that you and I use. It runs on a nearly identical system that is isolated and private to Amazon. I wouldn't be surprised if they were in entirely different physical locations.
- gcb 14y agoIsn't that bastard operator from hell excuse #74?
- 16s 14y agoMust be the same storm that took several of my trees down (east coast Virginia USA) last night. It was a violent storm. 90 MPH winds. Made 80 foot tall oaks bend like straws and they were almost touching the ground. I spent the morning running the chainsaw just to clear the downed trees from the driveway. AEP (local power company) says about 65% of customers in this area are w/o power. May be days before it's fully restored. Hope no one from the HN community got hurt. Edit: I posted this from a computer in town. No power at my place so I can't respond to follow-up posts.