6 ms·
AWS issues, affecting Heroku and others
- ftwinnovations 14y agoYup, my sites are down. I'm sure it's something new and complex, like a ferret or a leprechaun got into the 7th backup power switch which overloaded this and that.. as usual something impressively difficult to plan for, but come on! Virginia you're killing me...
- peterwwillis 14y agoThat Virginia datacenter has the worst reputation among my peers for any Northeast US datacenter. I highly recommend moving.
- alanbyrne 14y agoAffecting PHPFog/Appfog too - I think their MySQL services.
- pearle 14y agoI'm affected. Site is down, no servers are accessible.
- bryanh 14y agoUs too. RDS bit the bullet for us.
- twp 14y agoAlso affecting Coursera: https://twitter.com/coursera https://twitter.com/coursera It's rather annoying as the first assignments for the Alex Aiken's excellent Compilers course are due today.
- tocomment 14y agoSorry to hear that. Maybe you can get an extension.
- amalag 14y agoYes they will usually extend it, like they did with the Godaddy outage.
- twp 14y agoThanks - I've already emailed the lecturer. For info, Coursera seems to randomize the assignments. Specifically, in the case of multiple choice tests it seems to pick a random subset of answers to each question, and randomize the order. This is obviously a defense against dumb copying. What it does mean though is that you can't resubmit your own work. I've re-done the assignment twice this evening (the questions change enough that you have to think a lot, even when re-doing the same assignment), only to get 500 errors both times. Anyhow, I'm sure it'll all work out and these temporary technical problems are but minor hiccups in what is a fantastic course and a fantastic learning system. It's very, very much appreciated.
- chimeracoder 14y ago> I've re-done the assignment twice this evening (the questions change enough that you have to think a lot, even when re-doing the same assignment), only to get 500 errors both times. It's been about 8 years, so my memory may wrong, but I believe WebAssign used do have some sort of adaptive homework, where the questions would be selected based on your answers to previous questions (the same way the GMAT or GRE works). I'm surprised that Coursera doesn't do something of the sort, both to avoid the problem that you're having (the 'state' of your homework would be updated each time you submit an answer to a question), and also to capitalize on some of the flexibility of digital learning that isn't possible on paper.
- rz2k 14y agoThe choices can change between each time you initiate a quiz. However, if you complete part of a quiz, save, then return later, the choices will be the same, but the order will be different. Not all of the courses are set up with multiple choices though. In addition to varying sets of correct responses, some answers are mathematical. For instance it is defined such that a question might ask ("What is %d plus %d", a, b), then have choices involving the quantities: ("%d", a+1), ("%d", a*b), ("%d", a+b), etc. with a different random quantity for a and b each time you start a quiz. Interestingly, I saw one course where it had single-choice questions, with more differences than simply right or wrong. For instance one answer gave 100%, another 80%, and another 0% of the points available for that question.
- dopamean 14y agoI'm super excited for a course I am enrolled in that starts today and cant log in. Lame.
- erikcw 14y agoI'm also having trouble accessing RedisToGo.
- gsibble 14y agoWell, it did rain today. So obviously AWS is down.
- cryptoz 14y agoCloud computing indeed.
- pjscott 14y agoAmazon, those masters of tactful understatement, say: > 10:38 AM PDT -- We are currently investigating degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. http://status.aws.amazon.com/ http://status.aws.amazon.com/
- elliottcarlson 14y agoNot only that - the "information" icon that they use makes it very hard to spot issues quickly. Even on a minor issue it would be useful to have some kind of different icon.
- yuvadam 14y agoAWS cleverly designed a very informative status dashboard: green means "things might work", green with a little speck on it means "nothing is working, the entire datacenter is down". Effectively, this is exactly how it is.
- pjscott 14y agoAbout 33 minutes after they first acknowledged the problem, they've changed one of those indicators to as goldenrod-yellow color. Presumably this indicates that the world is on fire and the sky has gone black with carrion crows: > 11:11 AM PDT -- We can confirm degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. Instances using affected EBS volumes will also experience degraded performance.
- Cloven 14y agoAmazon clearly needs a change in management at the head of AWS. The engineers are either petrified of telling the truth (very, very bad) or incompetent (less bad but still bad). And it has been this way for an unreasonably long time.
- cpeterso 14y agohttp://status.aws.amazon.com/images/status0.gif http://status.aws.amazon.com/images/status0.gif http://status.aws.amazon.com/images/status1.gif http://status.aws.amazon.com/images/status1.gif http://status.aws.amazon.com/images/status2.gif http://status.aws.amazon.com/images/status2.gif http://status.aws.amazon.com/images/status3.gif http://status.aws.amazon.com/images/status3.gif
- bifrost 14y agoI checked out the AWS status page, it says EBS is busted in NoVA, yet the status is still green. I think someone needs to fix their UI problem statte...
- AtTheLast 14y agoAll they had was a little information icon by the green circle with a hover message letting you know what the problem was. They need to make that circle red!
- johnrgrace 14y agoThe green is hardcoded
- blantonl 14y agoI took our Web sites offline (RadioReference.com / Broadcastify.com) so our back end processes (audio archiving, audio serving etc) don't crater. Our Master MySQL server uses EBS (striped RAID across 4 EBS disks) and it is getting killed due to severely degraded EBS performance. There is definitely a major problem with EBS at this time.
- beneth 14y agoWe are running our sites across three AZs, all in US-EAST. Our three web servers (one in each AZ) are not even able to load static test pages from their EBS volumes at this time. We also have a Percona MySQL cluster across three AZs and we are seeing very degraded perfomance in the EBS volumes on all three servers as well. It seems like this is impacting more than a single AZ from what I'm seeing.
- blantonl 14y agoWell, we've gone from "degraded performance" to completely losing our EBS instance store for our master MySQL server. If we don't see resolution shortly we might have to look at promoting one of our slaves and moving forward from there.
- blantonl 14y agoWe ended up promoting a slave to master, and moving forward from there. Our master eventually returned to service, and we'll promote him tomorrow after things settle down. Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.
- antidoh 14y agoThat's you? I love listening to my local sheriff radio. Nicely done.
- SwaroopH 14y agoI was planning to launch our startup in a few hours. Oh well.
- johncoogan 14y agoPivot to a PaaS with a focus on uptime?
- flyt 14y agoOr build your application with the characteristics of the underlying platform in mind. Nobody would run a DB server on a single spindle drive since occasionally they fail. Accordingly, nobody should deploy non-multi-az AWS applications because occasionally they fail.
- nkohari 14y agoExcept if you want to, you know, ship.
- SwaroopH 14y agoas a matter of fact, our app is using multi-zones not just for instances but also for RDS. As of now, I can't reach anything including the loadbalancer.
- flyt 14y agoAll I've ever heard and seen in practice is that RDS should generally be avoided for production usage. Its opaque reliance on EBS is its downfall.
- SwaroopH 14y agoPossibly but there are bigger companies using RDS in production – also, the multi-az feature had worked out better than my own custom setup. Unfortunately lack of resources prevent me from rolling out my own setup on bare metal.
- kmfrk 14y agoI find it really hard to hate Heroku with a status page that gorgeous. There is something to be learnt about UX there, I think.
- UnoriginalGuy 14y agoI'm not a Heroku customer, but I want to be after just looking at that status page.
- pennywyze 14y agoI found it to be pretty non-sensical to be honest. Not sure what's wrong with having a simple table with title/description/timestamp columns. I'm going to that page to figure out why my site isn't working, not view some designer's portfolio.
- bobwaycott 14y agoHad exactly the same thought, particularly in comparison to AWS's status page. I scrolled all the way down just because I was enjoying the timeline.
- matt2000 14y agoYeah, we've had failovers on RDS and Heroku is having trouble as well.
- wyck 14y agoThis is bullshit, so many critical sites offline. Talk about having all your eggs in one basket.
- Ambadassor 14y agoReddit is down as well
- bifrost 14y agoIMHO that is a sev0 problem.
- randartie 14y agowas down, somehow they recover pretty quickly
- endersshadow 14y agoStill down for me.
- clientbiller 14y agoAmazon hosts so many big sites that when it goes down its a huge disaster.
- TylerE 14y agoSo the lesson seems to be, if you want to do AWS, don't use US-EAST? Seems like that's always where these big meltdowns are.
- ojiikun 14y agoThe lesson is: don't build a system that requires or advertizes high avilability but that runs in a single datacenter. Load balance. Elect leaders. Batch, queue, and distribute. Fail-over. Fail fast. So long as we have earthquakes, lightning, human malice, and human error, single AZs will fail.
- corford 14y agoLooks like multi AZ setups are also failing :( http://news.ycombinator.com/item?id=4684491 http://news.ycombinator.com/item?id=4684491
- pjscott 14y agoRemember the old saying about "Lies, damn lies, and availability zones"?
- MicahWedemeyer 14y agoWe're paying 2x for the RDS multi-AZ setup that is supposed to failover in these cases. So far no luck. Think I'll ask for a refund.
- bryanh 14y agoSame here! Even trying to boot up some replicas is painfully slow.
- dangrossman 14y agoI gave up on Amazon (and ate the reserved instance fees) 2+ years ago over this. I was paying through the nose for multi-AZ RDS and when EBS failed in one zone, nothing failed over.
- fallous 14y agoAs an AWS user since 2009, I can only imagine the tremendous difficulties of running such a service but given the stability of my long-running instances in US West vs East, I really do sometimes wonder if Mr Bean is on staff in Virginia.
- jrochkind1 14y agoI wonder if US East just has a LOT more customers.
- deleted 14y ago[deleted]
- johncoogan 14y agoSkynet is coming online.
- mcos 14y ago"11:03 AM PDT We are currently experiencing connectivity issues and degraded performance for a small number of RDS DB Instances in a single Availability Zone in the US-EAST-1 Region." Given that my sites are deployed using Multi-AZ RDS instances and yet they're still down, this takes the cake a little.
- tarkeshwar 14y agoNot even able to ssh to aws instance. AWS console shows normal. Are others able to ssh?
- saeedjabbar 14y agonope no luck SSHing
- pearle 14y agoNope, cannot ssh into affected instances.
- ubercore 14y agoSSH issues are isolated to instances that are EBS backed. Instance storage backed is working OK for SSH.
- corford 14y agoAnd this is why my new venture will be going pinboard.in's route and hosting everything on our own machines. More work for me but vastly cheaper and hopefully a lot more reliable.
- joevandyk 14y agoJust don't use EBS. Most everything else is pretty stable.
- knodi 14y agoThats easier said than done.
- elliottcarlson 14y agoEven then, there seems to be issues with ELB currently.
- dialtone 14y agoELBs unfortunately use EBS so they are always affected in these cases.
- deleted 14y ago[deleted]
- quaunaut 14y agoColor me ignorant(no really, I am), but really- cheaper? I thought part of why you used AWS was because short of the biggest properties, running your own servers was more expensive?
- corford 14y agoThe only way AWS is theoretically cheaper (and even then I find the argument a bit of a stretch) is if you factor in "sysadmin costs". The argument goes that setting up and monitoring your own servers (especially if you go colo rather than renting) is more costly in terms of labour than point and clicking it with AWS. If you strip the labour cost out of the equation, hosting it yourself can be up to an order of magnitude cheaper. Of course stripping the labour cost out is only possible if you can do it yourself :) In my case, I can (as could maciej of pinboard.in fame). Edit: also helps if you can find a well priced data centre or two. For me, I'm planning on going with hetzner.de and ovh.net
- colinhowe 14y agoI now regret using EBS boot.
- durpleDrank 14y agoBeen getting crap loads of alerts stemming from their North Virginia data centre. Got 9 of these yesterday ... "You are receiving this email because your Amazon CloudWatch Alarm "awsec2-WM-Live1-i-8747ece1-High-CPU-Utilization" in the US - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (93.13) was greater than or equal to the threshold (85.0)." at ." Maybe their data centre was over run by mutants ?
- jrochkind1 14y agoWhile it's more frustrating to have no control over the reliability, I suspect that most companies could not achieve the level of reliability from AWS by self-hosting, at anywhere close to the cost (inc. stafftime). Nothing is 100%, and getting as close to 100% as you can gets exponentially more expensive the closer you get. Of course, this could change if AWS's reliability trends downward. But it's never going to be 100%, and neither is anything you do yourself.
- dangrossman 14y agoI don't think this is true. US-East's incident history over the past 3 years is terrible. If you had a completely non-redundant set of rented servers sitting in Softlayer or Rackspace's data center over the same period, even factoring in expected hardware failures over the time period (replacing a hard drive or two, probably), you'd have been available more than even a multi-AZ deployment with failover in AWS US-East. I've got heavily used servers at Softlayer that haven't had more than a few minutes of downtime in 4+ years, and have never been rebooted. During that same time, I ran a site out of US-East for two years (EC2 + EBS + Multi-AZ RDS + ELB), and had more downtime and spent more admin time working around significant issues with Amazon. Amazon neither saved me time nor money, nor did it provide better availability.
- jrochkind1 14y agowonder if anyone's been keeping and publishing statistics on exactly how often AWS US East has been down. Perhaps it's more than I thought. Of course, most failures haven't been effecting _all_ of US East, so maybe what would be interesting to look at is just _some_ customer's data on exactly how much downtime they had in the past 3 years or whatever. Without knowing what the uptime/downtime was, hard to say how terrible it was, or how it compares to other options.
- patrickgzill 14y agoI have a server for a client in our colo, that has been up for 1753 days. (Client doesn't want to move to new hardware, etc. otherwise we would have moved them long ago). During that time, there was maybe network outage (at 2AM mostly) of perhaps 2 hours. Complexity reduces reliability. Claiming AWS' uptime approaches that of fairly standard colocation in a decent facility, is laughable.
- rabidsnail 14y agoNote to all the postgres users cursing EBS right now: you can persist your db instances to s3 with https://github.com/heroku/WAL-E https://github.com/heroku/WAL-E, which, combined with streaming replication and some redundancy, should let you run off ephemeral disk.
- joevandyk 14y agoI recommend this as well. Performance is way more consistent on the instance/ephemeral storage as well.
- jonny_eh 14y agoAt least Netflix is still up.
- knodi 14y ago-_- I'm really beginning to hate you ec2/ebs.
- gfodor 14y agoJust a friendly reminder that AMZN now runs a large chunk of the Internet.
- ergo14 14y agoYou know guys, this always makes me wonder... So many of you experience issues because of cloud, why not got dedicated at hetzner or whatever your baremetal provider is?
- Axsuul 14y agoOn-demand instances
- whileonebegin 14y agoRan a remote dedicated server to host apps for several years. While it was an enlightening experience, it was a lot of sleepless nights and high anxiety. Traffic spikes, OS updates, security, and plain old downtime. I've used shared, VPS, dedicated, and cloud hosting for apps. They all have their fair share of downtime, but I like shared and cloud hosting the best. It allows me to focus on development, rather than running the server.
- zorlem 14y agoSo far I haven't solved the problem of keeping a non-trivial service without any interventions. How does running your servers and services on a PaaS provider solve the need for OS updates and security? What are your typical architecture and operations like? I personally think that OS updates and security fixes/hardening are crucial in all cases if you're running any production site (and even for the dev environment). Granted, some of the colocation providers don't offer services like AWS VPC (and the associated SecurityGroups and ACLs) but that's pretty rare nowadays. In all cases, and Ops or a Sysadmin person is quite important for the stability of your business. The only big advantage I see about using PaaS is the rapid and flexible provisioning of new instances in case you need them. That's why the best solution to me, so far, has been running a combination of bare-metal servers with optional provisioning of additional cloud instances if you need them. The price difference can go to orders of magnitude (even if you factor in the costs of paying somebody to be a system administrator and take care of your servers). Edit: added the last paragraph
- seunosewa 14y agoThe issues you had with dedicated servers are easy to fix: Traffic Spikes: Implement caching (most spikes are read-only). Implement queueing of expensive but less essential operations, so they can be delayed during traffic spikes. Make sure your CPU & RAM are 1.5 - 2x what you need. (Don't worry, it will still be cheaper than AWS) Plain Old Downtime: Network outages happen with cloud hosts too. Focus on development: Cloud servers are unmanaged too. OS Updates & Security: This is no more an issue on dedicated servers than it is on cloud servers. It's an OS issue. Were you running Windows Servers. That's the problem!
- pearle 14y agoI'm having issues trying accessing the AWS web console...
- krenoten 14y agoHow often do these happen? I've been using EC2 for about a week and it was a surprise when my instance ground to a halt before my eyes. Here's the series of events from my perspective: Nginx is no longer able to connect to uwsgi Instance successfully rebooted via EC2 console Celery tasks are not able to reach the RabbitMQ broker Halfway into a dump of Postgresql the session goes dead Green check on EC2 console for the instance still rclick -> stop causes instance to get stuck in a 'stopping' state, which persists to this moment.
- UnoriginalGuy 14y agoEC2 seems to be about as reliable as a mid-range host provider. Which is to say that they can do 99% but not really 99.99%. They also seem to either have no issues is everything goes out all at once.
- krenoten 14y agoThe wikipedia page for EC2 lists a couple similar incidents at the Northern Virginia datacenter. Victims of the previous NVa outages included Reddit, Quora, SpringPad, FourSquare, Netflix, etc... Many of those companies are down right now... why didn't they learn? FourSquare seems to be up - although slow. I can load Quora but didn't try to log in. Netflix, Reddit, SpringPad, seem to be down for me. http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud#Issues http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud#Is...
- dangrossman 14y agoLearning doesn't magically triple your hosting budget or your ops team headcount. Sometimes 99.9% availability is all you can afford... or all that makes sense to do the work for. Netflix didn't go down, though; I've been watching TV on it throughout this incident.
- krenoten 14y agoTrue, and learning on a monday morning is certainly easier than learning during peak hours for some of these companies. It does lead one to reexamine the value of another 9, however. Out of curiosity, does it often cost triple to shift half of the instances/LB's/etc... to another datacenter and implement the proper consistency mechanisms?
- ericcholis 14y agoI know it's been said, but the Heroku status page is the easiest to use. Very effective use of a timeline.
- aaronbrethorst 14y agoInteresting. My Heroku-hosted site seems to be doing fine: http://www.cocoacontrols.com http://www.cocoacontrols.com
- IheartApplesDix 14y agoHeroku has implemented AWS services correctly, most websites can't afford to do that.
- taligent 14y agoWell clearly they have a lot of work still to do: https://status.heroku.com/ https://status.heroku.com/ Complete outage of an AZ shouldn't ever take services offline.
- BIackSwan 14y ago12:32 PM PDT We are working on recovering the impacted EBS volumes in a single Availability Zone in the US-EAST-1 Region. Hopefully recovering in the next hour.
- BIackSwan 14y ago12:32 PM PDT We are working on recovering the impacted EBS volumes in a single Availability Zone in the US-EAST-1 Region. Hopefully recovering in the next hour.
- jpdoctor 14y agoAnyone calculated how many 9's Heroku is down to this year?
- bgentry 14y agoWe do not post a yearly number right now, but the previous 1 & 3 months are prominently displayed on our status site: https://status.heroku.com/ https://status.heroku.com/