5 ms·
AWS Elastic Block Store issues in one AZ in us-east-1
The Tarsnap website went offline at 1:05 PDT and attempts to launch a new instance in the same availability zone have failed. I'm also seeing the EC2 Management Console getting very slow -- possibly because lots of other people are logging in to investigate their failing systems.
As of 1:40 PDT I'm seeing EBS requests served and the Tarsnap website is back online.
Amazon's status page reports:
1:22 PM PDT We are investigating degraded performance for some volumes in a single AZ in the US-EAST-1 Region
1:29 PM PDT We are investigating degraded performance for some EBS volumes and elevated EBS-related API and EBS-backed instance launch errors in a single AZ in the US-EAST-1 Region.
RDS and ELB are also now (1:40 PM PDT) showing "investigating connectivity issues". Sounds like there was a core networking outage in that AZ.
2:21 PM PDT We have identified and fixed the root cause of the performance issue. EBS backed instance launches are now operating normally. Most previously impacted volumes are now operating normally and we will continue to work on instances and volumes that are still experiencing degraded performance.
- josephpmay 13y agoIs Instagram still on Amazon servers?
- flyt 13y agoYes.
- josephpmay 13y agoI wonder if this outage will encourage them to migrate over to Facebook's servers.
- aioprisan 13y agodoubt it, since they probably properly set up their infrastructure with multi-az setup
- nowarninglabel 13y agoI was getting ngnix 504's on their site consistently during this outage time, perhaps coincidence, but seems likely to be related.
- jaytaylor 13y agoThis is the kind of thing you're susceptible to when you use AWS products that start with E: EBS, ELB, etc..
- mh- 13y agowhere's EC2 fall in that?
- nieksand 13y agoAt a minimum, you should use at least two availability zones if you care about high-availability on AWS. The vast majority of issues (including the current EBS hiccup) tend to be limited to one AZ. For us, today's outage triggered a bunch of alerts, knocked 1/4 of our USE attribution servers offline (we're in four zones there) but otherwise caused no impact. Mostly just acked alerts and twiddled thumbs waiting for situation to resolve.
- weitzj 13y agoif you want a safer EBS, create a raid ontop i and use snapshots.
- gregr401 13y agoNo, this is the kind of thing that happens with a poor application architecture (ie, using one AZ). Sad that companies still can't get this right.
- aren55555 13y agoThis is currently causing Heroku to have issues too. https://status.heroku.com/incidents/548 https://status.heroku.com/incidents/548
- jdleesmiller 13y agoNow reporting as resolved... Except that my database is still down. Sigh. And it's not just me -- a few others have tweeted at @herokustatus.
- bgentry 13y agoHi, we usually track individual database issues via outbound support tickets opened on your account. The exception is for widespread outages, but this one was actually fairly small in scope. If you didn't receive a support notification, please contact support@heroku.com so we can figure out why.
- jdleesmiller 13y agoThanks. I've now done this and referred some others to this comment.
- jday 13y agothis affected me as well, i had to provision a new database and promote it to master in order to bring my app back online
- kennyma 13y agoI had to do the same and Heroku support is non-responsive. This is beyond frustrating.
- roncohen 13y agoAWS knows which AZ is affected and they permutate them so AZ-A can be different for different people, but why can't they tell me which one is affected (from my perspective) when I'm logged into the console? "Availability Zone Status" shows all green.
- helper 13y agoIf you care about uptime you should really get off EBS (and services that depend on it like ELB and RDS). Having a network between your disks and your CPU make outages like this one inevitable.
- akurilin 13y agoWhat do you recommend as an alternative for hosting large DBs on?
- jacquesm 13y agoDefine 'large'. Up to 80T can be done in one box these days with up to 32 cores right next to it. Not cheap but fairly reliable. You'll still need to replicate off-site.
- helper 13y agoUse ephemeral disks on EC2 and a good distributed database like Cassandra or Riak. These scale well as your data grows and allow you to lose individual nodes without losing data (in sane configurations).
- eikenberry 13y ago+1 to the use of Riak. It is great. But if you are stuck using Postgresql or similar you put your live servers on Ephemeral instances and put one (unused otherwise) slave on EBS so you can snapshot it for backups. And you need at least 2 ephemeral systems so you have a slave to fail over to.
- alrs 13y agoIt used to be when EBS wiped out it didn't respect the boundary between availability zones. Every AZ in a region would be lost in tandem. If they have managed to keep this incident confined to one AZ without trashing the whole region they have made significant progress.
- bgentry 13y agoor maybe it just wasn't as bad as previous incidents :)