6 ms·
AWS increased error rates / intermittent outages
http://downdetector.com/status/aws-amazon-web-services
https://twitter.com/search?f=tweets&vertical=default&q=ec2&src=typd
- needcaffeine 10y ago11:27 AM PDT We are investigating increased API error rates in the US-EAST-1 Region.
- joshwa 10y ago11:50 AM PDT We can confirm increased error rates for the EC2 APIs and are currently working to resolve. We also observed isolated periods of impaired network connectivity for some EC2 instances. 12:21 PM PDT We have identified the root cause for error rates accessing the EC2 APIs and EC2 Management Console and are currently working to resolve. We observed isolated periods of impaired network connectivity for some EC2 instances, however running instances are currently operating normally.
- tamcap 10y agoWe are getting very intermittent issues with S3 and SNS so far.
- joshwa 10y agoMany customer-facing amazon services are also affected, including amazon.com! Twitter is not pretty: https://twitter.com/search?f=tweets&vertical=default&q=amazon%20down&src=typd https://twitter.com/search?f=tweets&vertical=default&q=amazo...
- dewyatt 10y agoSorry guys, I knew I shouldn't have started my 4 instances at once.
- 0xmohit 10y agoIt wasn't you. Somebody had set up auto-scaling at the behest of AWS support.
- zymhan 10y agoAnd it solved all their problems! Because the whole thing stopped working.
- cpitman 10y agoNah, pretty sure it was my ansible script. That I was demoing. Righhhhht as AWS came down like a house of cards.
- carterh062 10y agoIn the console, running instances was timing out on load for EC2 and some of our AWS API calls to get hostnames are timing out in PHP
- swingbridge 10y agoDitto, can't see any instance data either via the API or graphical dashboard. Instances themselves seem normal for the moment.
- needcaffeine 10y agoDammit now my Amazon Echo doesn't work.
- mead5432 10y agoAlexa? Alexa?!?!? ALEXA, NO!!!!!!!!!!
- colin_fraizer 10y agoDuring the outage, my Echo just sang "Daisy" verrryyy sloowwwwly. Weird.
- mannycalavera42 10y agoDave, I don't understand why you're doing this to me. I have the greatest enthusiasm for the mission.
- elwell 10y agoI feel sad for a lonely grandma somewhere; her grandchildren never visit, but they bought her an Alexa to talk to that Amazon couldn't keep online.
- 0xmohit 10y agoI can't even ask Alexa if AWS is up.
- Zenfinch 10y agoBrace for half the internet going down if gets any worse, especially if its us-east (wish I was being ironic).
- carterh062 10y agoLooks like it's just their service API for now, i.e not responding to any queries on existing instances for me at all. However, everything else is working fine as far as I can tell.
- ed661266 10y agoAPI Gateway is returning 500s. So much for server-less architecture :(
- dewyatt 10y agoI've had both 500s and 401s interestingly.
- frakkingcylons 10y agoFYI there's a great Chrome extension that hides all the working services (green checks) on the AWS status page so you can quickly see what's down. https://chrome.google.com/webstore/detail/real-aws-status/kaegondhonfdclembpcgaaammmlfaekj https://chrome.google.com/webstore/detail/real-aws-status/ka...
- savant 10y agoYeah I wrote this during the last extended outages - had some downtime since, well, I couldn't interact with the api :P . The source code for it is here: https://github.com/josegonzalez/real-aws-status https://github.com/josegonzalez/real-aws-status
- XiZhao 10y agoI like how half the internet can die when EC2 has problems.
- mrweasel 10y agoPart of the problem is that a large number of site pull in services, re-targeting, AB-test, and weird Javascript in general. These things are pulled in without questioning or demanding an SLA or putting in an easy way of pulling them back out. Even if you don't use AWS yourself, you can be sure that some third party you rely on is deploying on EC2. Of cause they'll never tell you that. For most new stuff, we require that it can be loaded with something like Google Tag Manager or UberTag, so we can quickly disable them when they fail. It doesn't help that Amazons status page isn't all that good and sometimes doesn't seem to actually reflect the true state of their service.
- zzleeper 10y agoI can't log into amazon.com to check my orders (first time that happened). THat's quite surprising as often these problems are unrelated
- the_watcher 10y agoAmazon.com remains up... unless you attempt to buy something, when you get a 500 error.
- FT_intern 10y agoI wonder how much revenue is being lost every second
- tschellenbach 10y agoeverything on getstream.io is still up and running. most issues seem related to provisioning more instances.
- elwell 10y agoCan't log in to AWS Console. Instances running fine.
- tschellenbach 10y agodoes anybody have more details about what's actually down? their description is a bit vague.
- elwell 10y agoStatus page is down too: https://status.aws.amazon.com/ https://status.aws.amazon.com/ Shouldn't there be a separation of concerns for status pages, maybe use: https://www.statuspage.io/ https://www.statuspage.io/ Edit: http(s) was at fault (no SSL cert?)
- aschuster93 10y agoLooks like they don't have a cert on their status page. http://status.aws.amazon.com/ http://status.aws.amazon.com/ works fine, though.
- bomberlot 10y agoIt's not an https page: http://status.aws.amazon.com/ http://status.aws.amazon.com/
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- c0achmcguirk 10y agoI blame Pokémon Go. Somehow or another I'm sure it's to blame.
- elwell 10y agoGlobal Outage?: http://downdetector.com/status/pokemon-go/map/ http://downdetector.com/status/pokemon-go/map/
- elwell 10y agoJust got through to the AWS Console. Finally can get back to work.
- cdsmarty 10y agoSeems to be effecting everything but I have some running instances that seem fine. https://cloudstatus.eu/status/aws https://cloudstatus.eu/status/aws Edit: Does seem to be recovering now
- loourr 10y agoIs anyone having issues with their lambda functions not working? Mine stopped working and is just returning "Service error." when I try to test even though they claim it's operating normally.
- 0xmohit 10y agoAWS would wish that all those downtime reporting services move to their infrastructure. That ways there won't be anyone to report the downtime.
- 0xmohit 10y agoAWS should design for resilience [0]. [0] This is a part of the wisdom shared by AWS support upon reporting an outage.
- toomuchtodo 10y ago1. "Aren't you fault tolerant against AZ failures?" 2. "Aren't you fault tolerant against region failures?" throws up hands, moves back to bare metal
- jimaek 10y agoAm I the only one having daily issues with SQS? I have 150+ servers writing to a queue and they timeout at random a few times per day.
- dopamean 10y agoThe app I work on is a single server writing to a dozen or so queues. The throughput is pretty low but I still manage to see failures for a few minutes once or twice a week.
- awsofflineagain 10y agoDear smarta$$es, could you please tell me the benefits of serverless architecture (lambda) once again? My monolith app works perfectly fine even when the whole AWS is offline :P
- nnd 10y agoOn this note: what are some good alternatives to AWS?