5 ms·
AWS having major issues
- vonklaus 11y agoKilling me I was wondering why my sites weren't working and console access is killed. Can't log in, can't find any information. Thanks for this. Huge.
- uditsrn28 11y agohey vonklaus, what u can do is just login to console , when u see 500 error page . Just hit the back button you will be logged into the account
- vonklaus 11y agocouldn't get that to work for me. I tried doing an incognito window, hitting the back button, etc. I thought it was something specific to me as I had just killed a shell session and deleted my only droplet when I noticed. It's up now though, glad I saw this post though thanks.
- colinbartlett 11y agoI have been in that same situation before so I built StatusGator.io. It basically alerts you when a service posts a status update so you aren't spinning your wheels trying to figure out what's wrong with your app.
- dc_gregory 11y agoWas ~ an hour before the status page indicated even the potential for a fault, and we were seeing solid errors the entire time. Currently can't raise support tickets either, so left in the dark until they fix that...
- K0nserv 11y agoEU-West is also experiencing problems. A lot of our instances are currently in connection_lost status. EDIT: The console is working fine at the moment EDIT1: Apparently OpsWorks is hosted and managed by the North Virginia data center which is why all our opsworks instances in EU-West are experiencing issues too.
- kacy 11y agoOk. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be looking at the opsworks agent.
- codingninja 11y agoWe are getting errors across multiple AWS API's. It's nothing to do with Opsworks itself, rather it appears like there is an internal networking issue. Both SQS and SNS were erroring and now SQS has gone down completly with all requests timing out.
- deleted 11y ago[deleted]
- oliyoung 11y agoLooks like there's critical infrastructure in us-east-1 that's broken and causing a ripple effect across all of AWS Our platform is entirely hosted in ap-southeast-2 but we've had our EC2 instances deregistered and OpsWorks reporting them terminated where EC2 is showing them active and they're still reachable via SSH
- frankchn 11y agoYeah, we don't use OpsWorks and had SQS/SNS/SES trouble as well -- thankfully those are not used to serve production traffic. From the set of services affected, it looks like Amazon's internal Kafka-like pub/sub system went down.
- warren_s 11y agoWe started getting SQS errors from us-east-1 around 05:08 UTC. We also have a bunch of files on S3 which Cross-Region Replication hasn't yet replicated... I think that depends on SQS as well.
- robotpony 11y ago
- jayunit 11y agoUsing OpsWorks. Got several deploys timing out as "unreachable" between 2h ago and now.
- phrotoma 11y agoHeads up if you're using opsworks your instances may have been removed from their ELB.
- fredonrails 11y agoexactly what is happening here.
- technofriends 11y agoWe are in Singapore and our ELB is automatically deregistering our instances. Has happened twice in last 1 hour causing system downtime.
- technofriends 11y agowe use Opsworks.
- fredonrails 11y agoHalf of our servers in Singapore (100+) are all down, cant access console either. Damn it.
- uditsrn28 11y agohey fredonrails, what u can do is just login to console , when u see 500 error page . Just hit the back button you will be logged into the account. this seemed to be working for me please try
- codingninja 11y agoCAn confirm we are also experiencing massive issues, took AWS about an hour to update status page. Attempted to file support ticket and that timed out, when the request finished it had created 10 duplicated tickets and still waiting on the phone call from AWS RE ticket.
- codingninja 11y agoAll of our opsworks instances are currently being shut down, we have already had 20 servers automatically terminated... There goes the weekend!!
- deleted 11y ago[deleted]
- MichaelCrawford 11y agoSome of my Portland neighbors are concerned with rising rents as well as Oregon's No-Fault evictions. San Francisco too. While it has rent control there are such workarounds as remodeling an apartment. They figure they can give all the hipsters the boot by cutting optical fiber. Good thing it's clearly marked with all those "Call Before You Dig" signs! HOWTO Win a Cold War When It Suddenly Grows Hot http://www.warplife.com/tips/defense/national/c3i/fiber-optic-cable-cut/ http://www.warplife.com/tips/defense/national/c3i/fiber-opti... tl;dr: backhoes are cheap to rent and easy to operate. AWS Boston failed the same time Portland and San Francisco did, but I don't know why yet. My concern when I posted the above was the Livermore cut, I figured Vladimir Putin had grown weary of America's craft beer export restrictions. "The FBI is investigating" according to all the newspapers, but when I reported the Portland cut to their national crime tips hotline, the FBI agent I spoke too didn't even know there was such a thing as optical fiber. There was a cut in San Jose in June 2010. 8-1-1 exists specifically to prevent fiber cuts but there is no way to report that a cut had actually happened. Santa Clara Vallet 9-1-1 hung up on me when I requested they connect me to a Network Operations Center. 6-1-1 is what I wanted. I called the FBI for the Portland cut because 6-1-1 was down as well. Emergency Responders all use radio for many good reasons but that Portland cut disabled 9-1-1. What actually made the news around here? A local TV station's cable channel was down for a few hours. Mom's Earthlink was down. "That's because of the cable cut, Mom. There is nothing that Earthlink Tech Spport can do." "That's fine, so long as I can get my eMail!" Mom pays for her dialup, therefore she expects it to work.
- MichaelCrawford 11y ago"We don't assume you are stupid." http://prgmr.com/ Tell Brother Luke that Brother Mike sent you. Their tech support is excellent, it is run far far better than Amazon could ever hope to achieve. If any prgmr.com server ever goes down, Luke's girl puts his testicles in a bench clamp until that server comes back up.
- sdrothrock 11y agoServers in Japan seem ok, but e-mail (N. Virginia) is more or less dead for me.
- benjaminRRR 11y agoWe've been having issues with SES N.Virginia, but have managed to get mail through. It's intermittent, but we are getting our batches out. It's not pretty.
- sdrothrock 11y agoOur e-mails are more or less mission-critical, so we try to send, then try to send again -- I suspect the additional failures are causing this to back up a bit. For concrete numbers, of 27 e-mails we tried to send, only 2 got out successfully. Also of interest is that I only got a 454 ("Temporary service failure") three times -- the rest of the time, it just freezes/times out in ssl.
- benjaminRRR 11y agoAlthough it's looking like end-user deliverability is not great we are seeing huge delays in things actually getting out of SES.
- technofriends 11y agoAWS issues on Sysadmin Appreciation day.. what an irony.
- stephenr 11y agoFor some users of AWS this would be irony, its also schadenfreude for the Ops/Sysadmins who tell management types that just trusting everything to AWS is a bad idea.
- beilabs 11y agoSo three different environments have had the ELB just release all of their clients and not come back onboard. Manual intervention was the only way. Whats the recommended way to monitor if instances have become detached or non responsive? Want to immediately alert slack or send an email, etc.
- Rapzid 11y agoThere are a lot of ways to integrate, but we use datadog and I generally find it excellent. Alerts to hipchat, pagerduty, etc, with AWS Cloudwatch integration. Of course, you have to assume the Cloudwatch API is working... :)
- vacri 11y agoA vote for datadog here too. Just started using it - it's clear that someone there really loves data.
- MichaelCrawford 11y agodjbdns enables one to configure multiple IPs for just one http server. Host with multiple, geographically distant data centers. If any one server drops offline, remove it from your tinydns "data" file. For that to yield joy you need a shorter timeout than is normally used for webserver DNS. Each data center must be operated by a different company. The problem tonight appears to be internal to Amazon, so different AWS data centers won't help you. I only recommend four. The people that run them know more about hosting than you or I could ever hope to: http://prgmr.com/ http://www.budgetdedicated.com/ http://prq.se/?intl=1 Ivory Tower Group LLC is no longer in the hosting biz but I expect I can convince them to return to it, given tonight's market opportunity. If so I recommend them highly as well, and will post their hosting service URL in a reply. PRQ is somewhat pricy, but: Our boundless commitment to free speech has been tested and proven over and over again. If it is legal in Sweden, we will host it, and will keep it up regardless of any pressure to take it down. We have ZERO tolerance against SPAM and related services! We defend your integrity to the end. With our discreet customer relations policy we don't even have to know who you are, and if we do, we will keep that knowledge strictly confidential. PRQ has its own in-house legal team to fight takedown demands. However they only rent physical servers, they don't do VPS. Truth In Labeling: BudgetDedicated and prgmr.com host me for free, prgmr.com because I helped them with their marketing, BudgetDedicated because the server I use hosts an Open Source and a Free Software project.
- ranman 11y agogovcloud seems ok
- mmattax 11y agoWe seem to have had issues and we're not using OpWorks. I'm trying to determine if our issues are related. Our instances are docker hosts, network seemed to lag/stop when proxying traffic to the internal container IP addresses. Our ASG spun up other instances but health checks reported "Insufficient Data". The web console also seems buggy (API requests are failing).
- shashwat986 11y agoSQS seems to be resolved now. (As per website)
- vonklaus 11y agoOk, I can't tell if this is functioning normally. I am trying to launch a new instance. Numerous services have "increased API error rates" one even said "Elevated error rates" All services panel were green indicating: Service is operating normally I can't put confidence in this. Anyone have any info on whether this is resolved. I can do some on digital ocean, but I really need a few things on AWS. Confirmed up? COnfirmed working? Info? Edit: Successfully launched an AMI instance and it deployed successfully and is accessible from shell and ip.
- javiercr 11y agoWe have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status. What an awful way to start a day (in Europe). Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't touch it). Update 2: our new instances are finally being created and running setup. Update 3: we found that the Health Check for our Elastic Load Balancer changed its path to the default one (index.html). We had to edit it again.. This is weird, I don't like the idea of Amazon changing our Health Check path for no reason.
- K0nserv 11y agoMost of our OpsWorks projects in EU-West went down and all the critical ones at that. I envy your luck
- ghayes 11y agoI'm in the same situation on US West on OpsWorks. Servers all began to terminate and will not start back up. I put too many eggs in the OpsWorks basket, I suppose.
- chris7 11y agoAll our production servers in OpsWorks had "Connection lost" status. Currently most of them is back "online" but from time time downtime (few minutes) comes back on some instances. AWS Support is not working (50x and 40x errors) - they have moved the support for production to amazon.com
- toriaezu 11y agoHey javiecr, Can you please post your OpsWorks instances ids on the AWS forums and just c/p what you wrote here? https://forums.aws.amazon.com/forum.jspa?forumID=153 https://forums.aws.amazon.com/forum.jspa?forumID=153 thanks.
- javiercr 11y agoDo you work for Amazon? We already sent a support ticket using http://www.amazon.com/gp/html-forms-controller/support-center-issues-u http://www.amazon.com/gp/html-forms-controller/support-cente...
- econner 11y agoSo many pages...
- twykke 11y agoall our opsworks servers have been unreachable from the internet since around 05:13Z when we see a mysterious 'configure' command with no further details in the opsworks logs EDIT: they are in EU-West zone btw
- atopuzov 11y agoI go to vacation and everything collapses, guess they will chain me to the desk next time :-)
- fasteo 11y agoI can relate
- codingninja 11y agoFortunately the Opsworks service wasn't able to contact EC2 API so although it attempted to terminate the servers, the termination call failed so the instances are still alive and functioning but there is now a disconnect between Opsworks and EC2
- j42 11y agoChiming in as I was completely unaffected... maybe this can help someone here. I switched our applications off of OpsWorks a couple months back after losing faith in the OW team, in favor of a lower-level CI flow using auto-scaling groups and CodeDeploy. Aside from some headaches in the very beginning (where CodeDeploy maintenance window would crash the daemon and health checks would cause an "initialize/destruct" loop) it precluded this issue entirely. Two of our boxes were affected by the outage and disabled, however they were re-initialized within minutes by the ASG, meaning we experienced essentially no downtime. Just be aware that if you use CodeDeploy, it's essentially just a low-level deployment hook which takes the packaged revision passed to S3 from your continuous integration setup, unpackages and runs any initialization scripts you require. You'll need to configure the security groups and scaling policies on your own which is something I know OpsWorks tries to make easier with their higher level app/layer constructs...
- fredonrails 11y agoThanks for sharing your ideas. Actually I want to do that as well. I'm so tired of Opsworks, failing deploy and of course today issue. Could you share more about your code-deploy setup and auto-scaling ?
- vaibhavrajput 11y agoAWS SES 454 Temporary service failure
- halotrope 11y agoWe where also affected in Frankfurt, really scary. Does anyone know about some managed chef like Ops-Work that would be cloud-agnostic with a little less vendor-lock in?
- fizx 11y agoTheir status history is 100% green for the last week. You must be wrong about major issues. /s
- systemz 11y agoThere is your reliable cloud :) You can always count on cloud, right?
- mryan 11y agoWhat a pointless comment. Nobody expects 100% uptime on any cloud service.
- randomsearch 11y agoAre private servers 100% reliable?
- slaman 11y agoIs your ISP 100% reliable? Even with the perfect local infrastructure there is still a single point of failure.
- danieltamiosso 11y agoAll my (5) instances (OpsWorks) in South America were with status "Stopping" and detached from ELB. I manually stop and start again. Now it is working but ~ again ~ I have 2 instances with "Stopping" status. Welcome weekend!
- BIllyPeanutsDev 11y agoOur Amazon _POST_ORDER_FULFILLMENT_DATA_ feeds have started going through now so it must be nearly fixed!!! I thought I was going to have to tell all our customers they need to manually dispatch all their orders. Woo hoo!!!
- technofriends 11y agoWe were all good since last few hours (post the outage session in the morning) but just 10 mins back our ELB again automatically deregistered all the EC2 instances. Is the issue still happening?