8 ms·

AWS us-east-1 down

53 points by sionarancsle 3y ago

LazyMans 3y ago
Our apps are working fine, so I don't think "us-east-1" is down. Just the console..
- jrcii2 3y ago
  Eh I'm seeing more here. Athena is struggling, IMBD search is down, Vercel is struggling.
- metadaemon 3y ago
  Yep, us-east-1 itself is online.
- lonnyk 3y ago
  It's more than just console. Lambda or API Gateway is borked. I can't log in to check though.
  - deleted 3y ago
    [deleted]
nodomain 3y ago
Same here. But status page is happy. Lots of alerts in parallel throughout the infrastructure.
totaldude87 3y ago
every single time https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/
aketchum 3y ago
all of my lambda functions are returning "We currently do not have sufficient capacity in the region you requested." and the console is definitely down.
- grep_it 3y ago
  ya, we're also seeing lambda failing across the board
g00bdin 3y ago
Several of our production redshift clusters are giving 504s in the console and InternalServerError exceptions related to xen_aws_credentials_mgr.cpp
mliezun 3y ago
Degraded performance for lambda https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
bovermyer 3y ago
We're seeing elevated error rates and latency and dramatically reduced throughput on our apps. This is not just Console.
xer0x 3y ago
"Degraded performance" -- ha! Our Lambdas have been 100% down for an hour!
Corrado 3y ago
Man, when Reddit goes dark it really means it!
Corrado 3y ago
UPDATE: Jun 13 3:42 PM PDT Between 11:49 AM PDT and 3:37 PM PDT, we experienced increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use of other AWS services. Additionally, customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support. As of 2:47 PM, the issue initiating calls and chats to AWS Support was resolved. By 1:41 PM, the underlying issue with the subsystem responsible for AWS Lambda was resolved. At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM, the backlog was fully processed. The issue has been resolved and all AWS Services are operating normally.

AWS us-east-1 down

53 points by sionarancsle 3y ago

LazyMans 3y ago

Our apps are working fine, so I don't think "us-east-1" is down. Just the console..

jrcii2 3y ago
Eh I'm seeing more here. Athena is struggling, IMBD search is down, Vercel is struggling.
metadaemon 3y ago
Yep, us-east-1 itself is online.
lonnyk 3y ago
It's more than just console. Lambda or API Gateway is borked. I can't log in to check though.
- deleted 3y ago
  [deleted]

nodomain 3y ago

Same here. But status page is happy. Lots of alerts in parallel throughout the infrastructure.

totaldude87 3y ago

every single time https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/

aketchum 3y ago

all of my lambda functions are returning "We currently do not have sufficient capacity in the region you requested." and the console is definitely down.

grep_it 3y ago
ya, we're also seeing lambda failing across the board

g00bdin 3y ago

Several of our production redshift clusters are giving 504s in the console and InternalServerError exceptions related to xen_aws_credentials_mgr.cpp

mliezun 3y ago

Degraded performance for lambda https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status

bovermyer 3y ago

We're seeing elevated error rates and latency and dramatically reduced throughput on our apps. This is not just Console.

xer0x 3y ago

"Degraded performance" -- ha! Our Lambdas have been 100% down for an hour!

Corrado 3y ago

Man, when Reddit goes dark it really means it!

Corrado 3y ago

UPDATE: Jun 13 3:42 PM PDT Between 11:49 AM PDT and 3:37 PM PDT, we experienced increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use of other AWS services. Additionally, customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support. As of 2:47 PM, the issue initiating calls and chats to AWS Support was resolved. By 1:41 PM, the underlying issue with the subsystem responsible for AWS Lambda was resolved. At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM, the backlog was fully processed. The issue has been resolved and all AWS Services are operating normally.