5 ms·

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/b

by mabbo 5y ago

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

nowahe 5y ago
I can't access anything related to Cloudfront, either through the CLI or console : $ aws cloudfront list-distributions An error occurred (HttpTimeoutException) when calling the ListDistributions operation: Could not resolve DNS within remaining TTL of 4999 ms However I can still access the distribution fine
2bitlobster 5y ago
Interactive Video Service (IVS) is down too
Guest19023892 5y ago
One of my sites went offline an hour ago because the web server stopped responding. I can't SSH into it or get any type of response. The database server in the same region and zone is continuing to run fine though.
- bijoo 5y ago
  Interesting, is the site on a particular type of EC2 instance, e.g. bare metal? I see c4.xlarge is doing fine in us-east-1.
  - Guest19023892 5y ago
    It's just a t3a.nano instance since it's a project under development. However, I have a high number of t3a.nano instances in the same region operating as expected. This particular server has been running for years, so although it could be a coincidence it just went offline within minutes of the outage starting, it seems unlikely. Hopefully no hardware failures or corruption, and it'll just need a reboot once I can get access to AWS again.
bijoo 5y ago
I see started EC2 instances are doing fine. However, starting offline instances cannot be done through AWS SDK due to the HTTP 500 error, even for Ec2 service. The CLI should be getting the HTTP 500 error too since likely the same API as the SDK.
lambic 5y ago
We've had reports of some intermittent 500 errors from cloudfront, apart from that our sites are up.
pavel_lishin 5y ago
We're seeing issues with EventBridge, other folks are having trouble reaching S3. Looks like actual services.
0xCMP 5y ago
Using cli to describe instances isn't working. Instances themselves seem fine so far.
Waterluvian 5y ago
My ECS, EC2, Lambda, load balancer, and other services on us-east-1 still function. But these outages can sometimes propagate over time rather than instantly. I cannot access the admin console.
snewman 5y ago
Anecdotally, we're seeing a small number of 500s from S3 and SQS, but mostly our service (which is at nontrivial scale, but mostly just uses EC2, S3, DynamoDB, and some basic network facilities including load balancers) seems fine, knock on wood. Either the problem is primarily in more complex services, or it is specific to certain AZs or shards or something.
SEMW 5y ago
Definitely not just the console. We had hundreds of thousands of websocket connections to us-east-1 drop at 15:40, and new websocket connections to that region are still failing. (Luckily not a huge impact on our service cause we run in 6 other regions, but still).
- andrew_ 5y ago
  Side question: How happy are you with API Gateway's WebSocket service?
  - SEMW 5y ago
    No idea, we don't use it. These were websocket connections to processes on ec2, via NLB and cloudfront. Not sure exactly what part of that chain was broken yet.
    - zedpm 5y ago
      This whole time I've been seeing intermittent timeouts when checking a UDP service via NLB; I've been wondering if it's general networking trouble or something specifically with the NLB. EC2 hosts are all fine, as far as I can tell.
heartbreak 5y ago
I'm getting blank pages from Amazon.com itself.
grumple 5y ago
I can tell you that some processes are not running, possibly due to SQS or SWF problems. Previous outages of this scale were caused by Kinesis outages. Can't connect via aws login at the cli either since we use SSO and that seems to be down.
meepmorp 5y ago
EventBridge, CloudWatch. I've just started getting session errors with the console, too.
sophacles 5y ago
I wasn't able to load my amazon.com wishlist, nor the shopping page through the app. Not an aws service specifically, but an amazon service that I couldn't use.
dangrossman 5y ago
My website that runs on US-East-1 is up. However, my Alexa (Echo) won't control my thermostat right now. And my Ring app won't bring up my cameras. Those services are run on AWS.
- kingcharles 5y ago
  Now I'm imagining someone dying because they couldn't turn their heating on because AWS. The 21st Century is fucked up.
- deleted 5y ago
  [deleted]

by mabbo 5y ago

nowahe 5y ago

I can't access anything related to Cloudfront, either through the CLI or console : $ aws cloudfront list-distributions An error occurred (HttpTimeoutException) when calling the ListDistributions operation: Could not resolve DNS within remaining TTL of 4999 ms However I can still access the distribution fine

2bitlobster 5y ago

Interactive Video Service (IVS) is down too

Guest19023892 5y ago

One of my sites went offline an hour ago because the web server stopped responding. I can't SSH into it or get any type of response. The database server in the same region and zone is continuing to run fine though.

bijoo 5y ago
Interesting, is the site on a particular type of EC2 instance, e.g. bare metal? I see c4.xlarge is doing fine in us-east-1.
- Guest19023892 5y ago
  It's just a t3a.nano instance since it's a project under development. However, I have a high number of t3a.nano instances in the same region operating as expected. This particular server has been running for years, so although it could be a coincidence it just went offline within minutes of the outage starting, it seems unlikely. Hopefully no hardware failures or corruption, and it'll just need a reboot once I can get access to AWS again.

bijoo 5y ago

I see started EC2 instances are doing fine. However, starting offline instances cannot be done through AWS SDK due to the HTTP 500 error, even for Ec2 service. The CLI should be getting the HTTP 500 error too since likely the same API as the SDK.

lambic 5y ago

We've had reports of some intermittent 500 errors from cloudfront, apart from that our sites are up.

pavel_lishin 5y ago

We're seeing issues with EventBridge, other folks are having trouble reaching S3. Looks like actual services.

0xCMP 5y ago

Using cli to describe instances isn't working. Instances themselves seem fine so far.

Waterluvian 5y ago

My ECS, EC2, Lambda, load balancer, and other services on us-east-1 still function. But these outages can sometimes propagate over time rather than instantly. I cannot access the admin console.

snewman 5y ago

Anecdotally, we're seeing a small number of 500s from S3 and SQS, but mostly our service (which is at nontrivial scale, but mostly just uses EC2, S3, DynamoDB, and some basic network facilities including load balancers) seems fine, knock on wood. Either the problem is primarily in more complex services, or it is specific to certain AZs or shards or something.

SEMW 5y ago

Definitely not just the console. We had hundreds of thousands of websocket connections to us-east-1 drop at 15:40, and new websocket connections to that region are still failing. (Luckily not a huge impact on our service cause we run in 6 other regions, but still).

andrew_ 5y ago
Side question: How happy are you with API Gateway's WebSocket service?
- SEMW 5y ago
  No idea, we don't use it. These were websocket connections to processes on ec2, via NLB and cloudfront. Not sure exactly what part of that chain was broken yet.
  - zedpm 5y ago
    This whole time I've been seeing intermittent timeouts when checking a UDP service via NLB; I've been wondering if it's general networking trouble or something specifically with the NLB. EC2 hosts are all fine, as far as I can tell.

heartbreak 5y ago

I'm getting blank pages from Amazon.com itself.

grumple 5y ago

I can tell you that some processes are not running, possibly due to SQS or SWF problems. Previous outages of this scale were caused by Kinesis outages. Can't connect via aws login at the cli either since we use SSO and that seems to be down.

meepmorp 5y ago

EventBridge, CloudWatch. I've just started getting session errors with the console, too.

sophacles 5y ago

I wasn't able to load my amazon.com wishlist, nor the shopping page through the app. Not an aws service specifically, but an amazon service that I couldn't use.

dangrossman 5y ago

My website that runs on US-East-1 is up. However, my Alexa (Echo) won't control my thermostat right now. And my Ring app won't bring up my cameras. Those services are run on AWS.

kingcharles 5y ago
Now I'm imagining someone dying because they couldn't turn their heating on because AWS. The 21st Century is fucked up.
deleted 5y ago
[deleted]