4 ms·

Parts of AWS are down

33 points by Bob312371 7y ago

RDS & S3 is down for me right now in US-East region and apparently some parts of Frankfurt are also down. There own favicon is timing out: https://awsmedia.s3.amazonaws.com/favicon.ico

wldlyinaccurate 7y ago
From https://status.aws.amazon.com/ https://status.aws.amazon.com/ Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.
- joshuapinter 7y ago
  Finally acknowledging the issue.
faCeti0us 7y ago
Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution. https://status.aws.amazon.com/ https://status.aws.amazon.com/
- Bob312371 7y ago
  Why does the status page say resolved when S3 is still down for parts of US?
DevX101 7y ago
Having issues here in US East as well.
joshuapinter 7y ago
Having DNS issues with s3-us-west-2 as well.
joshuapinter 7y ago
We manually updated our `/etc/hosts` file with the static ip for our S3 bucket(s) and that fixed the issue temporarily.
nakodari 7y ago
Glad to hear the issue is on Amazon's side. Our team panicked and tried to find every way to resolve this thinking it was possibly a misconfiguration at our end. I hope it's resolved soon. We're receiving complains from many users who are unable to upload their work due to this issue, I will let them know Amazon is on it.
- Bob312371 7y ago
  Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account. I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.
  - ej12n 7y ago
    Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)
    - nakodari 7y ago
      The issue started happening much sooner, around morning time in EST, but it got worse as the day progressed. It looks like the Amazon engineers woke up and realized something was wrong and they updated their status page 5-6 hours late. But I agree, AWS is super reliable, it's unfortunate that this problem occurred and I hope it's resolved soon.
champbronc2 7y ago
Seeing the same thing
lflux 7y ago
DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com
- champbronc2 7y ago
  Where you see this is DDoS?
  - smudgymcscmudge 7y ago
    I'm not OP, but I heard that unofficially from somebody who works at aws.
  - jared52 7y ago
    I just got off a chat with an AWS agent: "We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name."
  - lflux 7y ago
    From the phonecall with our AWS TAM and support reps
raghava 7y ago
status check: Tue Oct 22 22:50:22 UTC 2019 DNS resolution to AWS S3 and RDS is still buggy on other providers (DigitalOcean, etc as well) https://status.digitalocean.com/incidents/1z3kmlvz69v6 https://status.digitalocean.com/incidents/1z3kmlvz69v6 https://developercommunity.visualstudio.com/content/problem/788578/job-fails-with-an-error-apparently-there-is-a-conn.html https://developercommunity.visualstudio.com/content/problem/...
MastaX1 7y ago
Seeing major issues here as well. All my modules were not working for the majority of the day and all images were not displaying. This is no doubt a Ddoss attack, I just don't see any other reason that the entire east coast AWS DNS servers would just stop responding all of a sudden. If the issue is not fixed by tomorrow our IT guy is gonna have a damn field day implementing all the temporary mitigation's.

33 points by Bob312371 7y ago

RDS & S3 is down for me right now in US-East region and apparently some parts of Frankfurt are also down. There own favicon is timing out: https://awsmedia.s3.amazonaws.com/favicon.ico

wldlyinaccurate 7y ago

From https://status.aws.amazon.com/ https://status.aws.amazon.com/ Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.

joshuapinter 7y ago
Finally acknowledging the issue.

faCeti0us 7y ago

Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution. https://status.aws.amazon.com/ https://status.aws.amazon.com/

Bob312371 7y ago
Why does the status page say resolved when S3 is still down for parts of US?

DevX101 7y ago

Having issues here in US East as well.

joshuapinter 7y ago

Having DNS issues with s3-us-west-2 as well.

joshuapinter 7y ago

We manually updated our `/etc/hosts` file with the static ip for our S3 bucket(s) and that fixed the issue temporarily.

nakodari 7y ago

Glad to hear the issue is on Amazon's side. Our team panicked and tried to find every way to resolve this thinking it was possibly a misconfiguration at our end. I hope it's resolved soon. We're receiving complains from many users who are unable to upload their work due to this issue, I will let them know Amazon is on it.

Bob312371 7y ago
Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account. I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.
- ej12n 7y ago
  Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)
  - nakodari 7y ago
    The issue started happening much sooner, around morning time in EST, but it got worse as the day progressed. It looks like the Amazon engineers woke up and realized something was wrong and they updated their status page 5-6 hours late. But I agree, AWS is super reliable, it's unfortunate that this problem occurred and I hope it's resolved soon.

champbronc2 7y ago

Seeing the same thing

lflux 7y ago

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

champbronc2 7y ago
Where you see this is DDoS?
- smudgymcscmudge 7y ago
  I'm not OP, but I heard that unofficially from somebody who works at aws.
- jared52 7y ago
  I just got off a chat with an AWS agent: "We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name."
- lflux 7y ago
  From the phonecall with our AWS TAM and support reps

raghava 7y ago

status check: Tue Oct 22 22:50:22 UTC 2019 DNS resolution to AWS S3 and RDS is still buggy on other providers (DigitalOcean, etc as well) https://status.digitalocean.com/incidents/1z3kmlvz69v6 https://status.digitalocean.com/incidents/1z3kmlvz69v6 https://developercommunity.visualstudio.com/content/problem/788578/job-fails-with-an-error-apparently-there-is-a-conn.html https://developercommunity.visualstudio.com/content/problem/...

MastaX1 7y ago

Seeing major issues here as well. All my modules were not working for the majority of the day and all images were not displaying. This is no doubt a Ddoss attack, I just don't see any other reason that the entire east coast AWS DNS servers would just stop responding all of a sudden. If the issue is not fixed by tomorrow our IT guy is gonna have a damn field day implementing all the temporary mitigation's.