4 ms·

Amazon S3 & SQS US East outages

30 points by bgentry 16y ago

Amazon S3 US East experienced "Elevated Error Rates" from 2:01 PDT to 2:30 PDT. Amazon SQS US East experienced "Elevated Error Rates" from 2:31 PDT to 2:45 PDT.

eclark 16y ago
Overall Amazon gets a good grade for their handling of the incident. They responded quickly and fixed everything in short order. ~30mins of downtime.
- seldo 16y ago
  A blog post giving a little clarity on the source of the outage would not go amiss, but I guess they're still investigating.
  - cperciva 16y ago
    They usually don't do blog posts, but do put significant detail onto the status page.
bgentry 16y ago
What's the extent of the damage that everybody experienced from this outage?
- deleted 16y ago
  [deleted]
- blasdel 16y ago
  I was having %90 of my requests time out. Very little of the software I'm using handled the exceptions raised while this was happening. I checked the status page first thing, but it didn't get set to yellow for another 5 minutes after I started having problems.
  - timf 16y ago
    > little of the software I'm using handled the exceptions raised while this was happening Are saying there is a library you are using that failed to treat a timeout as an issue at all? (if this is for SQS, may I ask what that library is?)
    - blasdel 16y ago
      The aws-s3 gem was throwing an exception correctly after a fairly long timeout ("We encountered an internal error. Please try again.") But none of the software that was using the aws-s3 gem or causing it to be called caught the exception. I ended up with exceptions getting thrown all over the place, mostly in views from methods on data coming out of models (Rails was catching it at the top-level as ActionView::TemplateError)
      - timf 16y ago
        Ah, thanks for explaining more.
- blantonl 16y ago
  I had an archive server go down because of queued up requests to send data to S3. Good lesson for us, we need to handle these types of issues better on our end.
zacharypinter 16y ago
I guess this explains why Unfuddle was down earlier today.
zzleeper 16y ago
In other words, my dropbox went down =/

30 points by bgentry 16y ago

Amazon S3 US East experienced "Elevated Error Rates" from 2:01 PDT to 2:30 PDT. Amazon SQS US East experienced "Elevated Error Rates" from 2:31 PDT to 2:45 PDT.

eclark 16y ago

Overall Amazon gets a good grade for their handling of the incident. They responded quickly and fixed everything in short order. ~30mins of downtime.

seldo 16y ago
A blog post giving a little clarity on the source of the outage would not go amiss, but I guess they're still investigating.
- cperciva 16y ago
  They usually don't do blog posts, but do put significant detail onto the status page.

bgentry 16y ago

What's the extent of the damage that everybody experienced from this outage?

deleted 16y ago
[deleted]
blasdel 16y ago
I was having %90 of my requests time out. Very little of the software I'm using handled the exceptions raised while this was happening. I checked the status page first thing, but it didn't get set to yellow for another 5 minutes after I started having problems.
- timf 16y ago
  > little of the software I'm using handled the exceptions raised while this was happening Are saying there is a library you are using that failed to treat a timeout as an issue at all? (if this is for SQS, may I ask what that library is?)
  - blasdel 16y ago
    The aws-s3 gem was throwing an exception correctly after a fairly long timeout ("We encountered an internal error. Please try again.") But none of the software that was using the aws-s3 gem or causing it to be called caught the exception. I ended up with exceptions getting thrown all over the place, mostly in views from methods on data coming out of models (Rails was catching it at the top-level as ActionView::TemplateError)
    - timf 16y ago
      Ah, thanks for explaining more.
blantonl 16y ago
I had an archive server go down because of queued up requests to send data to S3. Good lesson for us, we need to handle these types of issues better on our end.

zacharypinter 16y ago

I guess this explains why Unfuddle was down earlier today.

zzleeper 16y ago

In other words, my dropbox went down =/