5 ms·

S3 is currently (22:00 UTC) back up. The timeline, as observed by Tarsnap: First InternalError response from S3: 17:37:29 Last successful request: 17:

S3 is currently (22:00 UTC) back up. The timeline, as observed by Tarsnap: First InternalError response from S3: 17:37:29 Last successful request: 17:37:32 S3 switches from 100% InternalError responses to 503 responses: 17:37:56 S3 switches from 503 responses back to InternalError responses: 20:34:36 First successful request: 20:35:50 Most GET requests succeeding: ~21:03 Most PUT requests succeeding: ~21:52

thenewregiment2 10y ago
no. soundcloud uses aws s3. it is still down. this is false information.
- ta_wh 10y ago
  "[RESOLVED] Increased Error Rates Update at 2:08 PM PST: As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally." https://status.aws.amazon.com/ https://status.aws.amazon.com/
  - thenewregiment2 10y ago
    oh the famous downvote for the smear campaign. just admit it.
    - ta_wh 10y ago
      Think you're mistaken, I don't have downvote privileges!
    - joatmon-snoo 10y ago
      You're getting downvotes because you don't understand that B being down is not an effective indicator of the status of A, even if B depends on A.
    - espeed 10y ago
      Claiming a statement is false when it's demonstrably true is something that will likely get downvoted every time. It's misleading to others and fills the board with noise.
- endersshadow 10y ago
  Soundcloud recovering from this failure and S3 being operational are two separate issues. We use S3 and it will take us nominally an hour to recover after S3 went up, for example. S3 has started working as of about 20 minutes ago, and things are running smoothly.
  - quakeguy 10y ago
    Thanks!
- jeffasinger 10y ago
  There are other Amazon services that were affected. For example, we're still not seeing auto scaling groups working correctly.
josephb 10y ago
Thanks for taking the time to post a timeline from the perspective of an S3 customer. It will be interesting to see how this lines up against other customer timelines, or the AWS RFO.
kaishiro 10y ago
Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?
- greenleafjacob 10y ago
  Could be anything. Most likely scenario is the internal error is a load shedding error and the 503s were when the system became completely unresponsive. If it was a configuration issue then it is more likely that it would have directly recovered rather than going 'internal error -> 503 -> internal error'.
- hmottestad 10y ago
  503 is typically what we see when our proxy can't connect to the backend server. We usually get 500 with internal server error when we've messed up the backend server. So it's likely that the first 500s were the backend for s3 failing, then they took the failing backends offline causing the load balancers to throw 503 because they couldn't connect to the backend.
- cperciva 10y ago
  My guess based on the behaviour I've seen is that internal nodes were failing, and the 503 responses started because front-end nodes didn't have any back-end nodes which were marked as "not failing and ready for more requests". When Amazon fixed nodes, they would have marked the nodes as "not failed", at which point the front ends would have reverted to "we have nodes we can send traffic to" behaviour.
- Twirrim 10y ago
  S3 is not a monolithic architecture, Amazon is a strong proponent of Service Oriented Architecture for producing scalable platforms. There are a number of services behind the front end fleet in S3's architecture that handle different aspects of returning a response. Each of those will have their own code paths in the front end, very likely developed by different engineers over the years. As ever, appropriate status codes for various circumstances are something that always seems to spur debate amongst developers. The change in status code would likely be a reflection of the various components entering unhealthy & healthy states, triggering different code paths for the front end... which suggests whatever happened might have had quite a broad impact, at least on their synchronous path components.

thenewregiment2 10y ago

no. soundcloud uses aws s3. it is still down. this is false information.

ta_wh 10y ago
"[RESOLVED] Increased Error Rates Update at 2:08 PM PST: As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally." https://status.aws.amazon.com/ https://status.aws.amazon.com/
- thenewregiment2 10y ago
  oh the famous downvote for the smear campaign. just admit it.
  - ta_wh 10y ago
    Think you're mistaken, I don't have downvote privileges!
  - joatmon-snoo 10y ago
    You're getting downvotes because you don't understand that B being down is not an effective indicator of the status of A, even if B depends on A.
  - espeed 10y ago
    Claiming a statement is false when it's demonstrably true is something that will likely get downvoted every time. It's misleading to others and fills the board with noise.
endersshadow 10y ago
Soundcloud recovering from this failure and S3 being operational are two separate issues. We use S3 and it will take us nominally an hour to recover after S3 went up, for example. S3 has started working as of about 20 minutes ago, and things are running smoothly.
- quakeguy 10y ago
  Thanks!
jeffasinger 10y ago
There are other Amazon services that were affected. For example, we're still not seeing auto scaling groups working correctly.

josephb 10y ago

Thanks for taking the time to post a timeline from the perspective of an S3 customer. It will be interesting to see how this lines up against other customer timelines, or the AWS RFO.

kaishiro 10y ago

Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?

greenleafjacob 10y ago
Could be anything. Most likely scenario is the internal error is a load shedding error and the 503s were when the system became completely unresponsive. If it was a configuration issue then it is more likely that it would have directly recovered rather than going 'internal error -> 503 -> internal error'.
hmottestad 10y ago
503 is typically what we see when our proxy can't connect to the backend server. We usually get 500 with internal server error when we've messed up the backend server. So it's likely that the first 500s were the backend for s3 failing, then they took the failing backends offline causing the load balancers to throw 503 because they couldn't connect to the backend.
cperciva 10y ago
My guess based on the behaviour I've seen is that internal nodes were failing, and the 503 responses started because front-end nodes didn't have any back-end nodes which were marked as "not failing and ready for more requests". When Amazon fixed nodes, they would have marked the nodes as "not failed", at which point the front ends would have reverted to "we have nodes we can send traffic to" behaviour.
Twirrim 10y ago
S3 is not a monolithic architecture, Amazon is a strong proponent of Service Oriented Architecture for producing scalable platforms. There are a number of services behind the front end fleet in S3's architecture that handle different aspects of returning a response. Each of those will have their own code paths in the front end, very likely developed by different engineers over the years. As ever, appropriate status codes for various circumstances are something that always seems to spur debate amongst developers. The change in status code would likely be a reflection of the various components entering unhealthy & healthy states, triggering different code paths for the front end... which suggests whatever happened might have had quite a broad impact, at least on their synchronous path components.