5 ms·

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can't do anything about it. For example, many Heroku services are broken because of this.

Bombthecat 6y ago
I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...
- ben509 6y ago
  Yeah, the cascading failure of all the other services is a deep architectural issue. Having lots of services that do one thing and one thing well makes a lot of sense. Breaking them out into separate components brings a level of visibility into the system. And it's AWS's whole business model. But it does mean that, fundamentally, service X is available when and only when (WAOW?) services A, B, C, etc. are all available. Its uptime is no greater than min(uptime(A), uptime(B), etc) I'm trying to rework the authentication for our application and integrate it with our parent company's systems. As we talk to other teams, I see all these architecture diagrams where the solution to every problem is Yet Another Service, to where you're running a real rube goldberg machine.
  - adewinter 6y ago
    Interested to know what the alternative might be and why it would mean better uptime
    - aledalgrande 6y ago
      The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.
- one2know 6y ago
  Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.
  - ultimoo 6y ago
    Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?
    - narism 6y ago
      Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
      - jjoonathan 6y ago
        Yep. Everyone ships the org chart.
    - one2know 6y ago
      Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful. In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API. Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.
that_guy_iain 6y ago
This is what SLAs are for. Especially if you're using a 3rd party service.
- avh02 6y ago
  Yes, but those cute SLAs don't help your own customers who missed/lost/delayed things. Their downtime is your downtime, and with vendor lock in it means it's harder to just march elsewhere when there's a problem.
  - that_guy_iain 6y ago
    They don't help your customers, but they help you recover the lost revenue. The penalties we have in our SLA result in our company doing things they really don't want to pay.
    - avh02 6y ago
      fair, but if you lose your customer's trust that might not be recoverable. Depends on your business
donor20 6y ago
Seriously, I get that something falls over. But to have it be 5 hours to recover, for a service this critical is nuts
- jlmorton 6y ago
  More like ten hours at this point for Kinesis
- hallway_monitor 6y ago
  Pretty critical to other Amazon services too. We use Merchant Web Services to import orders from Amazon. Down since 9:30 AM. At this point we have thousands of orders we are unable to import and process.
ttam 6y ago
I can imagine that there are literally 100s of engineers involved in trying to fix this ASAP, since this is not only bringing down the systems of external customers, but also critical internal systems, plus the bad PR. All on the eve of thanksgiving.

Bombthecat 6y ago

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

ben509 6y ago
Yeah, the cascading failure of all the other services is a deep architectural issue. Having lots of services that do one thing and one thing well makes a lot of sense. Breaking them out into separate components brings a level of visibility into the system. And it's AWS's whole business model. But it does mean that, fundamentally, service X is available when and only when (WAOW?) services A, B, C, etc. are all available. Its uptime is no greater than min(uptime(A), uptime(B), etc) I'm trying to rework the authentication for our application and integrate it with our parent company's systems. As we talk to other teams, I see all these architecture diagrams where the solution to every problem is Yet Another Service, to where you're running a real rube goldberg machine.
- adewinter 6y ago
  Interested to know what the alternative might be and why it would mean better uptime
  - aledalgrande 6y ago
    The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.
one2know 6y ago
Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.
- ultimoo 6y ago
  Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?
  - narism 6y ago
    Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
    - jjoonathan 6y ago
      Yep. Everyone ships the org chart.
  - one2know 6y ago
    Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful. In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API. Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.

that_guy_iain 6y ago

This is what SLAs are for. Especially if you're using a 3rd party service.

avh02 6y ago
Yes, but those cute SLAs don't help your own customers who missed/lost/delayed things. Their downtime is your downtime, and with vendor lock in it means it's harder to just march elsewhere when there's a problem.
- that_guy_iain 6y ago
  They don't help your customers, but they help you recover the lost revenue. The penalties we have in our SLA result in our company doing things they really don't want to pay.
  - avh02 6y ago
    fair, but if you lose your customer's trust that might not be recoverable. Depends on your business

donor20 6y ago

Seriously, I get that something falls over. But to have it be 5 hours to recover, for a service this critical is nuts

jlmorton 6y ago
More like ten hours at this point for Kinesis
hallway_monitor 6y ago
Pretty critical to other Amazon services too. We use Merchant Web Services to import orders from Amazon. Down since 9:30 AM. At this point we have thousands of orders we are unable to import and process.

ttam 6y ago

I can imagine that there are literally 100s of engineers involved in trying to fix this ASAP, since this is not only bringing down the systems of external customers, but also critical internal systems, plus the bad PR. All on the eve of thanksgiving.