5 ms·

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

ben509 6y ago
Yeah, the cascading failure of all the other services is a deep architectural issue. Having lots of services that do one thing and one thing well makes a lot of sense. Breaking them out into separate components brings a level of visibility into the system. And it's AWS's whole business model. But it does mean that, fundamentally, service X is available when and only when (WAOW?) services A, B, C, etc. are all available. Its uptime is no greater than min(uptime(A), uptime(B), etc) I'm trying to rework the authentication for our application and integrate it with our parent company's systems. As we talk to other teams, I see all these architecture diagrams where the solution to every problem is Yet Another Service, to where you're running a real rube goldberg machine.
- adewinter 6y ago
  Interested to know what the alternative might be and why it would mean better uptime
  - aledalgrande 6y ago
    The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.
one2know 6y ago
Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.
- ultimoo 6y ago
  Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?
  - narism 6y ago
    Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
    - jjoonathan 6y ago
      Yep. Everyone ships the org chart.
  - one2know 6y ago
    Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful. In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API. Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.
    - Jabberwock12 6y ago
      I've been here for years in a senior capacity and I've never even heard of Simple Workflow. Some products, like Dynamo and Lambda became favorable internally to support the migration to native AWS. I'm not sure that assigning this to a perceived internal power grab aligns with reality. > obvious to me > would become very powerful > cabal

ben509 6y ago

Yeah, the cascading failure of all the other services is a deep architectural issue. Having lots of services that do one thing and one thing well makes a lot of sense. Breaking them out into separate components brings a level of visibility into the system. And it's AWS's whole business model. But it does mean that, fundamentally, service X is available when and only when (WAOW?) services A, B, C, etc. are all available. Its uptime is no greater than min(uptime(A), uptime(B), etc) I'm trying to rework the authentication for our application and integrate it with our parent company's systems. As we talk to other teams, I see all these architecture diagrams where the solution to every problem is Yet Another Service, to where you're running a real rube goldberg machine.

adewinter 6y ago
Interested to know what the alternative might be and why it would mean better uptime
- aledalgrande 6y ago
  The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.

one2know 6y ago

Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.

ultimoo 6y ago
Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?
- narism 6y ago
  Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
  - jjoonathan 6y ago
    Yep. Everyone ships the org chart.
- one2know 6y ago
  Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful. In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API. Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.
  - Jabberwock12 6y ago
    I've been here for years in a senior capacity and I've never even heard of Simple Workflow. Some products, like Dynamo and Lambda became favorable internally to support the migration to native AWS. I'm not sure that assigning this to a perceived internal power grab aligns with reality. > obvious to me > would become very powerful > cabal