3 ms·

Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decoup

by beambot 11y ago

Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?

Jgrubb 11y ago
I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.
- beambot 11y ago
  From the 2nd sentence: > ... subsequent impact to other AWS services that depend on DynamoDB ... And the metadata service is part of DynamoDB: > The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.
  - ckozlowski 11y ago
    I just asked the question in a separate post before seeing yours, but I questioned this as well. Would it have been better to have a separate instance of DynamoDB independent from the one customers are writing to? I understand this is added cost/overhead, but it would have resulted in isolating the failure to the customer side only, I'd think.
- threeseed 11y ago
  "Amazon SQS uses an internal table stored in DynamoDB to store information describing its queues" "EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB" "CloudWatch uses an internal table stored in DynamoDB" "Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB" Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.
cddotdotslash 11y ago
A large number of AWS services actually use other AWS services under the hood. Lambda is actually running code on EC2 stored on S3. API Gateway is actually CloudFront on the front end. Apparently most services use DynamoDB for metadata. I'm pretty sure CloudWatch logs are stored on S3. They really interconnect a lot of services. You could argue that it's good (dog fooding, better monitoring) or bad (too many dependencies) but at least they're putting confidence in their own products.

by beambot 11y ago

Jgrubb 11y ago

I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.

beambot 11y ago
From the 2nd sentence: > ... subsequent impact to other AWS services that depend on DynamoDB ... And the metadata service is part of DynamoDB: > The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.
- ckozlowski 11y ago
  I just asked the question in a separate post before seeing yours, but I questioned this as well. Would it have been better to have a separate instance of DynamoDB independent from the one customers are writing to? I understand this is added cost/overhead, but it would have resulted in isolating the failure to the customer side only, I'd think.
threeseed 11y ago
"Amazon SQS uses an internal table stored in DynamoDB to store information describing its queues" "EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB" "CloudWatch uses an internal table stored in DynamoDB" "Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB" Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.

cddotdotslash 11y ago

A large number of AWS services actually use other AWS services under the hood. Lambda is actually running code on EC2 stored on S3. API Gateway is actually CloudFront on the front end. Apparently most services use DynamoDB for metadata. I'm pretty sure CloudWatch logs are stored on S3. They really interconnect a lot of services. You could argue that it's good (dog fooding, better monitoring) or bad (too many dependencies) but at least they're putting confidence in their own products.