5 ms·

DynamoDB was a constant source of anxiety for me and my team. We fit the usage pattern perfectly, so the author's points didn't apply to us. Our primary proble

DynamoDB was a constant source of anxiety for me and my team. We fit the usage pattern perfectly, so the author's points didn't apply to us. Our primary problem was having to constantly manage and change our provisioned capacity so that we could respond to usage spikes and keep our bill from becoming stratospheric. Worse, anytime we'd add new functionality -- such as nighttime analytics jobs that scanned the table or daily backups, it meant revisiting the parameters for our auto-scaling or risk getting throttled and having our application randomly start spewing errors. DynamoDB's billing and provisioning model is awful to deal with. When your team is having serious conversations about the repercussions of your data values going from 4kb to 8kb, you very quickly begin questioning your decision to use DynamoDB in the first place.

eropple 9y ago
The high score I ever saw at a company I worked for was about $85K in DynamoDB spend. That's an eight, a five, and three zeroes after it. It was a feature so otherwise silly and simple that I don't want to flame the developers thereof by describing it, but it might have been the most gratuitous burning of cash I've ever seen at a startup.
- hesdeadjim 9y ago
  Ha, yea I've heard of numbers like that. I remember talking to someone at re:Invent who described their high five figures monthly bill in almost reverent terms -- it made sense after he described their 200+ MySQL server catastrophe DynamoDB replaced. We actually ended up engineering our system to use a variant of event sourcing. This let us dump DynamoDB and use RedisLabs and S3 in such a way that we were able to handle 1mil DAU with a 10:1 write/read ratio for around $5k/mo -- including front end/backend instances.
  - kornish 9y ago
    > This let us dump DynamoDB and use RedisLabs and S3 in such a way that we were able to handle 1mil DAU with a 10:1 write/read ratio This sounds really interesting – do you have a writeup of the system architecture anywhere?
    - twohlix_ 9y ago
      I would also be up for some reading regarding optimizations/architecture choices for a 10:1::write:read setup.
    - hesdeadjim 9y ago
      It's definitely a write-up I've wanted to make before. The short version of event sourcing is that instead of looking at your data model in standard CRUD terms, represent your data as an ordered list of changes. Updating an address field on a user model would become something like ChangeEvent{address=...}, rather than a full read/write of a user blob. To derive the current state of the object, you would replay and apply all events to a blank object. Obviously over time the list of changes could become enormous, so what you do to optimize is periodically create a snapshot of the current full state of the object and only replay events that happened after it was made. For our persistence layer we used Redis as the input queue for all events. Each object we stored required a list to push changes into, an integer counter to represent the sequence ID of the most recent event in the list, and a string value containing the location in S3 of the most recent snapshot (if it exists). Performance was incredible for writes because a) it's Redis and b) RedisLabs supports semi-transparent sharding. We used S3 to store the snapshots. We called the creation process a "compaction" because besides storing the most recent full state of the object, we would also store off the list of changes used to make it and purge those events from Redis. This kept our Redis storage usage tightly bounded and our costs down. This compaction process was also 100% safe with no chance of race conditions or data loss. Our algorithm would fetch the list of changes from Redis, create and serialize the current state of the object as a snapshot, and then write it and the list of changes to S3. Only upon success of those S3 writes would it then issue a command to Redis to purge that object's list of changes. To prevent race conditions on this part of the operation, we would send the sequence ID we are compacting from and in a Redis script we would atomically check to make sure no more-recent compaction has happened, update the snapshot pointer, and flush the list of changes only up to the sequence number we compacted from (changes were free to be added to the list while a compaction process was in progress). The killer part of this whole setup was that those snapshots and change lists stored in S3 were completely immutable and formed a linked list of the entire history of changes for an object. We could visualize and inspect the entire change history of an object, find bugs and retroactively fix them by changing how an event is applied, and if needed for read performance, cache the S3 snapshots in memcache without any concern for needing to invalidate them. Concurrent writes to an object became less of an issue as well. Instead of needing to do a CAS write or a transaction, you would just push the change you wanted to make to the object, read it back, and verify it was actually applied. No locking needed. The major downside to this architecture is that any querying you need to do will always be weakly consistent and require an external index. We had a small Elastisearch cluster for some of our objects that had fields needing indexing. One last major advantage of this setup was that it was trivial to "clone" production to your local dev machine. Because snapshots were immutable, we set up a tiered database layer that would first try and load snapshots locally, and failing that, would use a read-only IAM user to access S3 to pull down a snapshot from production. All you would then do is load a backup from a RedisLabs database locally and boom, you have an exact copy of production that you can freely modify because any new writes all happen locally. Believe it or not, the most unexpected cost from this architecture was PUT requests to S3. Depending upon the frequency of your compactions it can start to add up. However, if usage declines, that cost also declines with no impact on the availability of your data.
Zaheer 9y ago
FYI DynamoDB in last few weeks added auto-scaling: https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/ https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon... Additionally they somewhat recently added an optional caching layer: https://aws.amazon.com/dynamodb/dax/ https://aws.amazon.com/dynamodb/dax/
- hesdeadjim 9y ago
  Nice they finally added auto-scaling. It seems to function pretty similarly to the tool we built, with the same (documented) limitations. DAX would have definitely addressed many of our issues, had it been available at the time. Glad they added that.
  - azinman2 9y ago
    So does that address all your issues?
    - hesdeadjim 9y ago
      No, for our usage needing to worry about provisioned capacity at all was becoming a deal-breaker. That especially became tedious to estimate and plan for as we had more variability in the size of our values we needed to store.

eropple 9y ago

The high score I ever saw at a company I worked for was about $85K in DynamoDB spend. That's an eight, a five, and three zeroes after it. It was a feature so otherwise silly and simple that I don't want to flame the developers thereof by describing it, but it might have been the most gratuitous burning of cash I've ever seen at a startup.

hesdeadjim 9y ago
Ha, yea I've heard of numbers like that. I remember talking to someone at re:Invent who described their high five figures monthly bill in almost reverent terms -- it made sense after he described their 200+ MySQL server catastrophe DynamoDB replaced. We actually ended up engineering our system to use a variant of event sourcing. This let us dump DynamoDB and use RedisLabs and S3 in such a way that we were able to handle 1mil DAU with a 10:1 write/read ratio for around $5k/mo -- including front end/backend instances.
- kornish 9y ago
  > This let us dump DynamoDB and use RedisLabs and S3 in such a way that we were able to handle 1mil DAU with a 10:1 write/read ratio This sounds really interesting – do you have a writeup of the system architecture anywhere?
  - twohlix_ 9y ago
    I would also be up for some reading regarding optimizations/architecture choices for a 10:1::write:read setup.
  - hesdeadjim 9y ago
    It's definitely a write-up I've wanted to make before. The short version of event sourcing is that instead of looking at your data model in standard CRUD terms, represent your data as an ordered list of changes. Updating an address field on a user model would become something like ChangeEvent{address=...}, rather than a full read/write of a user blob. To derive the current state of the object, you would replay and apply all events to a blank object. Obviously over time the list of changes could become enormous, so what you do to optimize is periodically create a snapshot of the current full state of the object and only replay events that happened after it was made. For our persistence layer we used Redis as the input queue for all events. Each object we stored required a list to push changes into, an integer counter to represent the sequence ID of the most recent event in the list, and a string value containing the location in S3 of the most recent snapshot (if it exists). Performance was incredible for writes because a) it's Redis and b) RedisLabs supports semi-transparent sharding. We used S3 to store the snapshots. We called the creation process a "compaction" because besides storing the most recent full state of the object, we would also store off the list of changes used to make it and purge those events from Redis. This kept our Redis storage usage tightly bounded and our costs down. This compaction process was also 100% safe with no chance of race conditions or data loss. Our algorithm would fetch the list of changes from Redis, create and serialize the current state of the object as a snapshot, and then write it and the list of changes to S3. Only upon success of those S3 writes would it then issue a command to Redis to purge that object's list of changes. To prevent race conditions on this part of the operation, we would send the sequence ID we are compacting from and in a Redis script we would atomically check to make sure no more-recent compaction has happened, update the snapshot pointer, and flush the list of changes only up to the sequence number we compacted from (changes were free to be added to the list while a compaction process was in progress). The killer part of this whole setup was that those snapshots and change lists stored in S3 were completely immutable and formed a linked list of the entire history of changes for an object. We could visualize and inspect the entire change history of an object, find bugs and retroactively fix them by changing how an event is applied, and if needed for read performance, cache the S3 snapshots in memcache without any concern for needing to invalidate them. Concurrent writes to an object became less of an issue as well. Instead of needing to do a CAS write or a transaction, you would just push the change you wanted to make to the object, read it back, and verify it was actually applied. No locking needed. The major downside to this architecture is that any querying you need to do will always be weakly consistent and require an external index. We had a small Elastisearch cluster for some of our objects that had fields needing indexing. One last major advantage of this setup was that it was trivial to "clone" production to your local dev machine. Because snapshots were immutable, we set up a tiered database layer that would first try and load snapshots locally, and failing that, would use a read-only IAM user to access S3 to pull down a snapshot from production. All you would then do is load a backup from a RedisLabs database locally and boom, you have an exact copy of production that you can freely modify because any new writes all happen locally. Believe it or not, the most unexpected cost from this architecture was PUT requests to S3. Depending upon the frequency of your compactions it can start to add up. However, if usage declines, that cost also declines with no impact on the availability of your data.

Zaheer 9y ago

FYI DynamoDB in last few weeks added auto-scaling: https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/ https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon... Additionally they somewhat recently added an optional caching layer: https://aws.amazon.com/dynamodb/dax/ https://aws.amazon.com/dynamodb/dax/

hesdeadjim 9y ago
Nice they finally added auto-scaling. It seems to function pretty similarly to the tool we built, with the same (documented) limitations. DAX would have definitely addressed many of our issues, had it been available at the time. Glad they added that.
- azinman2 9y ago
  So does that address all your issues?
  - hesdeadjim 9y ago
    No, for our usage needing to worry about provisioned capacity at all was becoming a deal-breaker. That especially became tedious to estimate and plan for as we had more variability in the size of our values we needed to store.