3 ms·

DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support be

DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support before going with it, as it’s likely to be a dealbreaker for many use cases, whether or not you realize that up front. I think most apps will need a transaction involving more than 25 rows at some point, and with dynamo your only option is to fire them off in groups of 25 and hope none fail (plenty will at scale). You can get many of the benefits of dynamo (sans auto-sharding), by applying its elegant indexing strategy to an sql database. It will be as fast or faster, your transactions can be as big as you need them to be, and you retain the ability to occasionally fire off un-indexed ad hoc queries for development or convenience. Running and scaling an sql db is also fairly painless these days with options like aurora.

parsnips 6y ago
Agreed. This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. The transaction model we came up with is formally verified w/ TLA+. We're turning our work into a product: txlayer.com
- cbdumas 6y ago
  > This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. Sounds like a perfect use case for a traditional RDBMS. Why Dynamo?
  - parsnips 6y ago
    A fair question, and we’ve done it this way before at simple.com. When looking at the options for a pay-per use database, with global replication, streams and managed for you; We felt that if you could build a ledger on dynamo for these use cases, it’d be pretty compelling and fun.
cactus2093 6y ago
Interesting, idk that I've ever needed a transaction with more than 25 rows. But I agree in general about the limitations. Having used RDBMSes like Postgres a lot, as well used Cassandra and DynamoDB in production, I would almost certainly not create a new app with DynamoDB as the primary DB. Even if you have an app where you expect to need to scale writes heavily, it's not going to be on all tables equally. For instance, your users table, and related resources that are relatively small and grow linearly with your users, will probably fit fine in a Postgres DB for a very long time. And being able to have normalized models and powerful indexing and querying patterns available is a big benefit. DynamoDB can work well for a specific sub-system that needs very high scalability. For instance, if you needed to store pairwise info between every user and product combination for some reason. Of if every user can upload a huge number of resources of some type (though the access patterns need to fit dynamodb's constraints, if these are documents or files of some type then another system like S3 or Elasticsearch would probably make more sense). Or if you're tracking advertising views by an advertising identifier or something. Or scraping and importing a bunch of data from other places. In some specific use-cases like this, the downsides vs an RDMS can be very minimal, and the built-in scalability can save you a ton of time vs having to constantly tune and potentially shard your RDBMS system. But even in these cases, you might have better options depending on your access patterns. For instance if you don't ever need to refer to this data by reading it in an OLTP context, you might want to just write it to a log like Kafka to be ingested into Redshift or HDFS for offline processing or querying.
- abd12 6y ago
  I can understand the sentiment and don't fault you for it. That said, I think you can definitely handle complex, relational patterns in DynamoDB pretty easily. It will take some work to learn new modeling patterns, but it's absolutely doable.
  - ignoramous 6y ago
    At what cost, though? RDBMS do provide a better querying interface with fewer pitfalls that invariably one encounters trying to fit a square peg in a round hole. That said, I've seen people use DynamoDB as a timeseries database (modelling multi-dimensonal data with z-indices on top of the two dimensional partition-key and range-key pair), so it is definitely possible to be clever and practical at the same time. Disclaimer: ex-AWS.
zemo 6y ago
> I think most apps will need a transaction involving more than 25 rows at some point I ... can't think of a single time I've ever needed this.
- danenania 6y ago
  A common one is cascading deletes when you delete a user or a ‘project’ or something else that has a lot of stuff associated with it. Those will exceed 25 rows very quickly. Also any kind of bulk update or data import... hell even just initializing a new account can easily require writing more than 25 rows in a moderately complex app.
  - the_reformation 6y ago
    Still, why the need to be transactional? An eventually consistent delete seems fine here.
    - danenania 6y ago
      Maybe it is, or maybe some of the rows are involved in a security check. Or could cause race conditions if out of sync. Or otherwise need immediate read-after-write consistency. And eventually consistent isn't the worst case scenario. Being unable to rollback correctly from an error could mean you'll never end up in a consistent state... that's a lot worse than "eventually".
- kulig 6y ago
  Youve never worked on a minimally complex enterprise application?

parsnips 6y ago

Agreed. This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. The transaction model we came up with is formally verified w/ TLA+. We're turning our work into a product: txlayer.com

cbdumas 6y ago
> This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. Sounds like a perfect use case for a traditional RDBMS. Why Dynamo?
- parsnips 6y ago
  A fair question, and we’ve done it this way before at simple.com. When looking at the options for a pay-per use database, with global replication, streams and managed for you; We felt that if you could build a ledger on dynamo for these use cases, it’d be pretty compelling and fun.

cactus2093 6y ago

Interesting, idk that I've ever needed a transaction with more than 25 rows. But I agree in general about the limitations. Having used RDBMSes like Postgres a lot, as well used Cassandra and DynamoDB in production, I would almost certainly not create a new app with DynamoDB as the primary DB. Even if you have an app where you expect to need to scale writes heavily, it's not going to be on all tables equally. For instance, your users table, and related resources that are relatively small and grow linearly with your users, will probably fit fine in a Postgres DB for a very long time. And being able to have normalized models and powerful indexing and querying patterns available is a big benefit. DynamoDB can work well for a specific sub-system that needs very high scalability. For instance, if you needed to store pairwise info between every user and product combination for some reason. Of if every user can upload a huge number of resources of some type (though the access patterns need to fit dynamodb's constraints, if these are documents or files of some type then another system like S3 or Elasticsearch would probably make more sense). Or if you're tracking advertising views by an advertising identifier or something. Or scraping and importing a bunch of data from other places. In some specific use-cases like this, the downsides vs an RDMS can be very minimal, and the built-in scalability can save you a ton of time vs having to constantly tune and potentially shard your RDBMS system. But even in these cases, you might have better options depending on your access patterns. For instance if you don't ever need to refer to this data by reading it in an OLTP context, you might want to just write it to a log like Kafka to be ingested into Redshift or HDFS for offline processing or querying.

abd12 6y ago
I can understand the sentiment and don't fault you for it. That said, I think you can definitely handle complex, relational patterns in DynamoDB pretty easily. It will take some work to learn new modeling patterns, but it's absolutely doable.
- ignoramous 6y ago
  At what cost, though? RDBMS do provide a better querying interface with fewer pitfalls that invariably one encounters trying to fit a square peg in a round hole. That said, I've seen people use DynamoDB as a timeseries database (modelling multi-dimensonal data with z-indices on top of the two dimensional partition-key and range-key pair), so it is definitely possible to be clever and practical at the same time. Disclaimer: ex-AWS.

zemo 6y ago

> I think most apps will need a transaction involving more than 25 rows at some point I ... can't think of a single time I've ever needed this.

danenania 6y ago
A common one is cascading deletes when you delete a user or a ‘project’ or something else that has a lot of stuff associated with it. Those will exceed 25 rows very quickly. Also any kind of bulk update or data import... hell even just initializing a new account can easily require writing more than 25 rows in a moderately complex app.
- the_reformation 6y ago
  Still, why the need to be transactional? An eventually consistent delete seems fine here.
  - danenania 6y ago
    Maybe it is, or maybe some of the rows are involved in a security check. Or could cause race conditions if out of sync. Or otherwise need immediate read-after-write consistency. And eventually consistent isn't the worst case scenario. Being unable to rollback correctly from an error could mean you'll never end up in a consistent state... that's a lot worse than "eventually".
kulig 6y ago
Youve never worked on a minimally complex enterprise application?