4 ms·

This is a nice talk to explain what a document database is and how it is supposed to be used. Many think it is about not having a schema, but you could have the

This is a nice talk to explain what a document database is and how it is supposed to be used. Many think it is about not having a schema, but you could have the database enforce a schema in a document Database. Document storage freezes access patterns in a way that creates data locality for preferred data access patterns. The major downside is that you might not know all the access patterns and even if you do new access patterns can arise. Either way, it can be extremely difficult to accommodate unexpected access patterns. I have seen document storage work well for aggregating data from external APIs. The whole point is that you are making preferred access patterns for your data rather than use an external API. And you are probably ready to rebuild your database if your access pattern changes. I would also expect document storage to work well in an architecture of microservices under the conditions that: * The schema is small so there aren't many possible access patterns * Data is shipped to a SQL (warehouse) database for more complex processing anyways.

lliamander 8y ago
> I would also expect document storage to work well in an architecture of microservices under the conditions that: > * The schema is small so there aren't many possible access patterns > * Data is shipped to a SQL (warehouse) database for more complex processing anyways. That's pretty much exactly how I've seen it used. edit: formatting
013a 8y ago
I tend to believe that one of the most powerful things about Dynamo is the streaming capabilities. It means that you can cleanly use Dynamo as the source of truth for your data, reap all of the performance benefits it has, and automatically ship that data to another database for more advanced BI querying. Moreover, you can do this offline; queue up every mutation that hits Dynamo, then process them at a rate that your data warehouse can ingest without worrying about overprovisioning for spikes. I've worked on a team of 8 engineers that spent 6 months building a streaming replication system from MySQL to Redshift. Its crazy how simple this would have been had the originating database been Dynamo; its very possible one engineer could have done it in a fraction of the time. It also seems likely to me that an open source solution might already exist.

lliamander 8y ago

> I would also expect document storage to work well in an architecture of microservices under the conditions that: > * The schema is small so there aren't many possible access patterns > * Data is shipped to a SQL (warehouse) database for more complex processing anyways. That's pretty much exactly how I've seen it used. edit: formatting

013a 8y ago

I tend to believe that one of the most powerful things about Dynamo is the streaming capabilities. It means that you can cleanly use Dynamo as the source of truth for your data, reap all of the performance benefits it has, and automatically ship that data to another database for more advanced BI querying. Moreover, you can do this offline; queue up every mutation that hits Dynamo, then process them at a rate that your data warehouse can ingest without worrying about overprovisioning for spikes. I've worked on a team of 8 engineers that spent 6 months building a streaming replication system from MySQL to Redshift. Its crazy how simple this would have been had the originating database been Dynamo; its very possible one engineer could have done it in a fraction of the time. It also seems likely to me that an open source solution might already exist.