3 ms·

Data locality is not about seek times on disk, it's about network transfer times between different nodes! Linear horizontal scalability often needs sharding of

Data locality is not about seek times on disk, it's about network transfer times between different nodes! Linear horizontal scalability often needs sharding of data, and with a document store like MongoDB, you can easily shard a complex denormalized document. Now, if I have this complex document normalized into several tables, how am I going to easily shard several tables, all such that I can execute successful joins that only need to execute on one leaf node? What if I start reusing a small piece of data in one of these normalized tables? I might be forced to go between network nodes to get this data. Normalization is like premature optimization. I can take any program and modify it such that every piece executes optimally fast, but I am likely to compromise on clarity or to add complexity while doing such a refactoring. In the end, it probably got me no real-world performance boost that mattered. 80/20% rule and all. Same thing with normalization: automatically making all my data fully normalized from the start is a like a bad premature optimization habit that we are forced into with relational databases out of A) sheer habit and school teachings, B) lack of easy support for nested structured data.

gaius 13y ago
That is of course what I was referring to when I talked about logical vs physical data models. You define your LDM then selectively denormalize it when you create the PDM. The MySQL guys are very into sharding too but of course without hash joins, you can't join big tables anyway, so this "optimization" is easy because it costs you nothing.
wheaties 13y ago
You can do all of that with a SQL DB. In fact, Postgres works as an amazing key-value store if you want it to. They've even got some damned good performance in hstore compared to many use cases of Mongo. I use MongoDB and SQL DBs every day.
- astral303 13y ago
  Mongo is much more than a key-value store. Document sub-components and subarrays can be indexed and aggregated against. I don't see how you can do that with hstore. Yes, I can totally see how Postgres and hstore can run circles around key-value storage. And by "all that", I would love to know how to use the open-source versions of Postgres or MySQL to have transparently-sharded tables with joins AND have that be in a replicated environment where I can do real-time failover.
  - fusiongyro 13y ago
    I think the partitioning and replication stories with Postgres will have to get a lot less manual before there can be automatic sharding. It doesn't sound beyond the pale to me, but I wouldn't expect it in the next five years.

gaius 13y ago

That is of course what I was referring to when I talked about logical vs physical data models. You define your LDM then selectively denormalize it when you create the PDM. The MySQL guys are very into sharding too but of course without hash joins, you can't join big tables anyway, so this "optimization" is easy because it costs you nothing.

wheaties 13y ago

You can do all of that with a SQL DB. In fact, Postgres works as an amazing key-value store if you want it to. They've even got some damned good performance in hstore compared to many use cases of Mongo. I use MongoDB and SQL DBs every day.

astral303 13y ago
Mongo is much more than a key-value store. Document sub-components and subarrays can be indexed and aggregated against. I don't see how you can do that with hstore. Yes, I can totally see how Postgres and hstore can run circles around key-value storage. And by "all that", I would love to know how to use the open-source versions of Postgres or MySQL to have transparently-sharded tables with joins AND have that be in a replicated environment where I can do real-time failover.
- fusiongyro 13y ago
  I think the partitioning and replication stories with Postgres will have to get a lot less manual before there can be automatic sharding. It doesn't sound beyond the pale to me, but I wouldn't expect it in the next five years.