3 ms·

The difference is that today those DBs are generally considered as complements to SQL databases targeting specific use cases. Back in the peak NoSQL days people

by dagw 3y ago

The difference is that today those DBs are generally considered as complements to SQL databases targeting specific use cases. Back in the peak NoSQL days people were pushing the narrative that you'd never need to use a SQL database for anything ever again.

AtlasBarfed 3y ago
IMO this is the deal: RDBMS is such a mature and powerful technology, and with the vast power in modern single-node hardware, they will scale to considerable sizes. But there is a limit. Once a table or set of tables hit a certain scale, it needs to be distributed. Once you get to those scales, you are likely dealing with the threat of exponential data growth, so doing the "distributed Postgres" will bandaid the problem ... but you are starting to run into the CAP theorem's problems. You'll need to AP-scale the biggest data tables, and since that almost always means a big coding change lift and introduction of an AP-scaling database, it will almost always be a six months to a year transition, and possibly adding an entirely new DB technology (Cassandra / DynamoDB / maybe FoundationDB). I have yet to have anyone explain how joins scale on an AP distributed database except in limited situations where the joined data is somehow node-local to the other tables, usually some hierarchical situation. Otherwise you are pulling data from lots of nodes and aggregating and comparing the different sets to account for node drift / partitions / network failures. Cassandra and Dynamo basically say "you are scaling a single table/query/update/pre-joined data table". Which really isn't fun for RDBMS folks. Because it is a shitload of denormalization on top of all the AP headaches and distributed transactions / updates. As I said, megahuge machines really make that an outside case unless your use case really emphasizes the "A" in CAP, where write speed can be tolerated to be low and you want to tolerate entire cloud or datacenter outages. But yeah, nosql was never going to kill the rdbms. The functionality/power of rdbms/SQL is so so so so much higher. Just be aware when you're about to shoot over the limit and prepare for the code changes to handle true scale in the (unlikely) event you're going to need it.
- dalyons 3y ago
  Great explanation, and I agree with one caveat - from my experience if you get to a truly large single RDBMS you have already lost, even if you have a powerful enough machine to run it on. Massive RDBMS instances are awful to manage. Hard to backup/restore/fork. Hard to migrate tables and schemas without causing downtime. Accidental downtime happens all the time due to locks, bad indexes, bad query plans , etc etc. at large scale they are capricious beasts, care and feeding and most importantly changing them becomes a dark art. Don’t let them get too big you’ll regret it :)
regularfry 3y ago
Oh, they're still at it. In my relatively recent experience Mongo will still try to sell you on the idea of abandoning relational workloads because migrations are hard and schemas are optional.

by dagw 3y ago

AtlasBarfed 3y ago

IMO this is the deal: RDBMS is such a mature and powerful technology, and with the vast power in modern single-node hardware, they will scale to considerable sizes. But there is a limit. Once a table or set of tables hit a certain scale, it needs to be distributed. Once you get to those scales, you are likely dealing with the threat of exponential data growth, so doing the "distributed Postgres" will bandaid the problem ... but you are starting to run into the CAP theorem's problems. You'll need to AP-scale the biggest data tables, and since that almost always means a big coding change lift and introduction of an AP-scaling database, it will almost always be a six months to a year transition, and possibly adding an entirely new DB technology (Cassandra / DynamoDB / maybe FoundationDB). I have yet to have anyone explain how joins scale on an AP distributed database except in limited situations where the joined data is somehow node-local to the other tables, usually some hierarchical situation. Otherwise you are pulling data from lots of nodes and aggregating and comparing the different sets to account for node drift / partitions / network failures. Cassandra and Dynamo basically say "you are scaling a single table/query/update/pre-joined data table". Which really isn't fun for RDBMS folks. Because it is a shitload of denormalization on top of all the AP headaches and distributed transactions / updates. As I said, megahuge machines really make that an outside case unless your use case really emphasizes the "A" in CAP, where write speed can be tolerated to be low and you want to tolerate entire cloud or datacenter outages. But yeah, nosql was never going to kill the rdbms. The functionality/power of rdbms/SQL is so so so so much higher. Just be aware when you're about to shoot over the limit and prepare for the code changes to handle true scale in the (unlikely) event you're going to need it.

dalyons 3y ago
Great explanation, and I agree with one caveat - from my experience if you get to a truly large single RDBMS you have already lost, even if you have a powerful enough machine to run it on. Massive RDBMS instances are awful to manage. Hard to backup/restore/fork. Hard to migrate tables and schemas without causing downtime. Accidental downtime happens all the time due to locks, bad indexes, bad query plans , etc etc. at large scale they are capricious beasts, care and feeding and most importantly changing them becomes a dark art. Don’t let them get too big you’ll regret it :)

regularfry 3y ago

Oh, they're still at it. In my relatively recent experience Mongo will still try to sell you on the idea of abandoning relational workloads because migrations are hard and schemas are optional.