4 ms·

While I agree that this is a wonderfully simple case the main problem is that it isn't always that easy. As long as you get get away slicing tables into differe

While I agree that this is a wonderfully simple case the main problem is that it isn't always that easy. As long as you get get away slicing tables into different "shards" and aren't to join-heavy it isn't too baf. However it still adds complexity to the application code that is a constant maintenance cost. And eventually if you keep scaling you will get to the point where you need to shard a single table, and that is very painful. Both in the risk and difficulty of the conversion and further ongoing development costs. I do dream of the day when distributed databases are the "default" for new projects. Likely running just a single instance at the beginning. But then you have a seamless path forward when (and if) you need it. Even at medium size three small nodes will be easy to manage and give easy HA and allow you to do version updates safely and with no downtime. I don't think we are there yet, but there are a few contenders in the running but Postgres is tough competition. Its years of stability and predictability give it huge points even if it has the downsides of a centralized system. But I think the turning point is steadily approaching.

herrkanin 4y ago
Which databases are the main contenders, that you were referring to?
- kevincox 4y ago
  Disclaimer: I now work at https://www.yugabyte.com/yugabytedb/ https://www.yugabyte.com/yugabytedb/, I joined because I liked their approach and think it has a lot of merit. But there are also https://www.cockroachlabs.com/product/ https://www.cockroachlabs.com/product/, https://github.com/pingcap/tidb https://github.com/pingcap/tidb and https://ydb.tech/ https://ydb.tech/ which each has promise. There is definitely lots of progress to make in this space. From what I have seen they are all significantly slower on a single node and are not as battle-hardened. But I think with time we will have a few really nice options to pick from.
  - Gwypaas 4y ago
    You also have FoundationDB, which seems quite battle hardened with Snowflake's, Apple's and Datadog's adoption. Only provides a quite barebones Key Value interface to play around with though, so not anywhere near a drop in replacement to your traditional SQL database. https://www.foundationdb.org/ https://www.foundationdb.org/
    - kevincox 4y ago
      I didn't mention FoundationDB since it isn't relational. I think the relational model with automatic indexes and query optimization is a huge benefit. That being said of all of the options mentioned it does seem like the most battle-tested. I would love to see a relational layer built on top of it and see how that turns out.
      - kennywinker 4y ago
        I believe CloudKit is built on FoundationDB > CloudKit allows you to store your data as CKRecord objects, and relationships between those objects as CKRecord.Reference associations. https://developer.apple.com/documentation/cloudkit/designing_and_creating_a_cloudkit_database https://developer.apple.com/documentation/cloudkit/designing... I’m not a database guy so idk if that meets the technical definition of “relational” but it is a database with relationships :)
- genousti 4y ago
  Aurora, alloydb, spanner
- zbentley 4y ago
  Clickhouse is nice, though it does require you to give up some pretty significant capabilities shared by most traditional SQL DBs.
int0x2e 4y ago
I am not sure your utopia is quite as utopian as you think. I've been working on an internal service at a major tech company for a few years now. The project is awesome and super fun, the teams is great, etc. - but the overall size of the data and query load for key "working set" data store would easily be served by an RDBMS including (all likely future growth for 5+ years), yet we went for a (managed) distributed document store. We are paying more money and dealing with all the complexity and limitations of a NoSQL document store, while we could have just used great ORM/SQL and enjoyed those sweet, sweet transactions and joins and the wealth of tooling that exists for mature RDBMS solutions. Sometimes, it would be great to simply choose the right tool for the job, even if it isn't sexy and doesn't help anyone add "distributed NoSQL database" to their CV...
- kevincox 4y ago
  I never said NoSQL. I think the relational model is incredibly valuable. I'm talking about updating the implementation, not the interface. However there are some new databases that maintain a relational model (often fully compatible with PostrgeSQL or other relational DBs). The only real downside to these right now is stability and performance. Performance isn't a big deal when you can scale horizontally so I think when this "NewSQL" generation matures it will be nearly all upsides. I'm not sure we will be stuck with SQL forever but strong consistency, SQL and the database-backed constraints and indexes are invaluable and I wouldn't want to be stuck without them.

herrkanin 4y ago

Which databases are the main contenders, that you were referring to?

kevincox 4y ago
Disclaimer: I now work at https://www.yugabyte.com/yugabytedb/ https://www.yugabyte.com/yugabytedb/, I joined because I liked their approach and think it has a lot of merit. But there are also https://www.cockroachlabs.com/product/ https://www.cockroachlabs.com/product/, https://github.com/pingcap/tidb https://github.com/pingcap/tidb and https://ydb.tech/ https://ydb.tech/ which each has promise. There is definitely lots of progress to make in this space. From what I have seen they are all significantly slower on a single node and are not as battle-hardened. But I think with time we will have a few really nice options to pick from.
- Gwypaas 4y ago
  You also have FoundationDB, which seems quite battle hardened with Snowflake's, Apple's and Datadog's adoption. Only provides a quite barebones Key Value interface to play around with though, so not anywhere near a drop in replacement to your traditional SQL database. https://www.foundationdb.org/ https://www.foundationdb.org/
  - kevincox 4y ago
    I didn't mention FoundationDB since it isn't relational. I think the relational model with automatic indexes and query optimization is a huge benefit. That being said of all of the options mentioned it does seem like the most battle-tested. I would love to see a relational layer built on top of it and see how that turns out.
    - kennywinker 4y ago
      I believe CloudKit is built on FoundationDB > CloudKit allows you to store your data as CKRecord objects, and relationships between those objects as CKRecord.Reference associations. https://developer.apple.com/documentation/cloudkit/designing_and_creating_a_cloudkit_database https://developer.apple.com/documentation/cloudkit/designing... I’m not a database guy so idk if that meets the technical definition of “relational” but it is a database with relationships :)
genousti 4y ago
Aurora, alloydb, spanner
zbentley 4y ago
Clickhouse is nice, though it does require you to give up some pretty significant capabilities shared by most traditional SQL DBs.

int0x2e 4y ago

I am not sure your utopia is quite as utopian as you think. I've been working on an internal service at a major tech company for a few years now. The project is awesome and super fun, the teams is great, etc. - but the overall size of the data and query load for key "working set" data store would easily be served by an RDBMS including (all likely future growth for 5+ years), yet we went for a (managed) distributed document store. We are paying more money and dealing with all the complexity and limitations of a NoSQL document store, while we could have just used great ORM/SQL and enjoyed those sweet, sweet transactions and joins and the wealth of tooling that exists for mature RDBMS solutions. Sometimes, it would be great to simply choose the right tool for the job, even if it isn't sexy and doesn't help anyone add "distributed NoSQL database" to their CV...

kevincox 4y ago
I never said NoSQL. I think the relational model is incredibly valuable. I'm talking about updating the implementation, not the interface. However there are some new databases that maintain a relational model (often fully compatible with PostrgeSQL or other relational DBs). The only real downside to these right now is stability and performance. Performance isn't a big deal when you can scale horizontally so I think when this "NewSQL" generation matures it will be nearly all upsides. I'm not sure we will be stuck with SQL forever but strong consistency, SQL and the database-backed constraints and indexes are invaluable and I wouldn't want to be stuck without them.