3 ms·

Happy to find this blog, wish there was more talk about NewSQL databases. I've personally seen the amount of cost that goes into migrating off of traditional RD

Happy to find this blog, wish there was more talk about NewSQL databases. I've personally seen the amount of cost that goes into migrating off of traditional RDBMs systems to sharded RDBMs systems and was wondering why people don't go with these scalable systems from the start now that they exist.

likeabbas 4y ago
+1. Recently, Yugabyte has seemed intriguing to me. (It seems like) You can launch it as a standalone instance and use it just like Postgres early in your company, and then when you need to distribute, you can do that.
- brnewd 4y ago
  I think many love the promise of a distributed database, but in practice there is a real speed and feature set compromise to make.
  - likeabbas 4y ago
    I found that having a NoSQL distributed database for prod that stream Change Data Captures to a read-only SQL DB to be a great compromise. The SQL data was slightly stale but that wasn't an issue for most situations (like graphing data or certain error metrics)
pm90 4y ago
Probably because there is a lot more operational understanding of “traditional” RDBMs and they’re sufficient in early stages. Once your product takes off it can be difficult to scale but by that time you probably have enough money to attract engineers that can do the migration.
- friedman23 4y ago
  Yes, that's the common refrain, just migrate when you hit scale. Just pay people to do the migration. My experience at multiple, billion dollar companies has been that it's never that easy. People become afraid of doing schema migrations and tools like dynamodb become the default datastore choice for all new feature development. Every single startup I've worked at, technical cruft that existed from company founding never disappears despite the founders intent that it was never supposed to stay that way and it slows down feature development to a crawl. If NewSQL databases exist that support 90% of the features of postgres but give support for cross shard transactions then I'd rather use them than technology that has been proven to not scale. Unfortunately I'm not an expert on distributed databases and there is not much written about these databases in practice outside of the startups creating the databases themselves.
  - Andys 4y ago
    From my own experimentation, CockroachDB is fantastic and exactly what you just described (90%+ of the features of postgres, but transparently sharded). It is somehow new and shiny and boring old tech all at the same time.

likeabbas 4y ago

+1. Recently, Yugabyte has seemed intriguing to me. (It seems like) You can launch it as a standalone instance and use it just like Postgres early in your company, and then when you need to distribute, you can do that.

brnewd 4y ago
I think many love the promise of a distributed database, but in practice there is a real speed and feature set compromise to make.
- likeabbas 4y ago
  I found that having a NoSQL distributed database for prod that stream Change Data Captures to a read-only SQL DB to be a great compromise. The SQL data was slightly stale but that wasn't an issue for most situations (like graphing data or certain error metrics)

pm90 4y ago

Probably because there is a lot more operational understanding of “traditional” RDBMs and they’re sufficient in early stages. Once your product takes off it can be difficult to scale but by that time you probably have enough money to attract engineers that can do the migration.

friedman23 4y ago
Yes, that's the common refrain, just migrate when you hit scale. Just pay people to do the migration. My experience at multiple, billion dollar companies has been that it's never that easy. People become afraid of doing schema migrations and tools like dynamodb become the default datastore choice for all new feature development. Every single startup I've worked at, technical cruft that existed from company founding never disappears despite the founders intent that it was never supposed to stay that way and it slows down feature development to a crawl. If NewSQL databases exist that support 90% of the features of postgres but give support for cross shard transactions then I'd rather use them than technology that has been proven to not scale. Unfortunately I'm not an expert on distributed databases and there is not much written about these databases in practice outside of the startups creating the databases themselves.
- Andys 4y ago
  From my own experimentation, CockroachDB is fantastic and exactly what you just described (90%+ of the features of postgres, but transparently sharded). It is somehow new and shiny and boring old tech all at the same time.