5 ms·

I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder

I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder to set up than traditional RDBMSes but I can only assume that this gap will narrow and at some point in the future things will be "scalable by default". (Of course no DB is going to prevent all ways to shoot yourself in the foot)

WJW 6y ago
This question is similar to asking on a car forum when using a 40 foot lorry will be the default starter car for everyone. The answer is "probably never" because while it does offer superior cargo transport scalability, the tradeoffs are not worth it for the vast majority of users. The question is posed like distributed databases have no disadvantages over non-distributed databases, but that is simply not the case. Clustering introduces all sort of problems, from network splits to slow(er) joins across nodes.
- __s 6y ago
  To back up how far one server can go, Stack Overflow used a single database for a long time https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... (2013 post had much less redundancy, but even their 2016 architecture is pretty undistributed in terms of being able to recreate everything from the single source of truth database)
  - bcrosby95 6y ago
    Yeah, it's been a while since I've been in a high traffic situation, but back in the late 00s we had a couple sites with 3-5 million daily unique users. Each site had a single master, a couple read replicas, and some queries were cached in memcached. The problem in MySQL we eventually ran into was the read replicas fell behind master during peak load (by 30-60 minutes depending upon the day). So we still had to hit the master for certain queries. I left before we took the next step to fix this issue.
- kevincox 6y ago
  I don't think it is quite the same. - Switching from a car to a van to a lorry is fairly low cost. You don't need to recreate your product (probably). - You don't need to run distributed databases in a cluster to start. But I think most importantly the decrease in dev speed and performance is an investment in future scalability. And I only imagine that this different will shrink over time to where for example a 1 node "distributed" DB isn't that different to work with than a 1 node "traditional" DB. And that small difference pays off because adding a second node doesn't have huge tradeoffs. I agree that right now it doesn't make sense. If I was starting a new product I would still fire up a PostgreSQL instance. However I think that the day will come where the difference is small enough and software support is complete enough that we will start with something distributed, much like people don't often start with SQLite today, even though it is quicker to get going (also not a great comparison because it has less features).
  - yuribro 6y ago
    > But I think most importantly the decrease in dev speed and performance is an investment in future scalability. It makes sense only if you'll ever need this scalability. And you take a hit on other fronts too: Infra cost, Deployment complexity. And both deployment complexity and code complexity also increase QA cost, instability, product and company reputation. >> much like people don't often start with SQLite today Maybe they should start with SQLite by default
    - lrem 6y ago
      For personal projects, I start with SQLite by default. Never came close to its scalability limit too ;)
  - WJW 6y ago
    Making investments in future scalability at the cost of dev speed and performance is exactly the wrong tradeoff given that 90% of startups fail. At the start, when you have very little income, you want to allow for as much speed and flexibility as possible so that you can get to product/market fit ASAP. By the time your company gets big enough that plain MySQL/Postgres can't handle the load any longer, you will have more than enough money to afford a few experts that can help you migrate.
cuu508 6y ago
To me it looks like there are not many affordable options right now. CockroachDB understandably wants you to use their Cloud or Enterprise products: the OSS version is quite limited. For example it doesn't support row-level partitioning (https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html https://www.cockroachlabs.com/docs/stable/configure-replicat...). Which means, if I understand correctly, it is not of much help for scaling writes to a single big table.
- kevincox 6y ago
  Oh, I didn't realize that the free version was that limited. I guess I need another name to use as the default open source distributed database.
- berns 6y ago
  No, you didn't understand correctly. The feature that isn't supported is row level replication zones. Replication zones allows to define the location of replicas.
- nvanbenschoten 6y ago
  Hi cuu508, CockroachDB engineer here. You are correct that row-level partitioning is not supported in the OSS version of CRDB. However, it sounds like there's a bit of confusion about where manual table partitioning is and is not needed. The primary use-case for row-level partitioning is to control the geographic location of various data in a multi-region cluster. Imagine a "users" table where EU users are stored on European servers and NA users are stored on North American servers. If you are only looking to scale write throughput then manual partitioning is not be needed. This is because CRDB transparently performs range partitioning under-the-hood on all tables, so all tables scale in response to data size and load automatically. If you are interested in learning more, https://www.cockroachlabs.com/docs/stable/architecture/distribution-layer.html https://www.cockroachlabs.com/docs/stable/architecture/distr... discusses these concepts in depth.
  - cuu508 6y ago
    Thanks for explaining, and sorry -- looks like I jumped to conclusions too quickly!
kryptiskt 6y ago
Why would it ever be default? Very few applications need a distributed database. These days you can get a single machine with hundreds of terabytes of storage and terabytes of RAM. Add a spare machine for failover and you have a reliable setup for any but the biggest tasks. And the tasks that that setup isn't sufficient for will certainly demand more thought than a cookie-cutter setup of some random distributed DB.
jandrewrogers 6y ago
What we need is a database that can both scale-up and scale-out. Most distributed databases offer poor efficiency and performance on a per node basis, which has a high operational cost. This is why people avoid using distributed databases unless they need it. A scale-up database can serve as much workload as pretty large scale-out database in practice. This discontinuity creates the market for scale-up systems. There is literally nothing preventing distributed databases from having excellent scale-up performance too. Unfortunately, people who design distributed databases have a strong bias toward unnecessarily throwing hardware at performance and scalability problems. This is partly because very few people know how to design a modern scale-up database; making something (trivially) distributed is easier.
- mulmen 6y ago
  If nothing prevents scaling up a scale-out database does that not also mean nothing prevents scaling out a scale-up database? Aren't all scaling problems trivial at that point?
andreimatei1 6y ago
I think the "default" will evolve with whatever offers the best "serverless" experience in the public clouds. In particular, the cheapest and most granularly-billed option.

WJW 6y ago

This question is similar to asking on a car forum when using a 40 foot lorry will be the default starter car for everyone. The answer is "probably never" because while it does offer superior cargo transport scalability, the tradeoffs are not worth it for the vast majority of users. The question is posed like distributed databases have no disadvantages over non-distributed databases, but that is simply not the case. Clustering introduces all sort of problems, from network splits to slow(er) joins across nodes.

__s 6y ago
To back up how far one server can go, Stack Overflow used a single database for a long time https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... (2013 post had much less redundancy, but even their 2016 architecture is pretty undistributed in terms of being able to recreate everything from the single source of truth database)
- bcrosby95 6y ago
  Yeah, it's been a while since I've been in a high traffic situation, but back in the late 00s we had a couple sites with 3-5 million daily unique users. Each site had a single master, a couple read replicas, and some queries were cached in memcached. The problem in MySQL we eventually ran into was the read replicas fell behind master during peak load (by 30-60 minutes depending upon the day). So we still had to hit the master for certain queries. I left before we took the next step to fix this issue.
kevincox 6y ago
I don't think it is quite the same. - Switching from a car to a van to a lorry is fairly low cost. You don't need to recreate your product (probably). - You don't need to run distributed databases in a cluster to start. But I think most importantly the decrease in dev speed and performance is an investment in future scalability. And I only imagine that this different will shrink over time to where for example a 1 node "distributed" DB isn't that different to work with than a 1 node "traditional" DB. And that small difference pays off because adding a second node doesn't have huge tradeoffs. I agree that right now it doesn't make sense. If I was starting a new product I would still fire up a PostgreSQL instance. However I think that the day will come where the difference is small enough and software support is complete enough that we will start with something distributed, much like people don't often start with SQLite today, even though it is quicker to get going (also not a great comparison because it has less features).
- yuribro 6y ago
  > But I think most importantly the decrease in dev speed and performance is an investment in future scalability. It makes sense only if you'll ever need this scalability. And you take a hit on other fronts too: Infra cost, Deployment complexity. And both deployment complexity and code complexity also increase QA cost, instability, product and company reputation. >> much like people don't often start with SQLite today Maybe they should start with SQLite by default
  - lrem 6y ago
    For personal projects, I start with SQLite by default. Never came close to its scalability limit too ;)
- WJW 6y ago
  Making investments in future scalability at the cost of dev speed and performance is exactly the wrong tradeoff given that 90% of startups fail. At the start, when you have very little income, you want to allow for as much speed and flexibility as possible so that you can get to product/market fit ASAP. By the time your company gets big enough that plain MySQL/Postgres can't handle the load any longer, you will have more than enough money to afford a few experts that can help you migrate.

cuu508 6y ago

To me it looks like there are not many affordable options right now. CockroachDB understandably wants you to use their Cloud or Enterprise products: the OSS version is quite limited. For example it doesn't support row-level partitioning (https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html https://www.cockroachlabs.com/docs/stable/configure-replicat...). Which means, if I understand correctly, it is not of much help for scaling writes to a single big table.

kevincox 6y ago
Oh, I didn't realize that the free version was that limited. I guess I need another name to use as the default open source distributed database.
berns 6y ago
No, you didn't understand correctly. The feature that isn't supported is row level replication zones. Replication zones allows to define the location of replicas.
nvanbenschoten 6y ago
Hi cuu508, CockroachDB engineer here. You are correct that row-level partitioning is not supported in the OSS version of CRDB. However, it sounds like there's a bit of confusion about where manual table partitioning is and is not needed. The primary use-case for row-level partitioning is to control the geographic location of various data in a multi-region cluster. Imagine a "users" table where EU users are stored on European servers and NA users are stored on North American servers. If you are only looking to scale write throughput then manual partitioning is not be needed. This is because CRDB transparently performs range partitioning under-the-hood on all tables, so all tables scale in response to data size and load automatically. If you are interested in learning more, https://www.cockroachlabs.com/docs/stable/architecture/distribution-layer.html https://www.cockroachlabs.com/docs/stable/architecture/distr... discusses these concepts in depth.
- cuu508 6y ago
  Thanks for explaining, and sorry -- looks like I jumped to conclusions too quickly!

kryptiskt 6y ago

Why would it ever be default? Very few applications need a distributed database. These days you can get a single machine with hundreds of terabytes of storage and terabytes of RAM. Add a spare machine for failover and you have a reliable setup for any but the biggest tasks. And the tasks that that setup isn't sufficient for will certainly demand more thought than a cookie-cutter setup of some random distributed DB.

jandrewrogers 6y ago

What we need is a database that can both scale-up and scale-out. Most distributed databases offer poor efficiency and performance on a per node basis, which has a high operational cost. This is why people avoid using distributed databases unless they need it. A scale-up database can serve as much workload as pretty large scale-out database in practice. This discontinuity creates the market for scale-up systems. There is literally nothing preventing distributed databases from having excellent scale-up performance too. Unfortunately, people who design distributed databases have a strong bias toward unnecessarily throwing hardware at performance and scalability problems. This is partly because very few people know how to design a modern scale-up database; making something (trivially) distributed is easier.

mulmen 6y ago
If nothing prevents scaling up a scale-out database does that not also mean nothing prevents scaling out a scale-up database? Aren't all scaling problems trivial at that point?

andreimatei1 6y ago

I think the "default" will evolve with whatever offers the best "serverless" experience in the public clouds. In particular, the cheapest and most granularly-billed option.