7 ms·

I am genuinely interested in using CockroachDB as a primary datastore but feel like I have been burned too many times by hopping on board with a young database.

I am genuinely interested in using CockroachDB as a primary datastore but feel like I have been burned too many times by hopping on board with a young database. I tried lucene based databases that offered amazing search capability but were riddled with data corruption issues. Then there was RethinkDB which was very promising but ran out of funding. I am skeptical that a networked database with multiple nodes can match the performance of a single master database such as MySQL, PostgreSQL, or SQL Server. I did a quick benchmark of CockroachDB 1.x and MySQL last year and found that CockroachDB was 5-10x slower on simple CRUD queries: https://github.com/caleblloyd/MySqlCockroachBench/wiki/Concurrency-Benchmark-Results https://github.com/caleblloyd/MySqlCockroachBench/wiki/Concu... Are there any good independent benchmarks of performance? According to crunchbase, Cockroach Labs has raised $53.5m (RethinkDB had only raised $12m). Is there evidence that Cockroach Labs is on track to make money and is going to survive? If the company is healthy and the performance is verified as close to a major RDBMS, I would be comfortable trying it out as a primary datastore. If those questions can't be definitely answered right now, I'll probably continue to wait for the company and the tech to mature.

amelius 9y ago
When it comes to databases, always wait for the Jepsen test! https://jepsen.io/ https://jepsen.io/
- jordanlewis 9y ago
  CockroachDB underwent a Jepsen audit by Kyle Kingsbury before its 1.0 release. Check out the blog post[0] and Kyle's analysis[1]. We also run Jepsen tests against the database nightly, ensuring that its isolation guarantees do not regress. [0]: https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/ https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... [1]: http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829
- michaelmior 9y ago
  Which they've been doing for around 2 years at least https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockroachdb/ https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...
zellyn 9y ago
You are 100% correct about waiting for several years before using a new database product. However, if you look at the path Cockroach is taking, they're doing all the right things to shave those years down. They already Jepsen themselves, and they've been doing rigorous testing forever now. Very exciting.
- nkohari 9y ago
  So did RethinkDB. :(
  - zellyn 9y ago
    Touché
  - forgot-my-pw 9y ago
    It's still alive as an open source project, but probably less support now. https://github.com/rethinkdb/rethinkdb https://github.com/rethinkdb/rethinkdb
    - manigandham 9y ago
      And still available as a managed service on Compose: https://www.compose.com/databases/rethinkdb https://www.compose.com/databases/rethinkdb
    - welder 9y ago
      RethinkDB still has a 2 year old critical memory leak that probably won't ever be fixed: https://github.com/rethinkdb/rethinkdb/issues/5865 https://github.com/rethinkdb/rethinkdb/issues/5865
manigandham 9y ago
A distributed database will never be faster than a single-node RDBMS for simple queries because there is added overhead of coordination between nodes, especially if you're repeatedly writing to the same exact rows. What distributed databases give you is scalability as data grows, and high-availability for safety. Performance comes from that scale and concurrency when your data and queries fit the model.
- d4l3k 9y ago
  There's definitely cases where a distributed database will have higher bandwidth than a single-node database. Since each range is replicated across 3 machines, that means if you have a lot more nodes than 3, the overlap of ranges significantly decreases and you start being able to handle more requests than a single node can. However, the latency will typically be higher since it always has to talk to at least 3 nodes. If you consider CRDB's DistSQL which lets you run queries on multiple machines, it might be possible to beat a traditional RDBMS on read latency for large queries.
- truth_seeker 9y ago
  Not entirely correct. The distributed database can scale computation horizontally across the nodes in cluster. Single node is better only for the relatively small dataset. If you go higher volumes and velocity, Vertical scaling is still more expensive than Horizontal scaling. If your data is large, with network bandwidth of 1Gbps (or perhaps better) distributed database may choose to replicate it efficiently across the nodes and do scatter-gather (or divide and conquer) query parallelization and also maintain HA. Citus DB is one example of distributed database which does exactly that.
  - manigandham 9y ago
    That’s basically what I said...the point is that simple queries on the same keys will never be faster.
- caleblloyd 9y ago
  It would be nice if CRDB could have a mode when 1-3 nodes are active that is similar to since master performance with automatic replication. Then if horizontal scale-out is needed a performance hit could be taken. I like the automatic replication of CRDB but the vast majority of apps can't afford to take a big performance hit and don't need large horizontal scale-out.
darksaints 9y ago
> I am skeptical that a networked database with multiple nodes can match the performance of a single master database such as MySQL, PostgreSQL, or SQL Server. It likely can't unless there is some black magic going on. Single node speed will always be faster. But once you get to that point where a single node chokes on the amount of data or query throughput that you have, you don't really have a choice anymore. My personal plan is to start with Postgres RDS. Grow until RDS doesn't work anymore, then move to ultra beefy bare metal servers in colocation with AWS Direct Connect. If I ever outgrow an 4x24 core server with 3TB of memory on a RAIDed NVMe disk cluster, I might move to Citus. For the various distributed database companies out there, I believe the one that will win in the marketplace is the one working or partnering to develop specialized hardware and networking, and then optimizing for it.
- p0rkbelly 9y ago
  Why not bare metal on AWS? Going to DX is going to give you a conservative 10ms on your calls...
  - darksaints 9y ago
    That might be a good intermediate option, but they still don't have instances with 4x24 cpu, or >488GB ram.
    - openasocket 9y ago
      Did you look at the x1 options? Up to 128 CPUs and 3,904GB of RAM on the x1e.32xlarge.
      - darksaints 9y ago
        Those aren't bare metal. And in terms of database scalability, I would jump for bare metal before beefier servers, due to the fact that the disk is local and you don't have to deal with EBS noisy neighbors.
zzzcpan 9y ago
As others said, Cockroach's performance is limited by its consistency model. It is not something that can ever get close to a single master MySQL. This might also become an issue for the company and ultimately prevent it from reaching mass adoption and succeeding, at least commercially.
tnolet 9y ago
Do you need a distributed database for your use case? Or do you like the new tech? Genuine question, not trying to be snarky.
- caleblloyd 9y ago
  I do not have a particular project in mind, but think it would be good to have an easily containerizable, replicated OLTP database in my toolkit. I do some work with Entity Framework Core providers and am thinking about trying to implement a provider for CRDB.
  - MycroftH 9y ago
    There is already some work towards this with http://www.npgsql.org/ http://www.npgsql.org/ I haven't gone back to try the full entity framework recently, but if you find any bugs, please let us know. If you want to roll your own, we're here to help too.
  - manigandham 9y ago
    CRDB is using the Postgresql wire transport and already works with npgsql + ef core.
    - caleblloyd 9y ago
      Are you sure? I know that the NPGSQL DB driver works with CRDB. Data types, DDL, and supported features are a big part of EF Core providers though. I have not researched it yet but I was pretty sure the NPGSQL EF Core Provider implemetation was just for PG.
      - manigandham 9y ago
        CockroachDB uses the same PostgreSQL wire protocol and SQL dialect so it just looks and works like a pg database. There are some minor differences but DDL and data types are already included in the syntax: https://www.cockroachlabs.com/docs/stable/porting-postgres.html https://www.cockroachlabs.com/docs/stable/porting-postgres.h... It's great if you want to contribute though but I'd recommend just working on the PG EF Core provider which already has some work towards CRDB specific features: https://github.com/npgsql/Npgsql.EntityFrameworkCore.PostgreSQL/issues?utf8=%E2%9C%93&q=is%3Aissue+cockroachdb https://github.com/npgsql/Npgsql.EntityFrameworkCore.Postgre...
gred 9y ago
> I am genuinely interested in using CockroachDB as a primary datastore but feel like I have been burned too many times by hopping on board with a young database. > According to crunchbase, Cockroach Labs has raised $53.5m (RethinkDB had only raised $12m). Is there evidence that Cockroach Labs is on track to make money and is going to survive? I'm in a similar situation: new project, CRDB is a good fit on paper, but I'm unable to get traction even for a proof of concept exercise because of the risk associated with a database backed by a startup. This is at a large, conservative, 300k+ employee company, so I guess it's understandable... but regrettable nonetheless. See you guys in 5 years ;-)
- manigandham 9y ago
  Becoming a paying customer is a way to help them survive.
hnkimb3558 9y ago
Performance is a very tricky thing to measure in a database. CockroachDB's performance is certainly affected by its consistency model. In particular, CockroachDB handles transactions using serializable isolation, and writes using consensus replication. This means that it's difficult to make an apples to apples comparison for performance between a replicated CockroachDB cluster and a single master MySQL server. However, that is the whole point of a well thought out, standardized benchmark like TPC-C. It creates a pseudo-realistic workload, and mandates the use of transactions with a reasonable isolation level or higher (snapshot isolation), carefully calibrated degrees of contention and failures, and then leaves it up to the database vendor to optimize within those parameters. When Amazon did this for Aurora, they showed that with their custom storage backplane, they were able to absolutely murder normal RDS-based MySQL in terms of TPC-C throughput. Take a look: https://www.slideshare.net/AmazonWebServices/dat202getting-started-with-amazon-aurora/14 https://www.slideshare.net/AmazonWebServices/dat202getting-s.... The bottom right quadrant of that slide shows what happens to a standard MySQL server when given 10,000 TPC-C warehouses to chew on (about 800 GB of TPC-C dataset). The max throughput is supposed to be ~128,000 tpmC. RDS MySQL in their tests was able to do 69. That is quite simply, not impressive. Even Aurora, which is a full 136x the tpmC throughput, is still more than 10x less than the max throughput that should be achieved at that number of warehouses. CockroachDB can scale out, so while it may have more latency when it performs a simple transaction or millions of simple transactions over a small set of data, it can also easily scale to handle 10,000 TPC-C warehouses, with 126,000 tpmC. If you're going to talk about performance, you really need a serious benchmark, or you're just kidding yourself. Let me put this another way. While Amazon frequently talks about how Aurora can be scaled to 64TB of data, in the context of TPC-C, that's a risible claim. A maximal instance of Aurora, tuned by AWS, can handle max TPC-C throughput at something between 80GB and 800GB of data (i.e. 1,000 to 10,000 warehouses). It's like they're selling you a pickup truck with a bed they say can handle 64 tons, only if you load it that way, it can only travel 2mph, and can't turn. Caveat emptor!
- amq 9y ago
  When I tested CockroachDB, I did a single-node and multi-node comparison to Galera. I was ready to accept that multi-node is going to have a performance hit, but what surprised me the most was that the single-node performance was roughly 5-10 times worse with CockroachDB. https://github.com/cockroachdb/cockroach/issues/17777#issuecomment-348219495 https://github.com/cockroachdb/cockroach/issues/17777#issuec...
jpalomaki 9y ago
My general rule: always pick the boring major RDBMS - unless you have a specific reason not to.
- kodablah 9y ago
  High availability with automatic failover is a pretty common reason. To do this with a "boring major RDBMS" removes the boring part and requires leaving the beaten path of the core RDBMS product anyways (or paying lots of money). If multi-master w/ automatic scaling and failover were a first class feature of these boring major RDBMSs, you could have the best of all worlds.
  - AgentME 9y ago
    Yeah, I've got a few projects that I'm not doing much where scaling would be an issue, but I really want easy multi-master. I want to be able to have a few database servers and be able to take one offline for maintenance (or because of mishap) with minimal fanfare, without it causing downtime, and without it feeling like I'm performing open heart surgery. There's plenty of ways to fuck up a manual switchover, and I don't want to discover more of them again in production. Technically, I don't really need multi-master, a "standard RDBMS" with automatic failover would probably fit the bill, but I could not for the life of me figure out any standard (and free) way of doing that for mysql/mariadb or postgresql.
kureikain 9y ago
It's really to sad RethinkDB gone. While I keep hearing that the RDMS(Postgres, MySQL...) was built on year of experiences I alwawyas think why we cannot come up with something nicer, more friendlier to the old way. Eventually we will build up knowledge on the new thing and have a system that just as good but also as much friendlier. Example: when looking at how MongoDB handle replication, it's so easy to just add a new node and have it join cluster without seeding some data and set binlog position (like MySQL). Or looking at RethinkDB query language, it's just so much easier compare with SQL, at least to me. Nowsday, I cry whenever I see SQL query. I learn to master it, but life's too short to write those kind of thing.
- ddorian43 9y ago
  But, this is a thing in life when you are simply wrong. So find a way to learn it or remain a subpar dev.

amelius 9y ago

When it comes to databases, always wait for the Jepsen test! https://jepsen.io/ https://jepsen.io/

jordanlewis 9y ago
CockroachDB underwent a Jepsen audit by Kyle Kingsbury before its 1.0 release. Check out the blog post[0] and Kyle's analysis[1]. We also run Jepsen tests against the database nightly, ensuring that its isolation guarantees do not regress. [0]: https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/ https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... [1]: http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829
michaelmior 9y ago
Which they've been doing for around 2 years at least https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockroachdb/ https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...

zellyn 9y ago

You are 100% correct about waiting for several years before using a new database product. However, if you look at the path Cockroach is taking, they're doing all the right things to shave those years down. They already Jepsen themselves, and they've been doing rigorous testing forever now. Very exciting.

nkohari 9y ago
So did RethinkDB. :(
- zellyn 9y ago
  Touché
- forgot-my-pw 9y ago
  It's still alive as an open source project, but probably less support now. https://github.com/rethinkdb/rethinkdb https://github.com/rethinkdb/rethinkdb
  - manigandham 9y ago
    And still available as a managed service on Compose: https://www.compose.com/databases/rethinkdb https://www.compose.com/databases/rethinkdb
  - welder 9y ago
    RethinkDB still has a 2 year old critical memory leak that probably won't ever be fixed: https://github.com/rethinkdb/rethinkdb/issues/5865 https://github.com/rethinkdb/rethinkdb/issues/5865

manigandham 9y ago

A distributed database will never be faster than a single-node RDBMS for simple queries because there is added overhead of coordination between nodes, especially if you're repeatedly writing to the same exact rows. What distributed databases give you is scalability as data grows, and high-availability for safety. Performance comes from that scale and concurrency when your data and queries fit the model.

d4l3k 9y ago
There's definitely cases where a distributed database will have higher bandwidth than a single-node database. Since each range is replicated across 3 machines, that means if you have a lot more nodes than 3, the overlap of ranges significantly decreases and you start being able to handle more requests than a single node can. However, the latency will typically be higher since it always has to talk to at least 3 nodes. If you consider CRDB's DistSQL which lets you run queries on multiple machines, it might be possible to beat a traditional RDBMS on read latency for large queries.
truth_seeker 9y ago
Not entirely correct. The distributed database can scale computation horizontally across the nodes in cluster. Single node is better only for the relatively small dataset. If you go higher volumes and velocity, Vertical scaling is still more expensive than Horizontal scaling. If your data is large, with network bandwidth of 1Gbps (or perhaps better) distributed database may choose to replicate it efficiently across the nodes and do scatter-gather (or divide and conquer) query parallelization and also maintain HA. Citus DB is one example of distributed database which does exactly that.
- manigandham 9y ago
  That’s basically what I said...the point is that simple queries on the same keys will never be faster.
caleblloyd 9y ago
It would be nice if CRDB could have a mode when 1-3 nodes are active that is similar to since master performance with automatic replication. Then if horizontal scale-out is needed a performance hit could be taken. I like the automatic replication of CRDB but the vast majority of apps can't afford to take a big performance hit and don't need large horizontal scale-out.

darksaints 9y ago

> I am skeptical that a networked database with multiple nodes can match the performance of a single master database such as MySQL, PostgreSQL, or SQL Server. It likely can't unless there is some black magic going on. Single node speed will always be faster. But once you get to that point where a single node chokes on the amount of data or query throughput that you have, you don't really have a choice anymore. My personal plan is to start with Postgres RDS. Grow until RDS doesn't work anymore, then move to ultra beefy bare metal servers in colocation with AWS Direct Connect. If I ever outgrow an 4x24 core server with 3TB of memory on a RAIDed NVMe disk cluster, I might move to Citus. For the various distributed database companies out there, I believe the one that will win in the marketplace is the one working or partnering to develop specialized hardware and networking, and then optimizing for it.

p0rkbelly 9y ago
Why not bare metal on AWS? Going to DX is going to give you a conservative 10ms on your calls...
- darksaints 9y ago
  That might be a good intermediate option, but they still don't have instances with 4x24 cpu, or >488GB ram.
  - openasocket 9y ago
    Did you look at the x1 options? Up to 128 CPUs and 3,904GB of RAM on the x1e.32xlarge.
    - darksaints 9y ago
      Those aren't bare metal. And in terms of database scalability, I would jump for bare metal before beefier servers, due to the fact that the disk is local and you don't have to deal with EBS noisy neighbors.

zzzcpan 9y ago

As others said, Cockroach's performance is limited by its consistency model. It is not something that can ever get close to a single master MySQL. This might also become an issue for the company and ultimately prevent it from reaching mass adoption and succeeding, at least commercially.

tnolet 9y ago

Do you need a distributed database for your use case? Or do you like the new tech? Genuine question, not trying to be snarky.

caleblloyd 9y ago
I do not have a particular project in mind, but think it would be good to have an easily containerizable, replicated OLTP database in my toolkit. I do some work with Entity Framework Core providers and am thinking about trying to implement a provider for CRDB.
- MycroftH 9y ago
  There is already some work towards this with http://www.npgsql.org/ http://www.npgsql.org/ I haven't gone back to try the full entity framework recently, but if you find any bugs, please let us know. If you want to roll your own, we're here to help too.
- manigandham 9y ago
  CRDB is using the Postgresql wire transport and already works with npgsql + ef core.
  - caleblloyd 9y ago
    Are you sure? I know that the NPGSQL DB driver works with CRDB. Data types, DDL, and supported features are a big part of EF Core providers though. I have not researched it yet but I was pretty sure the NPGSQL EF Core Provider implemetation was just for PG.
    - manigandham 9y ago
      CockroachDB uses the same PostgreSQL wire protocol and SQL dialect so it just looks and works like a pg database. There are some minor differences but DDL and data types are already included in the syntax: https://www.cockroachlabs.com/docs/stable/porting-postgres.html https://www.cockroachlabs.com/docs/stable/porting-postgres.h... It's great if you want to contribute though but I'd recommend just working on the PG EF Core provider which already has some work towards CRDB specific features: https://github.com/npgsql/Npgsql.EntityFrameworkCore.PostgreSQL/issues?utf8=%E2%9C%93&q=is%3Aissue+cockroachdb https://github.com/npgsql/Npgsql.EntityFrameworkCore.Postgre...

gred 9y ago

> I am genuinely interested in using CockroachDB as a primary datastore but feel like I have been burned too many times by hopping on board with a young database. > According to crunchbase, Cockroach Labs has raised $53.5m (RethinkDB had only raised $12m). Is there evidence that Cockroach Labs is on track to make money and is going to survive? I'm in a similar situation: new project, CRDB is a good fit on paper, but I'm unable to get traction even for a proof of concept exercise because of the risk associated with a database backed by a startup. This is at a large, conservative, 300k+ employee company, so I guess it's understandable... but regrettable nonetheless. See you guys in 5 years ;-)

manigandham 9y ago
Becoming a paying customer is a way to help them survive.

hnkimb3558 9y ago

Performance is a very tricky thing to measure in a database. CockroachDB's performance is certainly affected by its consistency model. In particular, CockroachDB handles transactions using serializable isolation, and writes using consensus replication. This means that it's difficult to make an apples to apples comparison for performance between a replicated CockroachDB cluster and a single master MySQL server. However, that is the whole point of a well thought out, standardized benchmark like TPC-C. It creates a pseudo-realistic workload, and mandates the use of transactions with a reasonable isolation level or higher (snapshot isolation), carefully calibrated degrees of contention and failures, and then leaves it up to the database vendor to optimize within those parameters. When Amazon did this for Aurora, they showed that with their custom storage backplane, they were able to absolutely murder normal RDS-based MySQL in terms of TPC-C throughput. Take a look: https://www.slideshare.net/AmazonWebServices/dat202getting-started-with-amazon-aurora/14 https://www.slideshare.net/AmazonWebServices/dat202getting-s.... The bottom right quadrant of that slide shows what happens to a standard MySQL server when given 10,000 TPC-C warehouses to chew on (about 800 GB of TPC-C dataset). The max throughput is supposed to be ~128,000 tpmC. RDS MySQL in their tests was able to do 69. That is quite simply, not impressive. Even Aurora, which is a full 136x the tpmC throughput, is still more than 10x less than the max throughput that should be achieved at that number of warehouses. CockroachDB can scale out, so while it may have more latency when it performs a simple transaction or millions of simple transactions over a small set of data, it can also easily scale to handle 10,000 TPC-C warehouses, with 126,000 tpmC. If you're going to talk about performance, you really need a serious benchmark, or you're just kidding yourself. Let me put this another way. While Amazon frequently talks about how Aurora can be scaled to 64TB of data, in the context of TPC-C, that's a risible claim. A maximal instance of Aurora, tuned by AWS, can handle max TPC-C throughput at something between 80GB and 800GB of data (i.e. 1,000 to 10,000 warehouses). It's like they're selling you a pickup truck with a bed they say can handle 64 tons, only if you load it that way, it can only travel 2mph, and can't turn. Caveat emptor!

amq 9y ago
When I tested CockroachDB, I did a single-node and multi-node comparison to Galera. I was ready to accept that multi-node is going to have a performance hit, but what surprised me the most was that the single-node performance was roughly 5-10 times worse with CockroachDB. https://github.com/cockroachdb/cockroach/issues/17777#issuecomment-348219495 https://github.com/cockroachdb/cockroach/issues/17777#issuec...

jpalomaki 9y ago

My general rule: always pick the boring major RDBMS - unless you have a specific reason not to.

kodablah 9y ago
High availability with automatic failover is a pretty common reason. To do this with a "boring major RDBMS" removes the boring part and requires leaving the beaten path of the core RDBMS product anyways (or paying lots of money). If multi-master w/ automatic scaling and failover were a first class feature of these boring major RDBMSs, you could have the best of all worlds.
- AgentME 9y ago
  Yeah, I've got a few projects that I'm not doing much where scaling would be an issue, but I really want easy multi-master. I want to be able to have a few database servers and be able to take one offline for maintenance (or because of mishap) with minimal fanfare, without it causing downtime, and without it feeling like I'm performing open heart surgery. There's plenty of ways to fuck up a manual switchover, and I don't want to discover more of them again in production. Technically, I don't really need multi-master, a "standard RDBMS" with automatic failover would probably fit the bill, but I could not for the life of me figure out any standard (and free) way of doing that for mysql/mariadb or postgresql.

kureikain 9y ago

It's really to sad RethinkDB gone. While I keep hearing that the RDMS(Postgres, MySQL...) was built on year of experiences I alwawyas think why we cannot come up with something nicer, more friendlier to the old way. Eventually we will build up knowledge on the new thing and have a system that just as good but also as much friendlier. Example: when looking at how MongoDB handle replication, it's so easy to just add a new node and have it join cluster without seeding some data and set binlog position (like MySQL). Or looking at RethinkDB query language, it's just so much easier compare with SQL, at least to me. Nowsday, I cry whenever I see SQL query. I learn to master it, but life's too short to write those kind of thing.

ddorian43 9y ago
But, this is a thing in life when you are simply wrong. So find a way to learn it or remain a subpar dev.