5 ms·

Any newer Jepsen test since this one: https://jepsen.io/analyses/cockroachdb-beta-20160829 https://jepsen.io/analyses/cockroachdb-beta-20160829 Also, how's the

by didip 10y ago

Any newer Jepsen test since this one: https://jepsen.io/analyses/cockroachdb-beta-20160829 https://jepsen.io/analyses/cockroachdb-beta-20160829 Also, how's the performance nowadays?

knz42 10y ago
The tests that Aphyr describes are ran regularly, to prevent regressions.
elvinyung 10y ago
I noticed that the question on performance was ignored, but I think it's worth noting that FoundationDB died on exactly the same beach. [1] I'm not extremely familiar with the CockroachDB architecture (other than "it's kinda like the F1/Spanner/Bigtable stack"), but I suspect that there are some commonalities. For example, CockroachDB's Bigtable-style two-level range indexing means you could run into exactly this problem: > FDB-SQL stores metadata in the KV store itself. This is great from a distributed correctness point-of-view. It’s also great from a simplicity point-of-view. If I trust the underlying system to be safe and consistent, then my metadata is also safe and consistent. But now I need to do a ton of reads before I run my SQL to know how to run my SQL. Where’s my data, for example? [1] https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...
- petermattis 10y ago
  (Cockroach Labs co-founder here) CockroachDB also stores SQL metadata inside of the KV store, but that metadata is also gossiped around the cluster (i.e. it is replicated to every node) so that SQL execution almost never has to read it from the KV store. Handling changes to this SQL metadata is challenging and required a design [1] that is unable to take advantage of the easy correctness of simply doing reads from the KV store on every operation. [1] https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/table_descriptor_lease.md https://github.com/cockroachdb/cockroach/blob/master/docs/RF...
  - elvinyung 10y ago
    I see -- so how is performance? I see that there are no published numbers for e.g. TPC-C or YCSB. (For the record -- in case I seem to be especially pessimistic -- I'm actually extremely hopeful that CRDB will actually make scaling OLTP applications significantly easier for everyone. It's just that this probably shouldn't be FDB all over again.)
    - petermattis 10y ago
      Performance is "ok". CockroachDB 1.0 won't be winning benchmarks, but that was never the goal. We have seen near linear scaling from 3-64 nodes on a uniformly random write workload. And the new distributed SQL execution engine has shown O(num-nodes-in-cluster) speedup on some queries. But none of this has been wrapped up into published numbers, so take it with a grain of salt.
petermattis 10y ago
(Cockroach Labs co-founder here) This is a really hard question to answer because there are so many different scenarios to test. Even if you restrict the question to KV operations a simple answer isn't forthcoming. For example, if you want a distributed system with the fastest writes, then Cassandra is your best choice. Oh, but you better pay a lot of attention to all of the caveats regarding tombstones and node outages and overwriting data. Overall, performance has improved a lot over the past year, but we still have a lot of work to do. Some of the high-contention scenarios that caused poor performance in the Jepsen tests have seen a huge improvement (10-100x), though admittedly we were starting from a poor position. So far we've performed the most significant comparisons against Cassandra and MongoDB. In terms of performance Cassandra > MongoDB > CockroachDB, but note that most (all?) benchmarks test idealized situations. YCSB tests ideal write scenarios where data is never overwritten or deleted and data correctness is easily achieved because there are no node outages. Unsurprisingly, PostgreSQL will outperform a single-node Cockroach cluster for simple operations. The internal target was to be within 2x of PostgreSQL performance for simple INSERT, UPDATE, DELETE and SELECT statements. We were at that target earlier in the year, but a quick check reveals that we've slipped a bit. Sorry I don't have anything more concrete to offer at this time. I realize I've been saying "we're working on it" for too long. Really, we are.

by didip 10y ago

Any newer Jepsen test since this one: https://jepsen.io/analyses/cockroachdb-beta-20160829 https://jepsen.io/analyses/cockroachdb-beta-20160829 Also, how's the performance nowadays?

knz42 10y ago

The tests that Aphyr describes are ran regularly, to prevent regressions.

elvinyung 10y ago

I noticed that the question on performance was ignored, but I think it's worth noting that FoundationDB died on exactly the same beach. [1] I'm not extremely familiar with the CockroachDB architecture (other than "it's kinda like the F1/Spanner/Bigtable stack"), but I suspect that there are some commonalities. For example, CockroachDB's Bigtable-style two-level range indexing means you could run into exactly this problem: > FDB-SQL stores metadata in the KV store itself. This is great from a distributed correctness point-of-view. It’s also great from a simplicity point-of-view. If I trust the underlying system to be safe and consistent, then my metadata is also safe and consistent. But now I need to do a ton of reads before I run my SQL to know how to run my SQL. Where’s my data, for example? [1] https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

petermattis 10y ago
(Cockroach Labs co-founder here) CockroachDB also stores SQL metadata inside of the KV store, but that metadata is also gossiped around the cluster (i.e. it is replicated to every node) so that SQL execution almost never has to read it from the KV store. Handling changes to this SQL metadata is challenging and required a design [1] that is unable to take advantage of the easy correctness of simply doing reads from the KV store on every operation. [1] https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/table_descriptor_lease.md https://github.com/cockroachdb/cockroach/blob/master/docs/RF...
- elvinyung 10y ago
  I see -- so how is performance? I see that there are no published numbers for e.g. TPC-C or YCSB. (For the record -- in case I seem to be especially pessimistic -- I'm actually extremely hopeful that CRDB will actually make scaling OLTP applications significantly easier for everyone. It's just that this probably shouldn't be FDB all over again.)
  - petermattis 10y ago
    Performance is "ok". CockroachDB 1.0 won't be winning benchmarks, but that was never the goal. We have seen near linear scaling from 3-64 nodes on a uniformly random write workload. And the new distributed SQL execution engine has shown O(num-nodes-in-cluster) speedup on some queries. But none of this has been wrapped up into published numbers, so take it with a grain of salt.

petermattis 10y ago

(Cockroach Labs co-founder here) This is a really hard question to answer because there are so many different scenarios to test. Even if you restrict the question to KV operations a simple answer isn't forthcoming. For example, if you want a distributed system with the fastest writes, then Cassandra is your best choice. Oh, but you better pay a lot of attention to all of the caveats regarding tombstones and node outages and overwriting data. Overall, performance has improved a lot over the past year, but we still have a lot of work to do. Some of the high-contention scenarios that caused poor performance in the Jepsen tests have seen a huge improvement (10-100x), though admittedly we were starting from a poor position. So far we've performed the most significant comparisons against Cassandra and MongoDB. In terms of performance Cassandra > MongoDB > CockroachDB, but note that most (all?) benchmarks test idealized situations. YCSB tests ideal write scenarios where data is never overwritten or deleted and data correctness is easily achieved because there are no node outages. Unsurprisingly, PostgreSQL will outperform a single-node Cockroach cluster for simple operations. The internal target was to be within 2x of PostgreSQL performance for simple INSERT, UPDATE, DELETE and SELECT statements. We were at that target earlier in the year, but a quick check reveals that we've slipped a bit. Sorry I don't have anything more concrete to offer at this time. I realize I've been saying "we're working on it" for too long. Really, we are.