7 ms·

I just want to emphasize that: a.) this is a beta product where the team has been focusing on completeness and correctness before performance b.) I was testin

by aphyr 10y ago

I just want to emphasize that: a.) this is a beta product where the team has been focusing on completeness and correctness before performance b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0. c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.

qaq 10y ago
They need to be at least 1000x to 10,000x faster on the read side though
- aphyr 10y ago
  I agree that it can and should be faster, but the targets will depend on what you're going for. 20k linearizable reads/sec on EBS-backed instances is, uh, maybe a tad optimistic.
- hinkley 10y ago
  I'm a little confused about how read speed is 2x slower than write speed. With respect to 'correctness', you're drifting into pyrrhic victory or 'not even wrong' territory at that point. When there are basic expectations of behavior that aren't being met, many of us would reject the idea that this code is 'correct'.
  - aphyr 10y ago
    As I mentioned in the article's introductory paragraphs, where Spanner forces a minimum latency on writes to ensure consistency, CockroachDB pushes that latency to reads which contend with a write.
    - qaq 10y ago
      Latency is one thing throughput is a different animal 2 reads/second per m3.large node that really forces one to think there are some severe architectural issues.
      - hinkley 10y ago
        You're not wrong. Whoever downvoted you is pushing an agenda and I'm not happy about it. Reads per second is ultimately a measure of how many operations can be retired per second, not how long the operation takes. Anyone who has spent fifteen minutes learning about capacity planning should know that. 5 reads per second doesn't mean each read takes 200ms. It might mean 1 second per read spread across 5 threads of execution, or 3 seconds per read across 15 workers.
        stickfigure 10y ago
        The parent poster (and many others in this thread) are assuming that performance under this deliberately pathological test is reflective of performance in the real world. Optimistic locking systems inherently perform poorly under contention. But they also perform better than pessimistic concurrency systems overall because in the real world we design applications to avoid contention. As an example, the Google App Engine datastore runs zillions of QPS across petabytes of data in a massive distributed cluster. But if you build an app that does nothing but mutate a single piece of state over and over, you'll top out at a couple transactions per second. This is painful if you're trying to build a simple counter, but with minimal care you can build a system that scales to any sized dataset and traffic volume.
        hinkley 10y ago
        So the benchmark is only informative from the standpoint of determining which records should be split to avoid concurrent writes. For instance you wouldn't expect a single user to make 5 comments or upvotes per second so storing data about recent activity with the user isn't a bottleneck that you need to design for. Storing data about responses with an item might also be okay as long as you don't plan to be HN or Reddit (Github, for example, would be just fine). But if you want to track activity globally (eg, managing the watch list notifications in github), you will need to design around that number.
  - arjunnarayan 10y ago
    [Disclaimer: CockroachDB engineer here, working on performance and benchmarking] IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards. [1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3]. [2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size. [3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501 https://github.com/cockroachdb/cockroach/pull/13501
    - hinkley 10y ago
      Oh, I see. That makes sense, thank you. The benchmark isn't quite apples to apples. I'm used to people doing put/get benchmarks, not put/search workflows. Merging results from multiple machines is nothing to sneeze at, especially if you have replication between nodes. I worked for a search engine dotbomb, and I had a constant worry that the core engineering team had not solved a very similar problem. But since the funding round failed I'll never know for sure what they had planned.
    - grogers 10y ago
      Are read-only transactions handled without contention? I understand that a read-write transaction needs to be aborted if a later conflicting write is committed. But read-only transactions can view a snapshot of what was committed before they begin - you trade a little bit of latency for being conflict free. It doesn't even have to be the default as long as you offer it as an option, you might want this if you know you are reading hot keys, or as a fallback if you've been aborted X times. This was IMO the big innovation of true time in spanner - contention free reads at either now, or at some time in the past.
  - erichocean 10y ago
    > I'm a little confused about how read speed is 2x slower than write speed. FWIW, you're also describing Cassandra and it seems to do fine in the marketplace.
    - phamilton 10y ago
      Also describing random read/write disk benchmarks.

by aphyr 10y ago

qaq 10y ago

They need to be at least 1000x to 10,000x faster on the read side though

aphyr 10y ago
I agree that it can and should be faster, but the targets will depend on what you're going for. 20k linearizable reads/sec on EBS-backed instances is, uh, maybe a tad optimistic.
hinkley 10y ago
I'm a little confused about how read speed is 2x slower than write speed. With respect to 'correctness', you're drifting into pyrrhic victory or 'not even wrong' territory at that point. When there are basic expectations of behavior that aren't being met, many of us would reject the idea that this code is 'correct'.
- aphyr 10y ago
  As I mentioned in the article's introductory paragraphs, where Spanner forces a minimum latency on writes to ensure consistency, CockroachDB pushes that latency to reads which contend with a write.
  - qaq 10y ago
    Latency is one thing throughput is a different animal 2 reads/second per m3.large node that really forces one to think there are some severe architectural issues.
    - hinkley 10y ago
      You're not wrong. Whoever downvoted you is pushing an agenda and I'm not happy about it. Reads per second is ultimately a measure of how many operations can be retired per second, not how long the operation takes. Anyone who has spent fifteen minutes learning about capacity planning should know that. 5 reads per second doesn't mean each read takes 200ms. It might mean 1 second per read spread across 5 threads of execution, or 3 seconds per read across 15 workers.
      - stickfigure 10y ago
        The parent poster (and many others in this thread) are assuming that performance under this deliberately pathological test is reflective of performance in the real world. Optimistic locking systems inherently perform poorly under contention. But they also perform better than pessimistic concurrency systems overall because in the real world we design applications to avoid contention. As an example, the Google App Engine datastore runs zillions of QPS across petabytes of data in a massive distributed cluster. But if you build an app that does nothing but mutate a single piece of state over and over, you'll top out at a couple transactions per second. This is painful if you're trying to build a simple counter, but with minimal care you can build a system that scales to any sized dataset and traffic volume.
        hinkley 10y ago
        So the benchmark is only informative from the standpoint of determining which records should be split to avoid concurrent writes. For instance you wouldn't expect a single user to make 5 comments or upvotes per second so storing data about recent activity with the user isn't a bottleneck that you need to design for. Storing data about responses with an item might also be okay as long as you don't plan to be HN or Reddit (Github, for example, would be just fine). But if you want to track activity globally (eg, managing the watch list notifications in github), you will need to design around that number.
- arjunnarayan 10y ago
  [Disclaimer: CockroachDB engineer here, working on performance and benchmarking] IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards. [1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3]. [2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size. [3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501 https://github.com/cockroachdb/cockroach/pull/13501
  - hinkley 10y ago
    Oh, I see. That makes sense, thank you. The benchmark isn't quite apples to apples. I'm used to people doing put/get benchmarks, not put/search workflows. Merging results from multiple machines is nothing to sneeze at, especially if you have replication between nodes. I worked for a search engine dotbomb, and I had a constant worry that the core engineering team had not solved a very similar problem. But since the funding round failed I'll never know for sure what they had planned.
  - grogers 10y ago
    Are read-only transactions handled without contention? I understand that a read-write transaction needs to be aborted if a later conflicting write is committed. But read-only transactions can view a snapshot of what was committed before they begin - you trade a little bit of latency for being conflict free. It doesn't even have to be the default as long as you offer it as an option, you might want this if you know you are reading hot keys, or as a fallback if you've been aborted X times. This was IMO the big innovation of true time in spanner - contention free reads at either now, or at some time in the past.
- erichocean 10y ago
  > I'm a little confused about how read speed is 2x slower than write speed. FWIW, you're also describing Cassandra and it seems to do fine in the marketplace.
  - phamilton 10y ago
    Also describing random read/write disk benchmarks.