4 ms·

> I’m not a database guy, I’m a node guy. Is this really a distinction your "scrappy startup type" can afford to make? > Although since version 3.4.x Mongo pa

> I’m not a database guy, I’m a node guy. Is this really a distinction your "scrappy startup type" can afford to make? > Although since version 3.4.x Mongo passes all Jepsen tests, when our original database was chosen the current version of Mongo at the time did not and was deemed unreliable by the team Does not pass all Jepsen tests => deeemed unreliable :( Wow. Of all possible reasons, THAT'S the reason MongoDB wasn't chosen? Jepsen, man. Pretty sad so many people just look at Jepsen results like this. Jepsen, really, is about the tests more than the results. It highlights various failure modes that may be difficult or uncommon to replicate in production. These failure modes are important to know about to understand the product you are using, but are not really the end-all when it comes to deciding whether that product is worth using! Anyway, show me a database that passes Jepsen and I'll show you a database that only accepts writes to a single node.

andreimatei1 9y ago
> Anyway, show me a database that passes Jepsen and I'll show you a database that only accepts writes to a single node. I know one - CockroachDB :). Relational, strongly-consistent, SQL, elastic, multi-active replication - the dream! (I work on it) https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/ https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829
- hmottestad 9y ago
  RethinkDB also passes jepsen.
  - linkmotif 9y ago
    It's also defunct so... :)
    - jjuel 9y ago
      Open Sourced is definitely different than defunct...
      - linkmotif 9y ago
        I know I know I was just kidding :D
      - manigandham 9y ago
        Without a company sponsoring and providing support, it is effectively dead for any serious commercial use.
  - a-robinson 9y ago
    For single-key operations only, but that is still much better than the average system he's looked at.
- linkmotif 9y ago
  Hehe ;) Not to mention—it's Postgres compatible!!
- StreamBright 9y ago
  I am a little hesitant using a database that is younger than 10 years of age (same goes to programming languages, frameworks, etc.).
- Thaxll 9y ago
  “For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state).” If someone can explains those abysmal low numbers ...
  - andreimatei1 9y ago
    Well, those numbers, if I remember correctly, refer to insert rate in the face of some extreme amounts of contention - tons of overlapping reads and writes (that's how the test was trying to trigger consistency violations). And moreover, they were measured while the Jepsen test framework was messing with the cluster. Contention is a problem for every database, and particularly so for CRDB. In the absence of contention, we routinely see thousands of queries per second per node. Since the time of that analyses, we have done a significant amount of work for speeding up these high-contention scenarios, with quite dramatic differences in some cases we looked at. We pretty much changed our transaction execution model from a more "optimistic" one where transactions can abort each other and induce thrashing, to something resembling more the traditional row locks. So, hopefully, even for these atypical uses cases we should generally perform much better.
  - AlisdairO 9y ago
    "These, of course, are pitiful numbers for a database. However, it’s important to note that these numbers are not representative of what real-world applications can expect to see from CockroachDB. First, this is a test in which the Jepsen nemesis is continuously tweaking the network configuration to cause (and heal) partitions between different nodes. You don’t commonly see behavior like this in a real world deployment. Second, tests like these deliberately try to reach high levels of contention, in which many transactions are trying to operate on the same keys. In real-world situations contention tends to be much lower as each user is operating mainly on their own data. The performance of all transactional systems suffer under high contention (although CockroachDB’s optimistic concurrency model fares worse than most)"
  - arjunnarayan 9y ago
    Reposting my comment from when this came up on our 1.0 launch: IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards. [1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3]. [2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size. [3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501 https://github.com/cockroachdb/cockroach/pull/13501
- didip 9y ago
  CockroachDB peeps, you guys should really publish a new blog post with updated performance benchmarks. Super low throughput numbers is deterring a lot of potential users, imho.
tootie 9y ago
I think if you're swimming a sea of options you look for any port and Aphyr's blog gives you all sorts of ammo to eliminate choices even if they're only failing on microscopic edge cases that will never affect your small-scale project.
- lomnakkus 9y ago
  Do you really think network partitions are an edge case? Jepsen doesn't actually do anything particularly edge-casey -- it just introduces network partitions and verifies that the cluster doesn't do any crazy things. (There are also relatively common cases it doesn't test, e.g. hard power-off.)
- manigandham 9y ago
  Jepsen is not microscopic edge-cases. It is the heart of the distributed system promises that database vendors make but routinely do not meet. If you would rather go back to trusting marketing speak instead, good luck to you.
  - tootie 9y ago
    I think the fact that most of the DBs that failed the Jepsen test are also running thousands of successful production installs is the counterargument.
    - manigandham 9y ago
      That's not much of an argument and "successful" is pretty subjective. Many teams have found issues where vendors have claimed otherwise, very common with mongodb and elasticsearch. There are also thousands of problems that happen with no public reporting and no vendor is going to willingly release details of their support cases, especially when they're being paid to fix their own problems. There's nothing but positive outcomes from jepsen and holding databases to strict standards to see if they actually meet the goals they state.

andreimatei1 9y ago

> Anyway, show me a database that passes Jepsen and I'll show you a database that only accepts writes to a single node. I know one - CockroachDB :). Relational, strongly-consistent, SQL, elastic, multi-active replication - the dream! (I work on it) https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/ https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829

hmottestad 9y ago
RethinkDB also passes jepsen.
- linkmotif 9y ago
  It's also defunct so... :)
  - jjuel 9y ago
    Open Sourced is definitely different than defunct...
    - linkmotif 9y ago
      I know I know I was just kidding :D
    - manigandham 9y ago
      Without a company sponsoring and providing support, it is effectively dead for any serious commercial use.
- a-robinson 9y ago
  For single-key operations only, but that is still much better than the average system he's looked at.
linkmotif 9y ago
Hehe ;) Not to mention—it's Postgres compatible!!
StreamBright 9y ago
I am a little hesitant using a database that is younger than 10 years of age (same goes to programming languages, frameworks, etc.).
Thaxll 9y ago
“For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state).” If someone can explains those abysmal low numbers ...
- andreimatei1 9y ago
  Well, those numbers, if I remember correctly, refer to insert rate in the face of some extreme amounts of contention - tons of overlapping reads and writes (that's how the test was trying to trigger consistency violations). And moreover, they were measured while the Jepsen test framework was messing with the cluster. Contention is a problem for every database, and particularly so for CRDB. In the absence of contention, we routinely see thousands of queries per second per node. Since the time of that analyses, we have done a significant amount of work for speeding up these high-contention scenarios, with quite dramatic differences in some cases we looked at. We pretty much changed our transaction execution model from a more "optimistic" one where transactions can abort each other and induce thrashing, to something resembling more the traditional row locks. So, hopefully, even for these atypical uses cases we should generally perform much better.
- AlisdairO 9y ago
  "These, of course, are pitiful numbers for a database. However, it’s important to note that these numbers are not representative of what real-world applications can expect to see from CockroachDB. First, this is a test in which the Jepsen nemesis is continuously tweaking the network configuration to cause (and heal) partitions between different nodes. You don’t commonly see behavior like this in a real world deployment. Second, tests like these deliberately try to reach high levels of contention, in which many transactions are trying to operate on the same keys. In real-world situations contention tends to be much lower as each user is operating mainly on their own data. The performance of all transactional systems suffer under high contention (although CockroachDB’s optimistic concurrency model fares worse than most)"
- arjunnarayan 9y ago
  Reposting my comment from when this came up on our 1.0 launch: IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards. [1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3]. [2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size. [3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501 https://github.com/cockroachdb/cockroach/pull/13501
didip 9y ago
CockroachDB peeps, you guys should really publish a new blog post with updated performance benchmarks. Super low throughput numbers is deterring a lot of potential users, imho.

tootie 9y ago

I think if you're swimming a sea of options you look for any port and Aphyr's blog gives you all sorts of ammo to eliminate choices even if they're only failing on microscopic edge cases that will never affect your small-scale project.

lomnakkus 9y ago
Do you really think network partitions are an edge case? Jepsen doesn't actually do anything particularly edge-casey -- it just introduces network partitions and verifies that the cluster doesn't do any crazy things. (There are also relatively common cases it doesn't test, e.g. hard power-off.)
manigandham 9y ago
Jepsen is not microscopic edge-cases. It is the heart of the distributed system promises that database vendors make but routinely do not meet. If you would rather go back to trusting marketing speak instead, good luck to you.
- tootie 9y ago
  I think the fact that most of the DBs that failed the Jepsen test are also running thousands of successful production installs is the counterargument.
  - manigandham 9y ago
    That's not much of an argument and "successful" is pretty subjective. Many teams have found issues where vendors have claimed otherwise, very common with mongodb and elasticsearch. There are also thousands of problems that happen with no public reporting and no vendor is going to willingly release details of their support cases, especially when they're being paid to fix their own problems. There's nothing but positive outcomes from jepsen and holding databases to strict standards to see if they actually meet the goals they state.