4 ms·

> It's debatable these bugs would've existed if a Jepson pipeline had been consistently in use from the 0.4.x days. I don't think it would have helped: the Jep

by aphyr 7y ago

> It's debatable these bugs would've existed if a Jepson pipeline had been consistently in use from the 0.4.x days. I don't think it would have helped: the Jepsen tests I wrote in 2014 only checked single-key gets, puts, and CaS operations; the problems we found in this report were in watches and locks.

jzelinskie 7y ago
That's a good point that I hadn't remembered (first-class locks weren't totally baked back then), but I think the intent of my original comment holds true: we should have valued the Jepson test suite more and continuously leveraged and improved our usage of it, rather than doing one-off tests every now and then. As a result, the community has no idea if there were regressions between now and then. I will admit what I don't know: I have no idea if this was actually feasible or desirable for us or you at the time, but I'd feel more comfortable if every release of etcd, Zookeeper, etc... had some kind of Jepson stamp of approval on it in terms of API coverage. I'm pretty sure CockroachDB has tried this[0], but it's been a few years and I don't know how it turned out for them long term. [0]: https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockroachdb/ https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...
- irfansharif 7y ago
  CockroachDB runs the Jepsen test suite nightly. We've been following along Aphyr's recent test additions (`multi-register` for instance, which immediately caught [0]), porting them over when appropriate. We definitely have work to be done incorporating the more DDL focused tests that tripped up YugaByte. [0]: https://github.com/cockroachdb/cockroach/pull/40600 https://github.com/cockroachdb/cockroach/pull/40600
  - breakingcups 7y ago
    That's pretty damn cool. I wish I had more time to seriously try out CockroachDB.
- AtlasBarfed 7y ago
  A distributed state system should have jepsen as a bare minimum to qualify for serious consideration. It would be nice to get follow-ups too, personally for Cassandra, Kafka, couch, and elastic.

by aphyr 7y ago

jzelinskie 7y ago

That's a good point that I hadn't remembered (first-class locks weren't totally baked back then), but I think the intent of my original comment holds true: we should have valued the Jepson test suite more and continuously leveraged and improved our usage of it, rather than doing one-off tests every now and then. As a result, the community has no idea if there were regressions between now and then. I will admit what I don't know: I have no idea if this was actually feasible or desirable for us or you at the time, but I'd feel more comfortable if every release of etcd, Zookeeper, etc... had some kind of Jepson stamp of approval on it in terms of API coverage. I'm pretty sure CockroachDB has tried this[0], but it's been a few years and I don't know how it turned out for them long term. [0]: https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockroachdb/ https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...

irfansharif 7y ago
CockroachDB runs the Jepsen test suite nightly. We've been following along Aphyr's recent test additions (`multi-register` for instance, which immediately caught [0]), porting them over when appropriate. We definitely have work to be done incorporating the more DDL focused tests that tripped up YugaByte. [0]: https://github.com/cockroachdb/cockroach/pull/40600 https://github.com/cockroachdb/cockroach/pull/40600
- breakingcups 7y ago
  That's pretty damn cool. I wish I had more time to seriously try out CockroachDB.
AtlasBarfed 7y ago
A distributed state system should have jepsen as a bare minimum to qualify for serious consideration. It would be nice to get follow-ups too, personally for Cassandra, Kafka, couch, and elastic.