6 ms·

Jepsen: Scylla 4.2-rc3

172 points by aphyr 6y ago

PeterCorless 6y ago
Companion blog on ScyllaDB's site: https://www.scylladb.com/2020/12/23/jepsen-and-scylla-putting-consistency-to-the-test/ https://www.scylladb.com/2020/12/23/jepsen-and-scylla-puttin...
gred 6y ago
The wolf also shall dwell with the lamb, The leopard shall lie down with the young goat, The calf and the young lion and the fatling together; And a little child shall lead them. All popular databases shall routinely undergo Jepsen reviews; And none shall prohibit publication of test or benchmark results. Isaiah 11:6-7 (New Database Translation)
- PeterCorless 6y ago
  We worked closely with Kyle (@Aphyr) as he prepared this report. He was nothing but gracious with his time, brutal but fair with his feedback, and unrelenting in his commitment to discerning precisely where and how we strayed from the one true path of rectitudinous database behavior. Hat's off to him.
- __s 6y ago
  http://www.solidrockbaptist.net/satans-niv-bible.html http://www.solidrockbaptist.net/satans-niv-bible.html > Just a few examples should be enough to show you the problem. In the New International Version (NIV) it says in 2 Samuel 21:19 that someone other than David killed the giant Goliath. Anyone knows this is not true, except the 'scholars' who translated it. According to Hebrews 3:16 in the NIV all the children of Israel rebelled in the wilderness and none made it into the promised land; however, we know that Joshua and Caleb did not rebel, and God promised them each a place in the promised land (just read Numbers 13 & 14). Another mistake is found in Mark 1:2, 3 where according to the NIV the writer of Malachi was Isaiah! Just check out Isaiah 40:3 and Malachi 3:1 - the KJV has it right and the NIV has it wrong. For comparison of the verse this comment parodies, https://biblehub.com/isaiah/11-6.htm https://biblehub.com/isaiah/11-6.htm
  - PeterCorless 6y ago
    So are you saying the Bible suffers from a stale read problem? Or are you asserting it be eventually consistent?
    - Rapzid 6y ago
      The bible is most certainly AP.
- nn3 6y ago
  Hell will freeze over before Jepsen runs on Oracle
  - winrid 6y ago
    I would be surprised if it did poorly, however.
josephg 6y ago
I love Kyle's work with Jepsen. Has anyone done anything similar with filesystems? I'm imagining a filesystem mounted on a virtual device, and having the virtual device randomly suffer write faults and various other failures. And then a fuzzer asserting that all consistency requirements are still met. (All operations before a successful fsync should be stored, no lost writes or files, no unexpected garbage data, etc.)
- Smaug123 6y ago
  Relatedly, at the Midlands Grad School 2019 conference, John Hughes did an introductory course to QuickCheck in which one of the final exercises (meant to demonstrate the use of Haskell and QuickCheck to black-box test arbitrary binaries) was to model `fopen`, `fprintf`, `fclose` and friends in C. I never actually did it, but I guess that's one of the most comprehensive ways to find out how something behaves!
- eis 6y ago
  Yes filesystems are another big important part when it comes to the topic of data consistency. I'm not aware of a project similar to Jepsen for filesystems but there have been research papers like the one presented in this page that explore the guarantees of popular local filesystems: https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ Of course distributed filesystems are even harder but we can see that even writing to a local file is surprisingly complex and in many (most?) apps probably not really guaranteeing proper data consistency. Imho the whole Posix filesystem API is flawed to begin with and it would be great if a modern replacement would emerge.
  - thekozmo 6y ago
    We did something in the neighborhood, check it out: https://www.scylladb.com/2016/02/16/fault-injection-filesystem-software-testing/ https://www.scylladb.com/2016/02/16/fault-injection-filesyst...
  - redis_mlc 6y ago
    > I'm not aware of a project similar to Jepsen for filesystems Not as elaborate as Jepsen, but there has been some work: - Kirk McKusick's papers and work on BSD fs log semantics https://www.researchgate.net/scientific-contributions/Marshall-Kirk-McKusick-70507097 https://www.researchgate.net/scientific-contributions/Marsha... I believe the term to look for is "soft updates." - the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version. You can Bing versions of that by using quotes "fstest.c". - the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.) - Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines. - Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production. SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out. https://en.wikipedia.org/wiki/Dancing_tree https://en.wikipedia.org/wiki/Dancing_tree Source: DBA and storage engineer.
throwdbaaway 6y ago
> Insert is a destructive operation by design. In fact, writing any collection literal (e.g. {} or (1, 2)) is internally implemented by writing a deletion tombstone followed by the new values. These range tombstones are also evil. If you keep doing it to the same partition, even at a rather low QPS, you can easily bring down some nodes by keeping their CPU at 100%. Only tested this with Cassandra though, not Scylla.
- thekozmo 6y ago
  Correct but Jepsen focuses on consistency and correctness and not on destructive. Scylla is exposed to range tombstone issue like many other LSMs. We recently have been improving them - check the most recent issues: https://github.com/scylladb/scylla/issues?q=is%3Aissue+is%3Aopen+tombstone https://github.com/scylladb/scylla/issues?q=is%3Aissue+is%3A...
Luker88 6y ago
TL;DR: Pay lots of attention before choosing C* or scylla. only use the most basic stuff, don't work on near-realtime. More pifalls than the number of seconds those software have been out. Every time I see people talking about cassandra or scylla I am reminded of the huge pains and pitfalls of what feels like hacked-together solutions that were built on a very shaky theory. I worked only 2 years on systems that used cassandra and scylla (both on dev and ops side). About 7TB total. I admit they are very fast, provided you do proper continuous maintenance and that your application goes out of the way to conform to the database (absolutely not the other way around) The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting Anything else, and you are in the realm of cases and special cases. You can't trust errors, your operation might have been applied anyway. LWT terribly slow, limited and you should not run more than a very limited number of them. It's full of features that are added, then the cassandra guys realize that it really does not work, in both theory and practice, and than they deprecate those a few releases later. Compared to SQL, it feels worse than navigating blind on a minefield of special cases. The CQL language does not help. Only use the very basics of the language, otherwise it's a constant game of: This is the rule / With This exception / Unless you also have This / Except That / but it works again with X and on and on. It's like they had a very basic, specific use case, and went to build a ton of features that don't stack on top of it. Like adding full SQL on top of redis, but worse.
- thekozmo 6y ago
  LWT on Scylla is much faster than its C* implementation. As one of the co-founders, I agree that SQL and the relational model are superior than CQL. However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. The right way to use NoSQL is when you know your scaling model. LWT helps you to serialize important tables. Scylla consistently makes improvements in operations, consistency and functionality, so more to come, see what we'll announce at the summit next month
  - aphyr 6y ago
Rapzid 6y ago
I'm likely to bring this up every time I notice a Jepsen post on HN.. I still miss the flair from the old days Kyle :) I guess I'm nostalgic for it. Your posts were during a time when I was still on a steep trajectory in my career growth and discovery. Just soaking everything up, and it was FUN. And others in my "generation" seemed to also be having FUN, and having FUN with it. Things are just a bit more bland these days :|
- andreareina 6y ago
  Call-out to those fun days: https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/

172 points by aphyr 6y ago

Companion blog on ScyllaDB's site: https://www.scylladb.com/2020/12/23/jepsen-and-scylla-putting-consistency-to-the-test/ https://www.scylladb.com/2020/12/23/jepsen-and-scylla-puttin...

gred 6y ago

The wolf also shall dwell with the lamb, The leopard shall lie down with the young goat, The calf and the young lion and the fatling together; And a little child shall lead them. All popular databases shall routinely undergo Jepsen reviews; And none shall prohibit publication of test or benchmark results. Isaiah 11:6-7 (New Database Translation)

PeterCorless 6y ago
We worked closely with Kyle (@Aphyr) as he prepared this report. He was nothing but gracious with his time, brutal but fair with his feedback, and unrelenting in his commitment to discerning precisely where and how we strayed from the one true path of rectitudinous database behavior. Hat's off to him.
__s 6y ago
http://www.solidrockbaptist.net/satans-niv-bible.html http://www.solidrockbaptist.net/satans-niv-bible.html > Just a few examples should be enough to show you the problem. In the New International Version (NIV) it says in 2 Samuel 21:19 that someone other than David killed the giant Goliath. Anyone knows this is not true, except the 'scholars' who translated it. According to Hebrews 3:16 in the NIV all the children of Israel rebelled in the wilderness and none made it into the promised land; however, we know that Joshua and Caleb did not rebel, and God promised them each a place in the promised land (just read Numbers 13 & 14). Another mistake is found in Mark 1:2, 3 where according to the NIV the writer of Malachi was Isaiah! Just check out Isaiah 40:3 and Malachi 3:1 - the KJV has it right and the NIV has it wrong. For comparison of the verse this comment parodies, https://biblehub.com/isaiah/11-6.htm https://biblehub.com/isaiah/11-6.htm
- PeterCorless 6y ago
  So are you saying the Bible suffers from a stale read problem? Or are you asserting it be eventually consistent?
  - Rapzid 6y ago
    The bible is most certainly AP.
nn3 6y ago
Hell will freeze over before Jepsen runs on Oracle
- winrid 6y ago
  I would be surprised if it did poorly, however.

josephg 6y ago

I love Kyle's work with Jepsen. Has anyone done anything similar with filesystems? I'm imagining a filesystem mounted on a virtual device, and having the virtual device randomly suffer write faults and various other failures. And then a fuzzer asserting that all consistency requirements are still met. (All operations before a successful fsync should be stored, no lost writes or files, no unexpected garbage data, etc.)

Smaug123 6y ago
Relatedly, at the Midlands Grad School 2019 conference, John Hughes did an introductory course to QuickCheck in which one of the final exercises (meant to demonstrate the use of Haskell and QuickCheck to black-box test arbitrary binaries) was to model `fopen`, `fprintf`, `fclose` and friends in C. I never actually did it, but I guess that's one of the most comprehensive ways to find out how something behaves!
eis 6y ago
Yes filesystems are another big important part when it comes to the topic of data consistency. I'm not aware of a project similar to Jepsen for filesystems but there have been research papers like the one presented in this page that explore the guarantees of popular local filesystems: https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ Of course distributed filesystems are even harder but we can see that even writing to a local file is surprisingly complex and in many (most?) apps probably not really guaranteeing proper data consistency. Imho the whole Posix filesystem API is flawed to begin with and it would be great if a modern replacement would emerge.
- thekozmo 6y ago
  We did something in the neighborhood, check it out: https://www.scylladb.com/2016/02/16/fault-injection-filesystem-software-testing/ https://www.scylladb.com/2016/02/16/fault-injection-filesyst...
- redis_mlc 6y ago
  > I'm not aware of a project similar to Jepsen for filesystems Not as elaborate as Jepsen, but there has been some work: - Kirk McKusick's papers and work on BSD fs log semantics https://www.researchgate.net/scientific-contributions/Marshall-Kirk-McKusick-70507097 https://www.researchgate.net/scientific-contributions/Marsha... I believe the term to look for is "soft updates." - the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version. You can Bing versions of that by using quotes "fstest.c". - the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.) - Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines. - Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production. SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out. https://en.wikipedia.org/wiki/Dancing_tree https://en.wikipedia.org/wiki/Dancing_tree Source: DBA and storage engineer.

throwdbaaway 6y ago

> Insert is a destructive operation by design. In fact, writing any collection literal (e.g. {} or (1, 2)) is internally implemented by writing a deletion tombstone followed by the new values. These range tombstones are also evil. If you keep doing it to the same partition, even at a rather low QPS, you can easily bring down some nodes by keeping their CPU at 100%. Only tested this with Cassandra though, not Scylla.

thekozmo 6y ago
Correct but Jepsen focuses on consistency and correctness and not on destructive. Scylla is exposed to range tombstone issue like many other LSMs. We recently have been improving them - check the most recent issues: https://github.com/scylladb/scylla/issues?q=is%3Aissue+is%3Aopen+tombstone https://github.com/scylladb/scylla/issues?q=is%3Aissue+is%3A...

Luker88 6y ago

TL;DR: Pay lots of attention before choosing C* or scylla. only use the most basic stuff, don't work on near-realtime. More pifalls than the number of seconds those software have been out. Every time I see people talking about cassandra or scylla I am reminded of the huge pains and pitfalls of what feels like hacked-together solutions that were built on a very shaky theory. I worked only 2 years on systems that used cassandra and scylla (both on dev and ops side). About 7TB total. I admit they are very fast, provided you do proper continuous maintenance and that your application goes out of the way to conform to the database (absolutely not the other way around) The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting Anything else, and you are in the realm of cases and special cases. You can't trust errors, your operation might have been applied anyway. LWT terribly slow, limited and you should not run more than a very limited number of them. It's full of features that are added, then the cassandra guys realize that it really does not work, in both theory and practice, and than they deprecate those a few releases later. Compared to SQL, it feels worse than navigating blind on a minefield of special cases. The CQL language does not help. Only use the very basics of the language, otherwise it's a constant game of: This is the rule / With This exception / Unless you also have This / Except That / but it works again with X and on and on. It's like they had a very basic, specific use case, and went to build a ton of features that don't stack on top of it. Like adding full SQL on top of redis, but worse.

thekozmo 6y ago
LWT on Scylla is much faster than its C* implementation. As one of the co-founders, I agree that SQL and the relational model are superior than CQL. However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. The right way to use NoSQL is when you know your scaling model. LWT helps you to serialize important tables. Scylla consistently makes improvements in operations, consistency and functionality, so more to come, see what we'll announce at the summit next month
- aphyr 6y ago

Rapzid 6y ago

I'm likely to bring this up every time I notice a Jepsen post on HN.. I still miss the flair from the old days Kyle :) I guess I'm nostalgic for it. Your posts were during a time when I was still on a steep trajectory in my career growth and discovery. Just soaking everything up, and it was FUN. And others in my "generation" seemed to also be having FUN, and having FUN with it. Things are just a bit more bland these days :|

andreareina 6y ago
Call-out to those fun days: https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/