5 ms·

TL;DR: Pay lots of attention before choosing C* or scylla. only use the most basic stuff, don't work on near-realtime. More pifalls than the number of seconds t

TL;DR: Pay lots of attention before choosing C* or scylla. only use the most basic stuff, don't work on near-realtime. More pifalls than the number of seconds those software have been out. Every time I see people talking about cassandra or scylla I am reminded of the huge pains and pitfalls of what feels like hacked-together solutions that were built on a very shaky theory. I worked only 2 years on systems that used cassandra and scylla (both on dev and ops side). About 7TB total. I admit they are very fast, provided you do proper continuous maintenance and that your application goes out of the way to conform to the database (absolutely not the other way around) The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting Anything else, and you are in the realm of cases and special cases. You can't trust errors, your operation might have been applied anyway. LWT terribly slow, limited and you should not run more than a very limited number of them. It's full of features that are added, then the cassandra guys realize that it really does not work, in both theory and practice, and than they deprecate those a few releases later. Compared to SQL, it feels worse than navigating blind on a minefield of special cases. The CQL language does not help. Only use the very basics of the language, otherwise it's a constant game of: This is the rule / With This exception / Unless you also have This / Except That / but it works again with X and on and on. It's like they had a very basic, specific use case, and went to build a ton of features that don't stack on top of it. Like adding full SQL on top of redis, but worse.

thekozmo 6y ago
LWT on Scylla is much faster than its C* implementation. As one of the co-founders, I agree that SQL and the relational model are superior than CQL. However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. The right way to use NoSQL is when you know your scaling model. LWT helps you to serialize important tables. Scylla consistently makes improvements in operations, consistency and functionality, so more to come, see what we'll announce at the summit next month
- aphyr 6y ago
  > However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. There's something to be said about CQL specifically here, because its transactional model is an odd duck, and not always for scalability reasons. Scylla and Cassandra could offer richer transactions without significant performance penalties--indeed, which would significantly speed up some types of transactional workloads, but instead there are, as the grandparent poster notes, strange edge cases. For instance, you can't select multiple CQL rows where the select would cover a column backed by a CQL collection. That's been fixed in Cassandra, but is still present in Scylla, and limited the kinds of tests we could write for Scylla. You also can't perform an LWT write without a guard clause: linearizable upserts in the Jepsen tests require the presence of an always-null field in every row whose only purpose is to convince Scylla that yes, we really would like to use Paxos for this write. Likewise, there's no logical notion of a batch combining reads and writes, or, for that matter, a batch of reads. If you want those sorts of things in C* or Scylla, I imagine one winds up stringing together reads and CaS statements in retry loops, or something to that effect. We were trying to hack together a batch read via a no-op write, except that it's actually impossible to do that robustly--you can only "read" values involving guard clauses, and there's no guard which will pass on all values. It's just... awkward. This isn't a scalability thing--all three of these cases could be executed in a single Paxos round, scoped to a single partition. It's just... weird API design.
  - thekozmo 6y ago
    It's possible to improve cql to a cql++ .. if you start going this path, it will lead to sql which isn't bad a all ;) It's not 'fair' to compare CQL to SQL, if you compare CQL to DynamoDB's http api, which scylla implements as well, you'll see that cql is better: https://www.scylladb.com/2020/05/12/comparing-cql-and-the-dynamodb-api/ https://www.scylladb.com/2020/05/12/comparing-cql-and-the-dy...
    - 411111111111111 6y ago
      > It's not 'fair' to compare CQL to SQL He didn't do that. He pointed out issues with the cql syntax and specifically scylla's implementation of it. he is also uniquely qualified to make such a statement, as his Jepsen Reports are high quality analyses looking at various databases. But to address your other claim: they solve the same problem of providing data to your application. They should be compared if that's the decision you have to make.
cpleppert 6y ago
> The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting 1000x this That is exactly what Cassandra was originally designed to do. Cassandra doesn't implement complex distributed mechanisms that would make it hard to operate and implement like most distributed databases or impose a cost on application developers to understand the tradeoffs they will have to grapple with and make the. Cassandra is not AP with less than a quorum read/write or CP with a quorum read/write. It only fulfills those guarantees in the way you might expect if your cluster topology never changes and your timestamps are carefully synchronized. Absent these guarantees modifying existing data is essentially an undefined operation. This isn't a shot at cassandra developers. Cassandra isn't magic. There is no possible way it can magically use a simple last write wins strategy to guarantee that data updates won't be lost in the absence of outside guarantees. Cassandra is essentially a prototype of a class of NoSQL databases try to be incredibly easy to manage for write heavy workloads. Look, I get it, it was a collection of appealing technologies, I found the original white paper on Cassandra interesting too. columnar read-repair eventually consistent merkle trees NoSQL column family super columns cluster gossip phi accrual failure detector Of course, the 'columnar' attribute was originally intended so that applications designed around cassandra could reconcile multiple values associated with a single key:attribute path without having to utilize a complex api. Whether this is actually viable is another story entirely. CQL's only advantage is that it vaguely resembles SQL. CQL is not a query language. It is a collection of Cassandra operators mapped onto a vaguely SQLish syntax that bears little relation to the underlying semantics they are derived from. Cassandra just shoehorns a bunch of its idiosyncrasies onto an SQL look alike. INSERT and UPDATE do the same thing(which is not what SQL does for either and are probably not what you expect). The entire behavior of how a 'query' is actually executed can change drastically. In order to use Cassandra correctly you have to understand how the database physically implements every operation. So the query language doesn't provide a meaningful abstraction at all. CQL would have never made sense in the original implementation of cassandra because cassandra was never designed to do the things that people expect from a database that implements a query language.
- alexott 6y ago
kostja 6y ago
We at Scylla are perhaps better than anyone else aware of the Cassandra legacy we inherited. However, this is not what this analysis demonstrates. It shows we're working on making the great ideas of Cassandra work well, and bad ideas become irrelevant. Should anyone make the judgement about Scylla based on their prior negative experience with Cassandra? I think no.
- Luker88 6y ago
  Worked with both, only on the opensource version. I agree that scylla shows a lot more promise than C, but we hit (different) instabilities and gotchas on both All has its place. I am not saying C or scylla should not be used. I was just pointing out that the basic ideas and the CQL language itself is designed to trick you in in doing a lot of things that turn out to be really wrong on the data model used by these software. It's not an implementation thing. Should I consider only that, scylla'd be really good (provided you have the correct hardware) and much better than C*. Still, the issue (IMHO) is with the theory and the fact that the actual correct applications are much, much narrower than the initial impact lets you believe (especially due to CQL) --edit: Kudos on scylla btw, but basically I am doubtful how much it's possible to overcome that legacy without going really incompatible even on the API

thekozmo 6y ago

LWT on Scylla is much faster than its C* implementation. As one of the co-founders, I agree that SQL and the relational model are superior than CQL. However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. The right way to use NoSQL is when you know your scaling model. LWT helps you to serialize important tables. Scylla consistently makes improvements in operations, consistency and functionality, so more to come, see what we'll announce at the summit next month

aphyr 6y ago
> However, either they do not scale or even when they do (newsql), they are 10x slower and handle less volume, etc. There's something to be said about CQL specifically here, because its transactional model is an odd duck, and not always for scalability reasons. Scylla and Cassandra could offer richer transactions without significant performance penalties--indeed, which would significantly speed up some types of transactional workloads, but instead there are, as the grandparent poster notes, strange edge cases. For instance, you can't select multiple CQL rows where the select would cover a column backed by a CQL collection. That's been fixed in Cassandra, but is still present in Scylla, and limited the kinds of tests we could write for Scylla. You also can't perform an LWT write without a guard clause: linearizable upserts in the Jepsen tests require the presence of an always-null field in every row whose only purpose is to convince Scylla that yes, we really would like to use Paxos for this write. Likewise, there's no logical notion of a batch combining reads and writes, or, for that matter, a batch of reads. If you want those sorts of things in C* or Scylla, I imagine one winds up stringing together reads and CaS statements in retry loops, or something to that effect. We were trying to hack together a batch read via a no-op write, except that it's actually impossible to do that robustly--you can only "read" values involving guard clauses, and there's no guard which will pass on all values. It's just... awkward. This isn't a scalability thing--all three of these cases could be executed in a single Paxos round, scoped to a single partition. It's just... weird API design.
- thekozmo 6y ago
  It's possible to improve cql to a cql++ .. if you start going this path, it will lead to sql which isn't bad a all ;) It's not 'fair' to compare CQL to SQL, if you compare CQL to DynamoDB's http api, which scylla implements as well, you'll see that cql is better: https://www.scylladb.com/2020/05/12/comparing-cql-and-the-dynamodb-api/ https://www.scylladb.com/2020/05/12/comparing-cql-and-the-dy...
  - 411111111111111 6y ago
    > It's not 'fair' to compare CQL to SQL He didn't do that. He pointed out issues with the cql syntax and specifically scylla's implementation of it. he is also uniquely qualified to make such a statement, as his Jepsen Reports are high quality analyses looking at various databases. But to address your other claim: they solve the same problem of providing data to your application. They should be compared if that's the decision you have to make.

cpleppert 6y ago

> The only safe use of these 2 software is basically: 1) add new unique rows (no updates) 2) much later (granted only after repairs, which take a real lot) you can read, but only up to a certain time in the past. 3) basically almost avoid deleting 1000x this That is exactly what Cassandra was originally designed to do. Cassandra doesn't implement complex distributed mechanisms that would make it hard to operate and implement like most distributed databases or impose a cost on application developers to understand the tradeoffs they will have to grapple with and make the. Cassandra is not AP with less than a quorum read/write or CP with a quorum read/write. It only fulfills those guarantees in the way you might expect if your cluster topology never changes and your timestamps are carefully synchronized. Absent these guarantees modifying existing data is essentially an undefined operation. This isn't a shot at cassandra developers. Cassandra isn't magic. There is no possible way it can magically use a simple last write wins strategy to guarantee that data updates won't be lost in the absence of outside guarantees. Cassandra is essentially a prototype of a class of NoSQL databases try to be incredibly easy to manage for write heavy workloads. Look, I get it, it was a collection of appealing technologies, I found the original white paper on Cassandra interesting too. columnar read-repair eventually consistent merkle trees NoSQL column family super columns cluster gossip phi accrual failure detector Of course, the 'columnar' attribute was originally intended so that applications designed around cassandra could reconcile multiple values associated with a single key:attribute path without having to utilize a complex api. Whether this is actually viable is another story entirely. CQL's only advantage is that it vaguely resembles SQL. CQL is not a query language. It is a collection of Cassandra operators mapped onto a vaguely SQLish syntax that bears little relation to the underlying semantics they are derived from. Cassandra just shoehorns a bunch of its idiosyncrasies onto an SQL look alike. INSERT and UPDATE do the same thing(which is not what SQL does for either and are probably not what you expect). The entire behavior of how a 'query' is actually executed can change drastically. In order to use Cassandra correctly you have to understand how the database physically implements every operation. So the query language doesn't provide a meaningful abstraction at all. CQL would have never made sense in the original implementation of cassandra because cassandra was never designed to do the things that people expect from a database that implements a query language.

alexott 6y ago

kostja 6y ago

We at Scylla are perhaps better than anyone else aware of the Cassandra legacy we inherited. However, this is not what this analysis demonstrates. It shows we're working on making the great ideas of Cassandra work well, and bad ideas become irrelevant. Should anyone make the judgement about Scylla based on their prior negative experience with Cassandra? I think no.

Luker88 6y ago
Worked with both, only on the opensource version. I agree that scylla shows a lot more promise than C, but we hit (different) instabilities and gotchas on both All has its place. I am not saying C or scylla should not be used. I was just pointing out that the basic ideas and the CQL language itself is designed to trick you in in doing a lot of things that turn out to be really wrong on the data model used by these software. It's not an implementation thing. Should I consider only that, scylla'd be really good (provided you have the correct hardware) and much better than C*. Still, the issue (IMHO) is with the theory and the fact that the actual correct applications are much, much narrower than the initial impact lets you believe (especially due to CQL) --edit: Kudos on scylla btw, but basically I am doubtful how much it's possible to overcome that legacy without going really incompatible even on the API