4 ms·

> far easier to run than Cassandra I know precisely nothing about Scylla, but somehow I still agree with this. Cassandra is far and away the most horrendous so

by samhw 4y ago

> far easier to run than Cassandra I know precisely nothing about Scylla, but somehow I still agree with this. Cassandra is far and away the most horrendous software I've ever had to work with (Kafka coming in a close second). This was true even when we hired a major core contributor to help manage it; even he wasn't 100% comfortable with it. The word 'anticompaction' is still enough to give me nightmares. I very much welcome – not that it's Cassandra's only problem – this trend of rewriting '00s/early-'10s Java software in simpler languages. (And, languages aside, I think Cassandra - certainly inter alia - simply sits at a level of complexity which is beyond the comprehension of any one human being. And software which is beyond the comprehension of any one human being inevitably begets bugs, even as an operator rather than a developer.)

_benedict 4y ago
I'm intrigued: The Apache Cassandra developer community is quite small and well known to each other, and the only major contributor I know to have left the community to run a cluster did so almost a decade ago? Cassandra had a lot of rough edges until very recently, and was only really suitable for sophisticated users, as most contributors would have attested. The project has matured a lot over the past couple of years in particular, as the broader community stepped up in the face of DataStax's (temporary) withdrawal. The investment by some large scale users has transformed it, and the next couple of years will do so even more. None of the issues the project faced were really related to the language chosen, in my opinion, rather than the maturity of the project and how it grew at the time.
- samhw 4y ago
  His initials were LT, if that helps (if not, I can clarify his whole name over email or however people privately communicate on HN?). I may well be exaggerating his 'major'ness - I'm not that familiar with the Cassandra contributorship, but that was my understanding from my team! As for Cassandra itself, we were certainly quite sophisticated users. There's a decent chance you'd know the company in question. It's a fair point about the language: I'm not a fan of Java and it probably colours my opinion a little; I was speaking more from a philosophical standpoint about reducing the theoretical complexity of software to make it more deterministic & understandable (out of the tar pit and all that), more than to any specific deficiencies in Java that caused any actual issues for us, of which there were no direct examples I can recall. (Pathological GC did cause some occasional degradation, I suppose, though at best that's semi-specific to Java.) I think most of the actual issues, from my on-call years, were as a result of stuff like: (as I mentioned) anticompaction and suchlike causing pathological performance; our own misconfiguration of things like asynchronous replication / bootstrapping (which once caused a very severe incident, as in 'endemic data corruption and loss' severity); application-layer issues from product engineers misconfiguring consistency, choosing poor keys for partitioning, constructing poor data models that require table scans, all the ordinary stuff for which Cassandra is at most very obliquely to blame. Also, I agree it was stupid of us to use Cassandra in (what was) a very serious environment, in probably one of the most safety-critical sectors outside of medicine and rockets. We knew that. We did the same with several technologies. Literally, we had a diktat saying no engineers could mention it in blog posts. On reflection it's quite unfair to blame Cassandra for our decision - to a large extent, yeah, we were holding it very wrong. I would not have made that choice myself, at all.
  - _benedict 4y ago
    I probably know who you are referring to, but I don't feel it would be appropriate to discuss a specific identifiable individual in a public forum, no matter how innocuous that discussion might be! My fault for bring it up. Cassandra was super easy to shoot yourself in the foot with, and it remains quite easy. You mention a few foot guns that have gotten better, a few that remain but that will get better, and a few that are sort of inherent to distributed databases. Anti-compaction for instance should now not be a huge issue if you're running regular incremental repairs, and I hope even the few remaining caveats will be alleviated soon. Bootstrap is something you mention that is also going to get much easier for users this coming year, so that unsophisticated users can manage their cluster membership safely. Application-misconfigured consistency levels is a really obvious one that isn't strictly Cassandra's fault but for which much better help could be given to the user, and I expect some major improvements here in the next year or two, so that users can configure tables with consistency properties that the database guarantees (to some extent, the user will always be able to screw it up by providing the wrong consistency identifier, but at least the scope is reduced to accidental misconfiguration rather than misunderstanding). This is something we're considering as part of the introduction of general purpose transactions later this year. Poor data models and partition keys are things the database can offer less help with, though I anticipate much better support for ordered partitioning in future, that would help poorly-selected partition keys, as clustering keys can be used for partitioning there too. Regarding the choice of language, Java has upsides and downsides. GC spirals are something we have control over at the end of the day, and we continue to do better at (as does the JVM), but guaranteeing no segfaults (and not worrying about the ABA problem) is a big benefit we get in return. I wish we had more control over things like memory placement and execution, but these things may be coming to the JVM to some extent (Loom I expect to benefit Cassandra hugely, and value types later), but equally distributed systems problems often give you enough things to worry about. The visibility you have into a Java process is fantastic, however, and we are starting to make use of the ease with which you can modify the code Java runs for system validation, using byte weaving to permit us to simulate clusters as they are run, with adversarial event orderings, to ensure those notoriously hard distributed systems problems are correctly solved. If you do want to speak privately, about anything Cassandra related, the lowercase part of my username (i.e. without the _ prefix) at apache.org reaches me.
manigandham 4y ago
The underlying Dynamo architecture is great, and original Facebook and community releases did deliver on the overall promises. Unfortunately the implementations suffered, partly because of the language and baggage from Java and from the execution model. Scylla using C++ helped in bringing the low-level planning and development experience that is common in that area (and harder to do in Java), but also the ability to use all the low-level libraries like DPDK and advanced threading/reactor systems (like the underlying Seastar framework that Scylla made). The opensource Cassandra versions are now implementing the same model, although still using Java.
samhw 4y ago
NOTE: I can't now edit this comment, but I wanted to clarify, per below, that many of these issues are more about my experience of Cassandra in the specific context in which I experienced it, and not the database itself. It's a complex tool, but, for the constituency of users for whom it's appropriate, it's as competently architected as just about any other database. My criticisms mostly centre on the culture of engineering teams which choose to use a very complicated tool that requires significant skill, in contexts where it's simply not needed.
jjirsa 4y ago
> I think Cassandra - certainly inter alia - simply sits at a level of complexity which is beyond the comprehension of any one human being I think there are a handful of humans that actually get it. But it is a true handful, not more than 5.

by samhw 4y ago

_benedict 4y ago

I'm intrigued: The Apache Cassandra developer community is quite small and well known to each other, and the only major contributor I know to have left the community to run a cluster did so almost a decade ago? Cassandra had a lot of rough edges until very recently, and was only really suitable for sophisticated users, as most contributors would have attested. The project has matured a lot over the past couple of years in particular, as the broader community stepped up in the face of DataStax's (temporary) withdrawal. The investment by some large scale users has transformed it, and the next couple of years will do so even more. None of the issues the project faced were really related to the language chosen, in my opinion, rather than the maturity of the project and how it grew at the time.

samhw 4y ago
His initials were LT, if that helps (if not, I can clarify his whole name over email or however people privately communicate on HN?). I may well be exaggerating his 'major'ness - I'm not that familiar with the Cassandra contributorship, but that was my understanding from my team! As for Cassandra itself, we were certainly quite sophisticated users. There's a decent chance you'd know the company in question. It's a fair point about the language: I'm not a fan of Java and it probably colours my opinion a little; I was speaking more from a philosophical standpoint about reducing the theoretical complexity of software to make it more deterministic & understandable (out of the tar pit and all that), more than to any specific deficiencies in Java that caused any actual issues for us, of which there were no direct examples I can recall. (Pathological GC did cause some occasional degradation, I suppose, though at best that's semi-specific to Java.) I think most of the actual issues, from my on-call years, were as a result of stuff like: (as I mentioned) anticompaction and suchlike causing pathological performance; our own misconfiguration of things like asynchronous replication / bootstrapping (which once caused a very severe incident, as in 'endemic data corruption and loss' severity); application-layer issues from product engineers misconfiguring consistency, choosing poor keys for partitioning, constructing poor data models that require table scans, all the ordinary stuff for which Cassandra is at most very obliquely to blame. Also, I agree it was stupid of us to use Cassandra in (what was) a very serious environment, in probably one of the most safety-critical sectors outside of medicine and rockets. We knew that. We did the same with several technologies. Literally, we had a diktat saying no engineers could mention it in blog posts. On reflection it's quite unfair to blame Cassandra for our decision - to a large extent, yeah, we were holding it very wrong. I would not have made that choice myself, at all.
- _benedict 4y ago
  I probably know who you are referring to, but I don't feel it would be appropriate to discuss a specific identifiable individual in a public forum, no matter how innocuous that discussion might be! My fault for bring it up. Cassandra was super easy to shoot yourself in the foot with, and it remains quite easy. You mention a few foot guns that have gotten better, a few that remain but that will get better, and a few that are sort of inherent to distributed databases. Anti-compaction for instance should now not be a huge issue if you're running regular incremental repairs, and I hope even the few remaining caveats will be alleviated soon. Bootstrap is something you mention that is also going to get much easier for users this coming year, so that unsophisticated users can manage their cluster membership safely. Application-misconfigured consistency levels is a really obvious one that isn't strictly Cassandra's fault but for which much better help could be given to the user, and I expect some major improvements here in the next year or two, so that users can configure tables with consistency properties that the database guarantees (to some extent, the user will always be able to screw it up by providing the wrong consistency identifier, but at least the scope is reduced to accidental misconfiguration rather than misunderstanding). This is something we're considering as part of the introduction of general purpose transactions later this year. Poor data models and partition keys are things the database can offer less help with, though I anticipate much better support for ordered partitioning in future, that would help poorly-selected partition keys, as clustering keys can be used for partitioning there too. Regarding the choice of language, Java has upsides and downsides. GC spirals are something we have control over at the end of the day, and we continue to do better at (as does the JVM), but guaranteeing no segfaults (and not worrying about the ABA problem) is a big benefit we get in return. I wish we had more control over things like memory placement and execution, but these things may be coming to the JVM to some extent (Loom I expect to benefit Cassandra hugely, and value types later), but equally distributed systems problems often give you enough things to worry about. The visibility you have into a Java process is fantastic, however, and we are starting to make use of the ease with which you can modify the code Java runs for system validation, using byte weaving to permit us to simulate clusters as they are run, with adversarial event orderings, to ensure those notoriously hard distributed systems problems are correctly solved. If you do want to speak privately, about anything Cassandra related, the lowercase part of my username (i.e. without the _ prefix) at apache.org reaches me.

manigandham 4y ago

The underlying Dynamo architecture is great, and original Facebook and community releases did deliver on the overall promises. Unfortunately the implementations suffered, partly because of the language and baggage from Java and from the execution model. Scylla using C++ helped in bringing the low-level planning and development experience that is common in that area (and harder to do in Java), but also the ability to use all the low-level libraries like DPDK and advanced threading/reactor systems (like the underlying Seastar framework that Scylla made). The opensource Cassandra versions are now implementing the same model, although still using Java.

samhw 4y ago

NOTE: I can't now edit this comment, but I wanted to clarify, per below, that many of these issues are more about my experience of Cassandra in the specific context in which I experienced it, and not the database itself. It's a complex tool, but, for the constituency of users for whom it's appropriate, it's as competently architected as just about any other database. My criticisms mostly centre on the culture of engineering teams which choose to use a very complicated tool that requires significant skill, in contexts where it's simply not needed.

jjirsa 4y ago

> I think Cassandra - certainly inter alia - simply sits at a level of complexity which is beyond the comprehension of any one human being I think there are a handful of humans that actually get it. But it is a true handful, not more than 5.