4 ms·

Not surprised to see other companies facing issues with Cassandra and tombstones. Don't get me wrong, I understand the need for tombstones in a distributed syst

by joaodlf 10y ago

Not surprised to see other companies facing issues with Cassandra and tombstones. Don't get me wrong, I understand the need for tombstones in a distributed system like Cassandra... It doesn't make it any less of a pain though :).

jjirsa 10y ago
The tombstone problem described is due to misuse - probably from improper use of prepared statements. Looks like they worked around it well.
- dwenzek 10y ago
  > The tombstone problem described is due to misuse. My concerns with Cassandra are precisely here: this is easy to misuse it. There are a lot of constraints on the schema (more specifically on the design of partition & clustering keys). Each choice leads to many restrictions on what can be requested/updated/removed; and to different issues with tombstones and GC. The Discord's story is exactly what I experimented: a series of surprises, and even really bad surprises in production. In both cases, the story ended with an efficient system, but with by far more engineering work and rework than initially planned.
  - krinchan 10y ago
    This article is extremely similar to almost every other "OMG! We used Cassandra and it was nothing like a SQL database!" article by Netflix, Spotify, and so many more. The fact that every single one contains the same 6 or 7 self-inflicted issues is pretty funny to me. I mean, I thought we lived in the age of the Google Dev. Cassandra does require you to know more of it's internals than most other data stores. Unfortunately, the move to CQL and very SQL-like names for things that are nothing at ALL like their SQL counterparts is not helping. Also, our own personal death by tombstone: A developer who didn't even know those existed checked in some logic that would write null into a column every time a thing succeeded. After that passed QA and went into production, all hell broke loose with queries timing out everywhere. SUCH FUN.
  - jjirsa 10y ago
    > My concerns with Cassandra are precisely here: this is easy to misuse it. You get this massively scalable (to thousands of nodes and tens of millions of operations per second) database for free, and all you have to do is have your developers read about it before they use it. Is that expecting too much? Let me ask this, then: It's an OSS project. Let's pretend I'm a committer or on the PMC, and I'd like to fix this in a way that works for you. We need a null write to act as a delete. We need tombstones to make sure things that are meant to be deleted stay deleted. We have to have all the tombstones we scan in heap on reads to reconcile what's deleted and what's not deleted within a partition on any given slice query. What would you want to see changed to avoid the tombstone problem? There are dozens of blogs around that say "dont write null values if you dont want a tombstone to be created" (like this article, or [0]), but beyond that, do you expect to see errors in logs? We've made unbound values in prepared statements no longer equivalent to null/delete [1]. What else would you expect an OSS project to do to protect you from abusing it? Serious question. 0: http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-prepared-statements-and-undesired-tombstone-creation.html http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-pr... 1: https://issues.apache.org/jira/browse/CASSANDRA-7304 https://issues.apache.org/jira/browse/CASSANDRA-7304
  - manigandham 10y ago
    These same surprises keep being written about for Cassandra. At this point reading a few blog posts and the documentation (especially about how the data is stored) will cover all the issues you might have in production. Basically - do the research first.

by joaodlf 10y ago

jjirsa 10y ago

The tombstone problem described is due to misuse - probably from improper use of prepared statements. Looks like they worked around it well.

dwenzek 10y ago
> The tombstone problem described is due to misuse. My concerns with Cassandra are precisely here: this is easy to misuse it. There are a lot of constraints on the schema (more specifically on the design of partition & clustering keys). Each choice leads to many restrictions on what can be requested/updated/removed; and to different issues with tombstones and GC. The Discord's story is exactly what I experimented: a series of surprises, and even really bad surprises in production. In both cases, the story ended with an efficient system, but with by far more engineering work and rework than initially planned.
- krinchan 10y ago
  This article is extremely similar to almost every other "OMG! We used Cassandra and it was nothing like a SQL database!" article by Netflix, Spotify, and so many more. The fact that every single one contains the same 6 or 7 self-inflicted issues is pretty funny to me. I mean, I thought we lived in the age of the Google Dev. Cassandra does require you to know more of it's internals than most other data stores. Unfortunately, the move to CQL and very SQL-like names for things that are nothing at ALL like their SQL counterparts is not helping. Also, our own personal death by tombstone: A developer who didn't even know those existed checked in some logic that would write null into a column every time a thing succeeded. After that passed QA and went into production, all hell broke loose with queries timing out everywhere. SUCH FUN.
- jjirsa 10y ago
  > My concerns with Cassandra are precisely here: this is easy to misuse it. You get this massively scalable (to thousands of nodes and tens of millions of operations per second) database for free, and all you have to do is have your developers read about it before they use it. Is that expecting too much? Let me ask this, then: It's an OSS project. Let's pretend I'm a committer or on the PMC, and I'd like to fix this in a way that works for you. We need a null write to act as a delete. We need tombstones to make sure things that are meant to be deleted stay deleted. We have to have all the tombstones we scan in heap on reads to reconcile what's deleted and what's not deleted within a partition on any given slice query. What would you want to see changed to avoid the tombstone problem? There are dozens of blogs around that say "dont write null values if you dont want a tombstone to be created" (like this article, or [0]), but beyond that, do you expect to see errors in logs? We've made unbound values in prepared statements no longer equivalent to null/delete [1]. What else would you expect an OSS project to do to protect you from abusing it? Serious question. 0: http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-prepared-statements-and-undesired-tombstone-creation.html http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-pr... 1: https://issues.apache.org/jira/browse/CASSANDRA-7304 https://issues.apache.org/jira/browse/CASSANDRA-7304
- manigandham 10y ago
  These same surprises keep being written about for Cassandra. At this point reading a few blog posts and the documentation (especially about how the data is stored) will cover all the issues you might have in production. Basically - do the research first.