7 ms·

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassandra-cluster-using-yelps-data-pipeline.html#data-cor

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassandra-cluster-using-yelps-data-pipeline.html#data-corruption-overview https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can shed light on this? There seem to be plenty of people using Cassandra at scale -- is constantly repairing it normal practice?

CHY872 3y ago
I have >5 years of experience using Cassandra in production, involving thousands of clusters storing petabytes of data. My conclusion from that time is that Cassandra is simply not robust enough to be a general purpose database (the team are working on it but they're coming from a really rough starting place) - there are lots of ways to cause data corruption, and Cassandra does enough dynamic repairing that it can be hard to catch this before your backups are dropped due to time windowing. Unfortunately, the juice may still be worth the squeeze - Cassandra's storage model lends itself very nicely to disaster recovery workflows in a way which something like Oracle or FoundationDB does not (and it's Cassandra so you'll need it!), while the ability to horizontally scale gets you out of so many operations issues. If you've got a schema which works well in Cassandra, you've probably solved a lot of the issues you might have. Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back. If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs. Example of other Cassandra bug (again, might be outdated): Cassandra nodes identify themselves on startups with IPs, and the owned token ranges are not persisted, they're streamed from other nodes in the cluster. If you've deployed your Cassandra in K8s and you reboot multiple nodes in one go and they swap IPs upon reboot, you may now find yourself in a split brain situation in which nodes magically forget they own certain data ranges and think they own each others data (or maybe it's that the nodes still think they own the right ranges but other nodes think they own the wrong ranges). Wasn't close enough to fully debug that one. It's a mess. Would seek to avoid problem spaces where I might need to use it again, though if by chance ended up in a space where it made sense, probably wouldn't avoid the tech.
- hardwaresofton 3y ago
  Thanks for this insight -- this is one of the first time I've heard of someone constrasting FoundationDB and Cassandra which is nice. > Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back. Interesting, seems like there is a bunch of little knowledge like this needed to run a service properly... Managed Cassandra has more added value to provide I guess. > If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs. This seems... really bad -- I don't think I have the skill to run a Cassandra cluster (and not enough use cases to run it as a hobby to find these edges)... This sounds like the space for a consultancy to make a tidy killing though.
t90fan 3y ago
Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very not realisable, would consume huge volumes of memory (especially during repairs), bandwidth (doing a repair is very chatty as it has to sync lots of data) and disk space (tombstoning meant deleted records take up space until compaction runs), and was generally not much fun to manage, and it was difficult to hire people who knew much about it to do so. I would not build a solution myself using it going forward. We also had to periodically (weekly) do "full" repairs to work around Cassandra bugs, silent data corruption etc...
- TideAd 3y ago
  Scaling up also takes up a lot of resources so you're never able to scale up in response to load without hosing your database even more.
- rickette 3y ago
  Can recommend https://cassandra-reaper.io/ https://cassandra-reaper.io/ for most of the management stuff you're mentioning. Still not free though, running Cassandra requires (some) effort in my experience.
- hardwaresofton 3y ago
  Thanks for sharing your experience -- I know I've spent a lot of time in the past worrying about FS corruption, but generally expecting that the database sitting on top of it should never get corrupted, mostly because I use postgres so much. I don't have the experience you do in this situation, but my first reaction to this was definitely "don't use Cassandra". But I also never really understood the use-case where Cassandra shines as a solution either (seems like only companies with a lot of data really seem to get wins from it?)
- darkstar_16 3y ago
  We run a Cassandra cluster in production and its a pretty small cluster yet all that you mentioned seems to resonate. We do use Cassandra reaper to automate some of the tasks but no one wants to touch Cassandra in general in the team.
pbreit 3y ago
One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?
- avereveard 3y ago
  Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that age looked at the numbers and went yeah not in a database.
  - onlypositive 3y ago
    Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.html https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on one db? Nothing says you need to be able to join on the restaurant table and the comments table. Put them on different servers. That's all you're doing with Casandra anyway.
    - tomnipotent 3y ago
      Postgres didn't ship built-in replication until 2010, and prior replication solutions like Slony were not options I would have enjoyed building a business around.
      - onlypositive 3y ago
        MySQL shipped replication in 2000. This whole conversation is some webscale copium bullshit. I'm out.

CHY872 3y ago

I have >5 years of experience using Cassandra in production, involving thousands of clusters storing petabytes of data. My conclusion from that time is that Cassandra is simply not robust enough to be a general purpose database (the team are working on it but they're coming from a really rough starting place) - there are lots of ways to cause data corruption, and Cassandra does enough dynamic repairing that it can be hard to catch this before your backups are dropped due to time windowing. Unfortunately, the juice may still be worth the squeeze - Cassandra's storage model lends itself very nicely to disaster recovery workflows in a way which something like Oracle or FoundationDB does not (and it's Cassandra so you'll need it!), while the ability to horizontally scale gets you out of so many operations issues. If you've got a schema which works well in Cassandra, you've probably solved a lot of the issues you might have. Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back. If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs. Example of other Cassandra bug (again, might be outdated): Cassandra nodes identify themselves on startups with IPs, and the owned token ranges are not persisted, they're streamed from other nodes in the cluster. If you've deployed your Cassandra in K8s and you reboot multiple nodes in one go and they swap IPs upon reboot, you may now find yourself in a split brain situation in which nodes magically forget they own certain data ranges and think they own each others data (or maybe it's that the nodes still think they own the right ranges but other nodes think they own the wrong ranges). Wasn't close enough to fully debug that one. It's a mess. Would seek to avoid problem spaces where I might need to use it again, though if by chance ended up in a space where it made sense, probably wouldn't avoid the tech.

hardwaresofton 3y ago
Thanks for this insight -- this is one of the first time I've heard of someone constrasting FoundationDB and Cassandra which is nice. > Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back. Interesting, seems like there is a bunch of little knowledge like this needed to run a service properly... Managed Cassandra has more added value to provide I guess. > If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs. This seems... really bad -- I don't think I have the skill to run a Cassandra cluster (and not enough use cases to run it as a hobby to find these edges)... This sounds like the space for a consultancy to make a tidy killing though.

t90fan 3y ago

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very not realisable, would consume huge volumes of memory (especially during repairs), bandwidth (doing a repair is very chatty as it has to sync lots of data) and disk space (tombstoning meant deleted records take up space until compaction runs), and was generally not much fun to manage, and it was difficult to hire people who knew much about it to do so. I would not build a solution myself using it going forward. We also had to periodically (weekly) do "full" repairs to work around Cassandra bugs, silent data corruption etc...

TideAd 3y ago
Scaling up also takes up a lot of resources so you're never able to scale up in response to load without hosing your database even more.
rickette 3y ago
Can recommend https://cassandra-reaper.io/ https://cassandra-reaper.io/ for most of the management stuff you're mentioning. Still not free though, running Cassandra requires (some) effort in my experience.
hardwaresofton 3y ago
Thanks for sharing your experience -- I know I've spent a lot of time in the past worrying about FS corruption, but generally expecting that the database sitting on top of it should never get corrupted, mostly because I use postgres so much. I don't have the experience you do in this situation, but my first reaction to this was definitely "don't use Cassandra". But I also never really understood the use-case where Cassandra shines as a solution either (seems like only companies with a lot of data really seem to get wins from it?)
darkstar_16 3y ago
We run a Cassandra cluster in production and its a pretty small cluster yet all that you mentioned seems to resonate. We do use Cassandra reaper to automate some of the tasks but no one wants to touch Cassandra in general in the team.

pbreit 3y ago

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

avereveard 3y ago
Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that age looked at the numbers and went yeah not in a database.
- onlypositive 3y ago
  Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.html https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on one db? Nothing says you need to be able to join on the restaurant table and the comments table. Put them on different servers. That's all you're doing with Casandra anyway.
  - tomnipotent 3y ago
    Postgres didn't ship built-in replication until 2010, and prior replication solutions like Slony were not options I would have enjoyed building a business around.
    - onlypositive 3y ago
      MySQL shipped replication in 2000. This whole conversation is some webscale copium bullshit. I'm out.