4 ms·

I haven't used Cassandra for about 3 years, and this is awakening memories. At a previous company I inherited a very badly assembled cluster that was used for a

I haven't used Cassandra for about 3 years, and this is awakening memories. At a previous company I inherited a very badly assembled cluster that was used for a time series database. The guys who built it said (and no, I'm not kidding...) "we don't need to put a TTL on metrics because they're tiny and anyway we can just add more nodes and scale the cluster horizontally forever!". Well, forever was about 2 years, when the physical data center ran out of rack space, and teams abused the metrics system with TBs of bullshit data. That was when they handed the whole metrics system to yours truly. And I discover two things: 1. You can't just bulk delete a year of old stale data without breaking it 2. "Woops, did we really set replication factor to 1?" Fun.

vorpalhex 4y ago
What was the issue with the bulk deletes? Locking the cpu?
- raffraffraff 4y ago
  [sorry if you're reading in real-time, lots of edits] It's mentioned briefly in the original article. Cassandra achieves extremely good write performance by being append-only, which means that is disk IO patterns are sequential, with little random IO. This gives great performance on crappy spinning disks. However, with append-only, updating or deleting existing data is handled by writing new data or "tombstones". When you perform a search, Cassandra still scans all matching data in the table, including that which was deleted. It then uses tombstones to "ignore" deleted data. In other words, lazy delete is great for write performance (Cassandra is optimized for write), but creates overhead during read. So to avoid killing the cluster during a search, Cassandra imposes a tombstone limit. If a search contains more tombstones then the limit, the search will be abandoned. Real world example: if you're using Cassandra as a time series database and you don't add a sensible TTL to your metrics, a bulk delete would cause searches to suddenly fail, and they wouldn't work again until after compaction is performed on the table, rewriting the whole table minus data that has tombstones older than gc grace period. Metrics systems aren't just used for eye-baling dashboards during an incident, they're part of your alerting system (usually) so this could lead to missed alerts or false positives. This is why it's important to continually "age-out" data with a TTL, so that compactions are regularly removing tombstones older than the gc grace period. "Fucking gc grace period!" (me, on a few occasions) Why is there a gc grace period in here, making trouble for us? Because Cassandra is a distributed, replicated system that handles partitions and failures, it knows that some nodes may be temporarily unreachable. So it holds on to tombstones long enough to ensure that every node gets them. If it didn't do this, an offline mode may not get the tombstone before it's compacted out of existence in all other nodes. So when it comes back online you get zombie deleted data, back from the dead! Not nice! If you fuck this up and end up with a huge set of old data that you must delete, you are in for a treat. Easiest option is to manually schedule a bunch of operations to run during a quiet period. With metrics, write traffic volume is pretty linear but read traffic is user-driven. That could mean that quiet period are nightly, on weekends etc. The operations involve checking for downed nodes, modifying gc grace period, deleting a bunch of data and running a compaction. You have to carefully monitor logs for errors, and keep an eye on tombstone limits. Depending on your infrastructure it might be easier to build a new cluster side-by-side, write all the data from the old cluster into the new one with proper TTLs and delete the old cluster. If you're on bare metal and are out of space, you might have to get creative. (To keep a production metrics system running for a few years I had to get extremely creative) Sorry for the huge thread and possibly wrong info, it's a while since I touched Cassandra, but I still have mental scars from the people who set it up wrong.
  - tleilaxu 4y ago
    Really good brief explanation, thank you! I can feel your exasperation from here.

vorpalhex 4y ago

What was the issue with the bulk deletes? Locking the cpu?

raffraffraff 4y ago
[sorry if you're reading in real-time, lots of edits] It's mentioned briefly in the original article. Cassandra achieves extremely good write performance by being append-only, which means that is disk IO patterns are sequential, with little random IO. This gives great performance on crappy spinning disks. However, with append-only, updating or deleting existing data is handled by writing new data or "tombstones". When you perform a search, Cassandra still scans all matching data in the table, including that which was deleted. It then uses tombstones to "ignore" deleted data. In other words, lazy delete is great for write performance (Cassandra is optimized for write), but creates overhead during read. So to avoid killing the cluster during a search, Cassandra imposes a tombstone limit. If a search contains more tombstones then the limit, the search will be abandoned. Real world example: if you're using Cassandra as a time series database and you don't add a sensible TTL to your metrics, a bulk delete would cause searches to suddenly fail, and they wouldn't work again until after compaction is performed on the table, rewriting the whole table minus data that has tombstones older than gc grace period. Metrics systems aren't just used for eye-baling dashboards during an incident, they're part of your alerting system (usually) so this could lead to missed alerts or false positives. This is why it's important to continually "age-out" data with a TTL, so that compactions are regularly removing tombstones older than the gc grace period. "Fucking gc grace period!" (me, on a few occasions) Why is there a gc grace period in here, making trouble for us? Because Cassandra is a distributed, replicated system that handles partitions and failures, it knows that some nodes may be temporarily unreachable. So it holds on to tombstones long enough to ensure that every node gets them. If it didn't do this, an offline mode may not get the tombstone before it's compacted out of existence in all other nodes. So when it comes back online you get zombie deleted data, back from the dead! Not nice! If you fuck this up and end up with a huge set of old data that you must delete, you are in for a treat. Easiest option is to manually schedule a bunch of operations to run during a quiet period. With metrics, write traffic volume is pretty linear but read traffic is user-driven. That could mean that quiet period are nightly, on weekends etc. The operations involve checking for downed nodes, modifying gc grace period, deleting a bunch of data and running a compaction. You have to carefully monitor logs for errors, and keep an eye on tombstone limits. Depending on your infrastructure it might be easier to build a new cluster side-by-side, write all the data from the old cluster into the new one with proper TTLs and delete the old cluster. If you're on bare metal and are out of space, you might have to get creative. (To keep a production metrics system running for a few years I had to get extremely creative) Sorry for the huge thread and possibly wrong info, it's a while since I touched Cassandra, but I still have mental scars from the people who set it up wrong.
- tleilaxu 4y ago
  Really good brief explanation, thank you! I can feel your exasperation from here.