8 ms·

How discord stores billions of messages (2017)

204 points by greymalik 4y ago

judge2020 4y ago
(2017) As revealed in this blog post[0], they now see 4 billion messages per day. 0: https://news.ycombinator.com/item?id=32474093 https://news.ycombinator.com/item?id=32474093
- tmikaeld 4y ago
  And switched to ScyllaDB.
- samlambert 4y ago
  Around ~11k a second. Not small but not insane scale either.
  - judge2020 4y ago
    4,000,000,000/(24 • 60 • 60) ~= 46,000 And their mps definitely scales based on time-of-day, illustrated by this blog image[0] from this blog[1] (given their CPUs are only really handling messages). 0: https://assets-global.website-files.com/5f9072399b2640f14d6a2bf4/611ed6f4ff765c21c3ea810b_1*eT1PAnwaAcfCUikJ281v2g.png https://assets-global.website-files.com/5f9072399b2640f14d6a... 1: https://discord.com/blog/why-discord-is-switching-from-go-to-rust https://discord.com/blog/why-discord-is-switching-from-go-to...
  - chrsig 4y ago
    as someone that's worked with a similar workload (edit: in terms of write volume, not persistent chat messages), it's an incredibly annoying volume. Too high for simple solutions, not big enough for large scale solutions.
seydor 4y ago
how many are bots? AFAIK bot traffic is higher than human there. Plus they added so much bot APIs that now bots are hacking and spamming users' accounts
- tcfhgj 4y ago
  Bot traffic and human traffic aren't mutually exclusive, bots can be humans: https://t2bot.io/discord/ https://t2bot.io/discord/
  - PeterCorless 4y ago
    Elsewhere in social media and gaming many accounts may have both manual and automated activities associated with it. For example, in a WoW chat, that might be your friend typing "lol," or it could be an add-on putting out an automated raid warning. Similarly, your Twitter account could be a manual post, or a scheduled one from Tweetdeck. Are you now a "bot account?" Discord does permit bots in its system. But it tries to limit "self-bots" on the platform. It wants humans to be humans and bots to be bots. "Self-bots" in Discord could result in termination. Read more here: https://support.discord.com/hc/en-us/articles/115002192352-Automated-user-accounts-self-bots-#:~:text=Automating%20normal%20user%20accounts%20(generally,an%20account%20termination%20if%20found https://support.discord.com/hc/en-us/articles/115002192352-A....
- deleted 4y ago
  [deleted]
Jamie9912 4y ago
I believe they use ScyllaDB exclusively now for storing messages
mgoldstein5 4y ago
oreally 4y ago
I don't know databases and there's quite a number of them on the market, so posts like these are great. Sidetrack though, does anyone have a list for pros and cons of each db, with a preference towards low latency? Also how does it compare with say Postgres?
- Transfinity 4y ago
  That depends heavily on the shape of your data, what your workload looks like, what sort of consistency guarantees you need, etc. I recommend Designing Data Intensive Applications for getting a handle on this - it's the book I suggest to SDE IIs who are hungry for the jump to senior. Not a quick read, but well written, and there's not really a shortcut to deep understanding other than deep study.
  - oreally 4y ago
    thanks, will look that up
  - dominotw 4y ago
    > I suggest to SDE IIs who are hungry for the jump to senior Any suggestions for seniors who are hungry for a staff title. Is there any value to specializing in tech anymore after senior or is it all 'soft skills' at this point ?
    - asdfasgasdgasdg 4y ago
      It is not all any one thing. If you’re completely lacking in soft skills, you’ll probably have more trouble the higher up you go, and you may get stuck at some point. Hard technical skills become less important the higher you go, but even the CEO of a tech company probably still needs to have some basic understanding of technology. Being good at both things is obviously the best way forward.
    - dharmaturtle 4y ago
      I enjoyed this: https://staffeng.com/book https://staffeng.com/book Not a staff though so can't attest to how true/accurate/effective it is.
coldblues 4y ago
Unfortunately, they're not doing a good job at deleting them. If you press the "Delete Account" button, all it does is anonymize your profile, and leaves all of your messages intact. One of the reasons I avoid using Discord whenever possible.
- hombre_fatal 4y ago
  Why is HN acceptable then?
  - byyll 4y ago
    HN is a public forum, not behind a pay or auth wall.
    - vxNsr 4y ago
      Just treat it the same way, assume everything you will say on discord will become public. Problem solved.
      - unlog 4y ago
        Sorry, buy you have the wrong idea. That would be fine for public servers, but not for Private Conversations. I'ts unacceptable. edit:typo
        vel0city 4y ago
        Private conversations aren't really very private on Discord. Attachment URLs are globally visible.
        deleted 4y ago
        [deleted]
        hombre_fatal 4y ago
        I can't think of a good reason why the presence or effectiveness of a delete button should change what you write online, even in a so-called "private conversation", or whether you use a certain platform. Delete is useful, but it shouldn't change what you're willing to send online because delete doesn't time travel and unsend the message.
jpalomaki 4y ago
This is quite valuable advise: "The original version of Discord was built in just under two months in early 2015. Arguably, one of the best databases for iterating quickly is MongoDB. Everything on Discord was stored in a single MongoDB replica set and this was intentional, but we also planned everything for easy migration to a new database" Also the article links to Twitter blog, which gives similar point (it's from 2010): "We [Twitter] currently use MySQL to store most of our online data. In the beginning, the data was in one small database instance which in turn became one large database instance and eventually many large database clusters" [1] [1] https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake https://blog.twitter.com/engineering/en_us/a/2010/announcing...
- avodonosov 4y ago
  The first version of reddit also used a single PostgreSQL instance apparently: https://github.com/reddit-archive/reddit1.0/blob/bb4fbdb5871a32709df30540685cf44c637bf203/data.lisp https://github.com/reddit-archive/reddit1.0/blob/bb4fbdb5871...
  - MonkeyMalarky 4y ago
    Seems like sound advice? Starting with DB clusters would be premature optimization, but designing your schema with the expectation that it will be on multiple instances later is sane. Kind of like building a monolith with the expectation that certain components will eventually be moved out into their own services.
  - andrewxdiamond 4y ago
    Reddit is perhaps not the best company to look at for operational excellence. They seem to be barely keeping the ship aloft
    - ChuckNorris89 4y ago
      Modern Reddit for me runs like ass and I have a 2021 4.2GHz 8 core Ryzen CPU. I can't imagine what it's like on older systems where even my work Intel 9th gen was braking the site.
hestefisk 4y ago
Wonder if Postgres would scale to such volume.
- aeyes 4y ago
  Maybe its possible with Citus. But I wouldn't try, it's not the right tool for the job.
unlog 4y ago
Discord doesn't respect privacy, you cannot just get rid of a whole conversation. Users are the product, and they make it so difficult to delete entire convos that it's so obvious it's just valuable to them.
- roflyear 4y ago
  You can't on HN either.
  - unlog 4y ago
    on HN the convos aren't private.
    - magicalhippo 4y ago
      Discord is a closed-source service hosted by a 3rd party on the internet. It's by definition not private either.
      - richbell 4y ago
        Hacker News doesn't have "private messages", Discord does.
        magicalhippo 4y ago
        My point is neither has private messages. Discord has direct user-to-user messages. Those things are not the same.
- corndoge 4y ago
  Can say the same thing about IRC and logging…anything online is permanent
  - unlog 4y ago
    For public servers I understand, for private messages obviously not. edit: better wording
paxys 4y ago
As a point of comparison, Slack uses MySQL (Vitess) – https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v...
chrsig 4y ago
> Nothing was surprising, we got exactly what we expected. Such a satisfying feeling in the engineering world. > We noticed Cassandra was running 10 second “stop-the-world” GC constantly but we had no idea why. This makes me very thankful for the work that the Go team has put into the go GC. > In the scenario that a user edits a message at the same time as another user deletes the same message, we ended up with a row that was missing all the data except the primary key and the text since all Cassandra writes are upserts. Does cassandra not offer a mechanism to do a conditional update? I'd expect to be able to submit a upsert that fails if the row isn't present, or has a `deleted = true` field, or something to that effect.
- ryanworl 4y ago
  "Conditional update" requires consensus to be fault tolerant, and regular Cassandra operations don't use consensus. Cassandra has a feature called light-weight transactions which can sort of get you there, and they use Paxos to do it. It unfortunately has a history of bugs that invalidate the guarantees you'd want from this feature, but more modern versions of Cassandra are improving in this regard.
  - PeterCorless 4y ago
    Yes. ScyllaDB also has LWTs based on Paxos. More here: https://www.scylladb.com/2020/07/15/getting-the-most-out-of-lightweight-transactions-in-scylla/ https://www.scylladb.com/2020/07/15/getting-the-most-out-of-... [Disclosure: I work for ScyllaDB]
- cmrdporcupine 4y ago
  > This makes me very thankful for the work that the Go team has put into the go GC. If you read that part of the piece again, the problem with the GC wasn't necessarily with Java's GC implementation but with Cassandra's storage architecture combined with a mass delete, which tombstoned millions of records and then Cassandra had to scan them and pass them over: "Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)" Even if you had an incremental GC, you'd be eating CPU and thrashing the cache like crazy doing that. And you may never catch up. Actually even if you didn't have a GC language, you'd likely still have scan or free operations. Writing databases is hard. And actually the various JVM's GCs are top knotch and have had hundreds of thousands of engineering hours put into perfecting them for various workloads. Still, these days I'm glad not to be near it and I work in Rust or C++.
raffraffraff 4y ago
I haven't used Cassandra for about 3 years, and this is awakening memories. At a previous company I inherited a very badly assembled cluster that was used for a time series database. The guys who built it said (and no, I'm not kidding...) "we don't need to put a TTL on metrics because they're tiny and anyway we can just add more nodes and scale the cluster horizontally forever!". Well, forever was about 2 years, when the physical data center ran out of rack space, and teams abused the metrics system with TBs of bullshit data. That was when they handed the whole metrics system to yours truly. And I discover two things: 1. You can't just bulk delete a year of old stale data without breaking it 2. "Woops, did we really set replication factor to 1?" Fun.
- vorpalhex 4y ago
  What was the issue with the bulk deletes? Locking the cpu?
  - raffraffraff 4y ago
    [sorry if you're reading in real-time, lots of edits] It's mentioned briefly in the original article. Cassandra achieves extremely good write performance by being append-only, which means that is disk IO patterns are sequential, with little random IO. This gives great performance on crappy spinning disks. However, with append-only, updating or deleting existing data is handled by writing new data or "tombstones". When you perform a search, Cassandra still scans all matching data in the table, including that which was deleted. It then uses tombstones to "ignore" deleted data. In other words, lazy delete is great for write performance (Cassandra is optimized for write), but creates overhead during read. So to avoid killing the cluster during a search, Cassandra imposes a tombstone limit. If a search contains more tombstones then the limit, the search will be abandoned. Real world example: if you're using Cassandra as a time series database and you don't add a sensible TTL to your metrics, a bulk delete would cause searches to suddenly fail, and they wouldn't work again until after compaction is performed on the table, rewriting the whole table minus data that has tombstones older than gc grace period. Metrics systems aren't just used for eye-baling dashboards during an incident, they're part of your alerting system (usually) so this could lead to missed alerts or false positives. This is why it's important to continually "age-out" data with a TTL, so that compactions are regularly removing tombstones older than the gc grace period. "Fucking gc grace period!" (me, on a few occasions) Why is there a gc grace period in here, making trouble for us? Because Cassandra is a distributed, replicated system that handles partitions and failures, it knows that some nodes may be temporarily unreachable. So it holds on to tombstones long enough to ensure that every node gets them. If it didn't do this, an offline mode may not get the tombstone before it's compacted out of existence in all other nodes. So when it comes back online you get zombie deleted data, back from the dead! Not nice! If you fuck this up and end up with a huge set of old data that you must delete, you are in for a treat. Easiest option is to manually schedule a bunch of operations to run during a quiet period. With metrics, write traffic volume is pretty linear but read traffic is user-driven. That could mean that quiet period are nightly, on weekends etc. The operations involve checking for downed nodes, modifying gc grace period, deleting a bunch of data and running a compaction. You have to carefully monitor logs for errors, and keep an eye on tombstone limits. Depending on your infrastructure it might be easier to build a new cluster side-by-side, write all the data from the old cluster into the new one with proper TTLs and delete the old cluster. If you're on bare metal and are out of space, you might have to get creative. (To keep a production metrics system running for a few years I had to get extremely creative) Sorry for the huge thread and possibly wrong info, it's a while since I touched Cassandra, but I still have mental scars from the people who set it up wrong.
mannyv 4y ago
'build quickly to prove out a product feature, but always with a path to a more robust solution" Yes
- mcheung610 4y ago

204 points by greymalik 4y ago

judge2020 4y ago

(2017) As revealed in this blog post[0], they now see 4 billion messages per day. 0: https://news.ycombinator.com/item?id=32474093 https://news.ycombinator.com/item?id=32474093

tmikaeld 4y ago
And switched to ScyllaDB.
samlambert 4y ago
Around ~11k a second. Not small but not insane scale either.
- judge2020 4y ago
  4,000,000,000/(24 • 60 • 60) ~= 46,000 And their mps definitely scales based on time-of-day, illustrated by this blog image[0] from this blog[1] (given their CPUs are only really handling messages). 0: https://assets-global.website-files.com/5f9072399b2640f14d6a2bf4/611ed6f4ff765c21c3ea810b_1*eT1PAnwaAcfCUikJ281v2g.png https://assets-global.website-files.com/5f9072399b2640f14d6a... 1: https://discord.com/blog/why-discord-is-switching-from-go-to-rust https://discord.com/blog/why-discord-is-switching-from-go-to...
- chrsig 4y ago
  as someone that's worked with a similar workload (edit: in terms of write volume, not persistent chat messages), it's an incredibly annoying volume. Too high for simple solutions, not big enough for large scale solutions.

seydor 4y ago

how many are bots? AFAIK bot traffic is higher than human there. Plus they added so much bot APIs that now bots are hacking and spamming users' accounts

tcfhgj 4y ago
Bot traffic and human traffic aren't mutually exclusive, bots can be humans: https://t2bot.io/discord/ https://t2bot.io/discord/
- PeterCorless 4y ago
  Elsewhere in social media and gaming many accounts may have both manual and automated activities associated with it. For example, in a WoW chat, that might be your friend typing "lol," or it could be an add-on putting out an automated raid warning. Similarly, your Twitter account could be a manual post, or a scheduled one from Tweetdeck. Are you now a "bot account?" Discord does permit bots in its system. But it tries to limit "self-bots" on the platform. It wants humans to be humans and bots to be bots. "Self-bots" in Discord could result in termination. Read more here: https://support.discord.com/hc/en-us/articles/115002192352-Automated-user-accounts-self-bots-#:~:text=Automating%20normal%20user%20accounts%20(generally,an%20account%20termination%20if%20found https://support.discord.com/hc/en-us/articles/115002192352-A....
deleted 4y ago
[deleted]

Jamie9912 4y ago

I believe they use ScyllaDB exclusively now for storing messages

mgoldstein5 4y ago

oreally 4y ago

I don't know databases and there's quite a number of them on the market, so posts like these are great. Sidetrack though, does anyone have a list for pros and cons of each db, with a preference towards low latency? Also how does it compare with say Postgres?

Transfinity 4y ago
That depends heavily on the shape of your data, what your workload looks like, what sort of consistency guarantees you need, etc. I recommend Designing Data Intensive Applications for getting a handle on this - it's the book I suggest to SDE IIs who are hungry for the jump to senior. Not a quick read, but well written, and there's not really a shortcut to deep understanding other than deep study.
- oreally 4y ago
  thanks, will look that up
- dominotw 4y ago
  > I suggest to SDE IIs who are hungry for the jump to senior Any suggestions for seniors who are hungry for a staff title. Is there any value to specializing in tech anymore after senior or is it all 'soft skills' at this point ?
  - asdfasgasdgasdg 4y ago
    It is not all any one thing. If you’re completely lacking in soft skills, you’ll probably have more trouble the higher up you go, and you may get stuck at some point. Hard technical skills become less important the higher you go, but even the CEO of a tech company probably still needs to have some basic understanding of technology. Being good at both things is obviously the best way forward.
  - dharmaturtle 4y ago
    I enjoyed this: https://staffeng.com/book https://staffeng.com/book Not a staff though so can't attest to how true/accurate/effective it is.

coldblues 4y ago

Unfortunately, they're not doing a good job at deleting them. If you press the "Delete Account" button, all it does is anonymize your profile, and leaves all of your messages intact. One of the reasons I avoid using Discord whenever possible.

hombre_fatal 4y ago
Why is HN acceptable then?
- byyll 4y ago
  HN is a public forum, not behind a pay or auth wall.
  - vxNsr 4y ago
    Just treat it the same way, assume everything you will say on discord will become public. Problem solved.
    - unlog 4y ago
      Sorry, buy you have the wrong idea. That would be fine for public servers, but not for Private Conversations. I'ts unacceptable. edit:typo
      - vel0city 4y ago
        Private conversations aren't really very private on Discord. Attachment URLs are globally visible.
      - deleted 4y ago
        [deleted]
      - hombre_fatal 4y ago
        I can't think of a good reason why the presence or effectiveness of a delete button should change what you write online, even in a so-called "private conversation", or whether you use a certain platform. Delete is useful, but it shouldn't change what you're willing to send online because delete doesn't time travel and unsend the message.

jpalomaki 4y ago

This is quite valuable advise: "The original version of Discord was built in just under two months in early 2015. Arguably, one of the best databases for iterating quickly is MongoDB. Everything on Discord was stored in a single MongoDB replica set and this was intentional, but we also planned everything for easy migration to a new database" Also the article links to Twitter blog, which gives similar point (it's from 2010): "We [Twitter] currently use MySQL to store most of our online data. In the beginning, the data was in one small database instance which in turn became one large database instance and eventually many large database clusters" [1] [1] https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake https://blog.twitter.com/engineering/en_us/a/2010/announcing...

avodonosov 4y ago
The first version of reddit also used a single PostgreSQL instance apparently: https://github.com/reddit-archive/reddit1.0/blob/bb4fbdb5871a32709df30540685cf44c637bf203/data.lisp https://github.com/reddit-archive/reddit1.0/blob/bb4fbdb5871...
- MonkeyMalarky 4y ago
  Seems like sound advice? Starting with DB clusters would be premature optimization, but designing your schema with the expectation that it will be on multiple instances later is sane. Kind of like building a monolith with the expectation that certain components will eventually be moved out into their own services.
- andrewxdiamond 4y ago
  Reddit is perhaps not the best company to look at for operational excellence. They seem to be barely keeping the ship aloft
  - ChuckNorris89 4y ago
    Modern Reddit for me runs like ass and I have a 2021 4.2GHz 8 core Ryzen CPU. I can't imagine what it's like on older systems where even my work Intel 9th gen was braking the site.

hestefisk 4y ago

Wonder if Postgres would scale to such volume.

aeyes 4y ago
Maybe its possible with Citus. But I wouldn't try, it's not the right tool for the job.

unlog 4y ago

Discord doesn't respect privacy, you cannot just get rid of a whole conversation. Users are the product, and they make it so difficult to delete entire convos that it's so obvious it's just valuable to them.

roflyear 4y ago
You can't on HN either.
- unlog 4y ago
  on HN the convos aren't private.
  - magicalhippo 4y ago
    Discord is a closed-source service hosted by a 3rd party on the internet. It's by definition not private either.
    - richbell 4y ago
      Hacker News doesn't have "private messages", Discord does.
      - magicalhippo 4y ago
        My point is neither has private messages. Discord has direct user-to-user messages. Those things are not the same.
corndoge 4y ago
Can say the same thing about IRC and logging…anything online is permanent
- unlog 4y ago
  For public servers I understand, for private messages obviously not. edit: better wording

paxys 4y ago

As a point of comparison, Slack uses MySQL (Vitess) – https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v...

chrsig 4y ago

> Nothing was surprising, we got exactly what we expected. Such a satisfying feeling in the engineering world. > We noticed Cassandra was running 10 second “stop-the-world” GC constantly but we had no idea why. This makes me very thankful for the work that the Go team has put into the go GC. > In the scenario that a user edits a message at the same time as another user deletes the same message, we ended up with a row that was missing all the data except the primary key and the text since all Cassandra writes are upserts. Does cassandra not offer a mechanism to do a conditional update? I'd expect to be able to submit a upsert that fails if the row isn't present, or has a `deleted = true` field, or something to that effect.

ryanworl 4y ago
"Conditional update" requires consensus to be fault tolerant, and regular Cassandra operations don't use consensus. Cassandra has a feature called light-weight transactions which can sort of get you there, and they use Paxos to do it. It unfortunately has a history of bugs that invalidate the guarantees you'd want from this feature, but more modern versions of Cassandra are improving in this regard.
- PeterCorless 4y ago
  Yes. ScyllaDB also has LWTs based on Paxos. More here: https://www.scylladb.com/2020/07/15/getting-the-most-out-of-lightweight-transactions-in-scylla/ https://www.scylladb.com/2020/07/15/getting-the-most-out-of-... [Disclosure: I work for ScyllaDB]
cmrdporcupine 4y ago
> This makes me very thankful for the work that the Go team has put into the go GC. If you read that part of the piece again, the problem with the GC wasn't necessarily with Java's GC implementation but with Cassandra's storage architecture combined with a mass delete, which tombstoned millions of records and then Cassandra had to scan them and pass them over: "Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)" Even if you had an incremental GC, you'd be eating CPU and thrashing the cache like crazy doing that. And you may never catch up. Actually even if you didn't have a GC language, you'd likely still have scan or free operations. Writing databases is hard. And actually the various JVM's GCs are top knotch and have had hundreds of thousands of engineering hours put into perfecting them for various workloads. Still, these days I'm glad not to be near it and I work in Rust or C++.

raffraffraff 4y ago

I haven't used Cassandra for about 3 years, and this is awakening memories. At a previous company I inherited a very badly assembled cluster that was used for a time series database. The guys who built it said (and no, I'm not kidding...) "we don't need to put a TTL on metrics because they're tiny and anyway we can just add more nodes and scale the cluster horizontally forever!". Well, forever was about 2 years, when the physical data center ran out of rack space, and teams abused the metrics system with TBs of bullshit data. That was when they handed the whole metrics system to yours truly. And I discover two things: 1. You can't just bulk delete a year of old stale data without breaking it 2. "Woops, did we really set replication factor to 1?" Fun.

vorpalhex 4y ago
What was the issue with the bulk deletes? Locking the cpu?
- raffraffraff 4y ago
  [sorry if you're reading in real-time, lots of edits] It's mentioned briefly in the original article. Cassandra achieves extremely good write performance by being append-only, which means that is disk IO patterns are sequential, with little random IO. This gives great performance on crappy spinning disks. However, with append-only, updating or deleting existing data is handled by writing new data or "tombstones". When you perform a search, Cassandra still scans all matching data in the table, including that which was deleted. It then uses tombstones to "ignore" deleted data. In other words, lazy delete is great for write performance (Cassandra is optimized for write), but creates overhead during read. So to avoid killing the cluster during a search, Cassandra imposes a tombstone limit. If a search contains more tombstones then the limit, the search will be abandoned. Real world example: if you're using Cassandra as a time series database and you don't add a sensible TTL to your metrics, a bulk delete would cause searches to suddenly fail, and they wouldn't work again until after compaction is performed on the table, rewriting the whole table minus data that has tombstones older than gc grace period. Metrics systems aren't just used for eye-baling dashboards during an incident, they're part of your alerting system (usually) so this could lead to missed alerts or false positives. This is why it's important to continually "age-out" data with a TTL, so that compactions are regularly removing tombstones older than the gc grace period. "Fucking gc grace period!" (me, on a few occasions) Why is there a gc grace period in here, making trouble for us? Because Cassandra is a distributed, replicated system that handles partitions and failures, it knows that some nodes may be temporarily unreachable. So it holds on to tombstones long enough to ensure that every node gets them. If it didn't do this, an offline mode may not get the tombstone before it's compacted out of existence in all other nodes. So when it comes back online you get zombie deleted data, back from the dead! Not nice! If you fuck this up and end up with a huge set of old data that you must delete, you are in for a treat. Easiest option is to manually schedule a bunch of operations to run during a quiet period. With metrics, write traffic volume is pretty linear but read traffic is user-driven. That could mean that quiet period are nightly, on weekends etc. The operations involve checking for downed nodes, modifying gc grace period, deleting a bunch of data and running a compaction. You have to carefully monitor logs for errors, and keep an eye on tombstone limits. Depending on your infrastructure it might be easier to build a new cluster side-by-side, write all the data from the old cluster into the new one with proper TTLs and delete the old cluster. If you're on bare metal and are out of space, you might have to get creative. (To keep a production metrics system running for a few years I had to get extremely creative) Sorry for the huge thread and possibly wrong info, it's a while since I touched Cassandra, but I still have mental scars from the people who set it up wrong.

mannyv 4y ago

'build quickly to prove out a product feature, but always with a path to a more robust solution" Yes

mcheung610 4y ago