3 ms·

Discord Indexes Trillions of Messages

38 points by todsacerdoti 2y ago

lysace 2y ago
LLM training input treasure? Signal-to-noise ratio is going to be lowish though.
- DrillShopper 2y ago
  If you want to train a groomer then Discord messages are the way to do it.
  - lithos 2y ago
    If we're acting like Redditors: "you can just block Minecraft Discords, and remove most of the grooming"
- YetAnotherNick 2y ago
  I think discord's data has insane value because it has real time reasoning steps much more than any big social media. Obviously you might want to filter out low information messages like "Hello" but model based filtering for signal is already solved more or less. Discord has lot of very technical channels like the one which solved BB(5) after decades of research.
  - leo-notte 2y ago
    Agreed. discord has a unique structure with threaded conversations, context carryover, and back and forth reasoning that you don't get from places like twitter or even reddit. it's especially useful for training models on collaborative problem solving or exploratory dialogue. filtering is a challenge but definitely solvable with current tools.
- throwaway42167 2y ago
  [flagged]
coldblues 2y ago
Indexes trillions of messages and refuses to delete any of them and comply with GDPR. Deleting your account DOES NOT delete your messages, they remain on the platform for an indefinite period of time. The only way to get them to delete your messages is to go through this process (https://discordomicon.github.io/removal/overview/ https://discordomicon.github.io/removal/overview/) and you're still likely to go through several support tickets before they comply with your request.
JackSlateur 2y ago
Tldr: they used a single massive elasticsearch cluster. Now, they are using multiple elasticsearch cluster, running in k8s. And redis has been replaced by pubsub (GCP). Since sounds like expensives design but ok, I guess
rkwasny 2y ago
Whoha, that is definitely expensive to run.
cosmosgenius 2y ago
Does anyone know a low memory disk based search solution for home usage? By low memory i mean in the range of 256MB maybe 1GB. I am almost convinced of mellisearch but still looking. Elasticsearch would have been perfect but the memory usage is too high for the server i have. Or should i go with a custom solution? FYI the search is highly dynamic ranging from full books, medical records, chat messages to movie/shopping/recipe catalogue.
- snikolaev 2y ago
  Manticore Search + Columnar Storage

38 points by todsacerdoti 2y ago

lysace 2y ago

LLM training input treasure? Signal-to-noise ratio is going to be lowish though.

DrillShopper 2y ago
If you want to train a groomer then Discord messages are the way to do it.
- lithos 2y ago
  If we're acting like Redditors: "you can just block Minecraft Discords, and remove most of the grooming"
YetAnotherNick 2y ago
I think discord's data has insane value because it has real time reasoning steps much more than any big social media. Obviously you might want to filter out low information messages like "Hello" but model based filtering for signal is already solved more or less. Discord has lot of very technical channels like the one which solved BB(5) after decades of research.
- leo-notte 2y ago
  Agreed. discord has a unique structure with threaded conversations, context carryover, and back and forth reasoning that you don't get from places like twitter or even reddit. it's especially useful for training models on collaborative problem solving or exploratory dialogue. filtering is a challenge but definitely solvable with current tools.
throwaway42167 2y ago
[flagged]

coldblues 2y ago

Indexes trillions of messages and refuses to delete any of them and comply with GDPR. Deleting your account DOES NOT delete your messages, they remain on the platform for an indefinite period of time. The only way to get them to delete your messages is to go through this process (https://discordomicon.github.io/removal/overview/ https://discordomicon.github.io/removal/overview/) and you're still likely to go through several support tickets before they comply with your request.

JackSlateur 2y ago

Tldr: they used a single massive elasticsearch cluster. Now, they are using multiple elasticsearch cluster, running in k8s. And redis has been replaced by pubsub (GCP). Since sounds like expensives design but ok, I guess

rkwasny 2y ago

Whoha, that is definitely expensive to run.

cosmosgenius 2y ago

Does anyone know a low memory disk based search solution for home usage? By low memory i mean in the range of 256MB maybe 1GB. I am almost convinced of mellisearch but still looking. Elasticsearch would have been perfect but the memory usage is too high for the server i have. Or should i go with a custom solution? FYI the search is highly dynamic ranging from full books, medical records, chat messages to movie/shopping/recipe catalogue.

snikolaev 2y ago
Manticore Search + Columnar Storage