3 ms·
I'm the engineer who worked on the Search backend. Read our blog here: https://blog.discordapp.com/how-discord-indexes-billions-of-messages-e3d5e9be866f https:/
by jhgg 9y ago
I'm the engineer who worked on the Search backend. Read our blog here: https://blog.discordapp.com/how-discord-indexes-billions-of-messages-e3d5e9be866f https://blog.discordapp.com/how-discord-indexes-billions-of-...
Cost isn't actually too bad, given how we've designed our search schema - and how we lazily index.
- shados 9y agoMakes sense. I did read that blog before, but I just can't keep thinking about just the cost of storing that much text on disc, even if it was compressed, would quickly add up. Maybe my mental math is failing me (or I'm overestimating how many messages there are), but that still seemed expensive. In the end, as long as you don't go out of business, because I'd be very very sad :)
- jhgg 9y agoStorage is actually really cheap for this, we don't store the message data in ES, it just builds the inverted indices. I explain this more in-depth on the blog post. On GCP, 50TB of Persistent SSD is roughly $9000/month before any discounts.