12 ms·
How Discord stores trillions of messages (2023)
- pavel_lishin 2y agoAnyone else reading this and being quite happy that they're not working at this scale?
- mystified5016 2y agoAny time I read anything about any web-adjacent technology I'm incredibly thankful that I don't work anywhere near that industry. Embedded can be complex, but web stuff is just a Lovecraftian nightmare in comparison
- milesvp 2y agoI have stared into the abyss and seen the eyes of cthulu. I am much happier writing embedded drivers than I was trying to make sense of why previous devs thought it was a good idea to move bounded tunable server side api calls to the client, allowing it to effectively write arbitrary sql calls across multiple databases.
- qudat 2y agoIteration speed is significantly fast on the client. Perf is an afterthought — for better or worse
- swyx 2y agospoken like someone who doesnt deploy clients at discord scale? the 200 backend nodes surely update significantly faster than the hundreds of millions of clients.
- artursapek 2y agoSounds like a fun time lol
- bdcravens 2y agoFortunately the web is starting (very slowly) to return to sanity, pushing back towards the simpler server-rendered pattern with Javascript being relegated to specific use cases.
- Aeolun 2y agoI really like the client rendered UI part. It’s a lot more efficient than sending the whole page again every time.
- gonzo41 2y agoBut you can cache the whole server side page and the cost is once. Whereas if you have the client side do the render then every client wears the cost.
- iknowstuff 2y agoYou usually can't because of users who are signed in needing slightly different pages etc.
- bdcravens 2y agoWhile not as fast as a purely client cached page, the server can selectively cache content, even when some bits of the page are dynamic.
- asynchronous 2y agoWe can also cache some of the dynamic JavaScript, depending on the scenario but your point stands.
- bdcravens 2y agoWhich becomes a far more important issue when dealing with bandwidth or CPU constrained devices, or artificially imposed constraints due to data usage costs.
- Aeolun 2y ago
- Twirrim 2y agoBut that's where the really fun and complicated problems are. The ones that really make you stop and think, and not just think, but be creative. 95% of the work is still the same "treading in well trod paths", same old same old tech work, but that 5% is really something.
- pavel_lishin 2y agoI think I can understand the appeal, but it's just not there for me. I have enough complicated problems outside of work, some of which are even fun to solve.
- Olreich 2y agoThis was a “double-pump” migration to a faster database and building a caching service. There’s nothing particularly fancy or creative about their solutions. The migration efforts and working out issues with the reverse table scan were probably way more creative, but they didn’t get into that unfortunately.
- wavemode 2y agoI don't mind scale. I mind the bureaucracy and promotion-driven-development that comes with working in a bloated engineering org.
- pm90 2y ago+100 Many companies have products that operate at “scale”. They manage to do so with pretty boring techniques (sharding, autoscaling) and technologies (postgres, cloud storage). Because of the insane blog driven tech culture, many of these teams get questioned by clueless leadership (who read these blogs) and ask why the company isn’t using cassandra / some other hot technology. And it always causes much consternation and wastage.
- rnts08 2y agoAnyone wanting to introduce $new/$other language, database, library, deployment system, build system into a large enough system that doesn't solve any actual problem is a nightmare for someone working at this scale. I don't mind the scale, I like it. I don't like having to fend off questions and complaints why we aren't deploying the latest shiny new thing in our core this week.
- secondcoming 2y agoWell we use Cassandra (actually ScyllaDB) because Redis no longer cut it.
- est 2y agoI am happy that I dont have to deal with this. I am sad that my business aren't as big as this scale.
- deleted 2y ago[deleted]
- Aeolun 2y agoHonestly, 77 nodes doesn’t sound like a terrific scale? The more I scale things up, the more I realize that the tone of the problems doesn’t really change. You just get more layers to your data structures.
- twelve40 2y agoI'm happy I'm currently not working at this scale. I'm not happy when idiots (including one of our self-important ex-Google VP's) set this as a benchmark for backend interviews (for careers that 99% likely will never come close to such problems).
- leetrout 2y agoNeeds (2023) That services layer reminds be of a big, fancy, distributed Varnish Cache... they don't mention caching and they chose the word coalesce so I assume it doesn't do much actual caching. But made me think of Varnish's "grace mode" and it's use to prevent the thundering herd problem (which is where I first heard of 'request coalescing') https://varnish-cache.org/docs/6.1/users-guide/vcl-grace.html#grace-mode https://varnish-cache.org/docs/6.1/users-guide/vcl-grace.htm... Also love to see consistent hashing come up again and again. It's a great piece of duct tape that has proven useful in many similar situations. If you know where something should be then you know where everything is gonna come look for it!
- loloquwowndueo 2y agoCoalescing and “origin shielding” tend to be more common terms for that - I’ve never heard of “grace” until today :)
- atombender 2y agoVarnish does call it coalescing. Grace is used for a specific situation: When a previously cached object has expired, Varnish won't evict it from the cache immediately, but will continue to serve the old content, while sending exactly 1 request to the background to refetch. How long an object can live after expiring is called the grace. The HTTP standard calls this behaviour "stale-while-revalidate".
- dang 2y agoYear added above. Thanks!
- hinkley 2y agoNginx always more businesslike. proxy_cache_use_stale updating;
- mnutt 2y agoGrace mode itself doesn’t prevent thundering herd; varnish coalesces all requests automatically and grace mode is used to increase the likelihood of clients receiving cached (albeit stale) responses.
- tonetegeatinst 2y agoMy love of embedded stuff is growing. I'm self teaching C and assembly....to get better at low level programming and interactions with hardware but it all seems much simpler than the big data systems. Granted I'm sure it call be broken down into steps and issues to solve like any programming issue but I'm happy focusing on low level stuff for now.
- codexon 2y ago> The ScyllaDB team prioritized improvements and implemented performant reverse queries, removing the last database blocker in our migration plan. I wonder how much they paid ScyllaDB to do this before even using ScyllaDB.
- jsnell 2y agoThe article says they were using ScyllaDB for everything except the message store two years before they did the migration for messages.
- awesomeMilou 2y agoMy takeaway from this is maybe somewhat different from what the authors intended: > The last one? Our friend, cassandra-messages. [...] To start with, it’s a big cluster. With trillions of messages and nearly 200 nodes, any migration was going to be an involved effort. To me, that's a surprisingly small amount of nodes for message storage, given the size of discord. I had honestly expected a much more intricate architecture, engineered towards quick scalability, involving a lot more moving parts. I'm sure the complexity is higher than stated in the article, but it makes me wonder, given that I've been partially responsible for more than 200 physical nodes that did less, how much of modern cloud architecture is over engineered.
- hiyer 2y agoAlso bear in mind that they're now doing the same with just 72 nodes.
- nicholasjarnold 2y ago> ...how much of modern cloud architecture is over engineered. I would wager a good majority of it is. The Stack Overflow architecture[0] sticks out to me in this regard as an example on the other end of the spectrum. [0] https://news.ycombinator.com/item?id=34950843 https://news.ycombinator.com/item?id=34950843
- romanhn 2y agoThey are talking about 177 database nodes, which is not an indicator of architecture complexity. I assume they have dozens/hundreds of services consisting of multiple highly available nodes each across various geographies. Having seen a much smaller set of Cassandra nodes used to store billions (rather than trillions) of records, I can say that Cassandra was definitely a total PITA for on-call, and a cause of several major outages.
- tcfhgj 2y agoStoring is one thing. Performing data mining on them is another
- CamperBob2 2y agoAlso, people need to keep in mind that those trillions of messages are archived nowhere. Thanks to the walled gardens we're obsessed with building, far-future anthropologists will know more about Pompeii and Machu Picchu than San Francisco.
- xboxnolifes 2y agoI don't think every single individual message ever needs to be archived. Every text, every email, every post-it, every poke, every emoji, every reaction GIF...
- famahar 2y agoDefinitely not everything, but it's still wild to me that so many products and services have all their troubleshooting and customer support in a discord server.
- foobazgt 2y agoSadly, I can understand why Discord doesn't have a lot of incentive to do this. Maybe the community should popularize an open-source free/low-costing bot and hosting solution for exported chat? (I couldn't find one in a few minutes of searching).
- tbrockman 2y agoHere ya go: https://github.com/AnswerOverflow/AnswerOverflow https://github.com/AnswerOverflow/AnswerOverflow
- ekianjo 2y agoEven FOSS communities. shame on the devs who decide to do so.
- zombiwoof 2y agoI think it’s annoying they interview engineers like they are Google and reading the blog they made it up and learned some basic “pitfalls” as they went along
- foobazgt 2y agoThis blog post seems to blame GC heavily, but if you look back at their earlier blog post [0], it seems to be more shortcomings in either how they're using Cassandra or how Cassandra handles heavy deletes, or some combination: "It was at that moment that it became obvious they deleted millions of messages using our API, leaving only 1 message in the channel. If you have been paying attention you might remember how Cassandra handles deletes using tombstones (mentioned in Eventual Consistency). When a user loaded this channel, even though there was only 1 message, Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)." And although the blog post talks about GC tuning, there's mention here [1] that they didn't do much tuning and were actually running on an old version of Cassandra (and presumably JVM) - having just switched over from CMS (!). 0) https://discord.com/blog/how-discord-stores-billions-of-messages 1) https://news.ycombinator.com/item?id=33136453
- Aeolun 2y agoBut then it’s still nice that they’re using ScyllaDB and now it’s not a concern at all right? Even if they were using their original solution wrong, I think the solution that cannot use wrong is superior.
- ericvolp12 2y agoThe funny part is ScyllaDB still uses tombstones for deletions, though they do have configurable compaction strategies and iirc Discord uses Scylla's Incremental Compaction Strategy that I suppose solves the specific issue they were dealing with. iirc that compaction strategy will trigger a compaction once a certain threshold of a partition is tombstones and then the table is rebuilt without the tombstoned content (which effectively pauses writes on that specific node and that specific table and partition for the duration of that process). Compacting a massive partition is really expensive. Scylla defaults to warning you that a partition is too large if it has at least 100,000 rows in it. My guess is when they moved to ScyllaDB they also adopted a new strategy for partitioning messages in a channel that keeps partition sizes reasonable so compactions don't take a super long time.
- jaimehrubiks 2y agoUntil they don't, or they can't, and they need to start deleting. (Not trying to undermine the engineering efforts, or the welcoming engineering blog posts though! I really think all these is needed)
- cynicalpeace 2y agoIs there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.
- cowthulhu 2y agoI’m not sure if Postgres would have enough horizontal scaling to accommodate the insane volume of reads and writes. I would be super interested to be proven wrong though… anyone know of a cluster being run at that scale?
- riku_iki 2y ago> Scale certainly wouldn't be it. vanilla postgres can't scale to such size, you need some sharding solution on top, which likely will be much harder to maintain than ScyllaDB..
- justnoise 2y agoI'd guess that Discord's storage systems lean towards processing a lot more writes than reads. Postgres and other databases that use B-tree indexing are ideally suited for read heavy workloads. LSM based databases like Cassandra/Scylla are designed for write intensive workloads and have very good horizontal scaling properties built into the system.
- Aeolun 2y agoWould you actually have more writes than reads? Are messages read by fewer people than post them?
- sadeshmukh 2y agoWhen you send a message, afaik it sends to all people looking at it at the time. So there is no read when in a conversation, and maybe the reads are batched when reading multiple.
- jhgg 2y agoRead traffic is much higher than write traffic due to mobile clients needing to sync chat history more often as their sessions are much shorter lived. Also search queries execute 1 query per result. And don't forget people doing GDPR data dump requests. It adds up.
- dancemethis 2y ago[flagged]
- pawelduda 2y agoPretty fun read, even tho I'll never work at such scale lol
- KaoruAoiShiho 2y agoDid they go with ScyllaDB just because it was compatible with Cassandra? Would it make sense to use a totally different solution altogether if they didn't start with that.
- jhgg 2y agoYes, we wanted to migrate all our data stores away from Cassandra due to stability and performance issues. Moving to something that didn't have those issues (or at least had a different set of less severe issues) while also not having to rewrite a bunch of code was a positive.
- ericvolp12 2y agoDid you guys end up redesigning the partitioning scheme to fit within Scylla's recommended partition sizes? I assume the tombstone issue didn't disappear with a move to Scylla but incremental compaction and/or SCTS might have helped a bunch?
- jhgg 2y agoNope. Didn't change the schema, mainly added read coalescing and used ICS. I think the big thing is when Scylla is processing a bunch of tombstones it's able to do so in a way that doesn't choke the whole server. Latest Scylla version also can send back partial/empty pages to the client to limit the amount of work per query that is run.
- ericvolp12 2y agoOh that's pretty neat. Did you just end up being okay with large partitions? I've been really afraid to let partition sizes grow beyond 100k rows even if the rows themselves are tiny but I'm not really sure how much of a real-world performance impact it has. It definitely complicates the data model to break the partitions up though.
- 2y ago
- hiyer 2y agoVery well-written article. I'm happy for them that part of the solution was switching from Cassandra to drop-in replacement Scylla, rather than having to deal with something entirely different.
- SupremumLimit 2y ago[flagged]
- nerdponx 2y agoNo, what a ridiculous thing to say. Storytelling in the present tense is not new.
- phist_mcgee 2y agoYour question is rude and I hope you know that. He's walking us through the process of designing the solution. Why wouldn't present tense work for this? We're discovering things with him as he takes us along for the journey.
- GrantMoyer 2y agoThe post appears to consistenly use past tense for things that were true in the past at time of writing, and present tense for things that are true in the present or are always true. So the use of tense appears to be valid, though not following commonly prescribed style.
- m-hodges 2y agoFun article. Also fun to think about how many people have decided to document their crimes in these Cassandra nodes.
- molszanski 2y ago> In an afternoon, we extended our data service library to perform large-scale data migrations. It reads token ranges from a database, checkpoints them locally via SQLite, and then firehoses them into ScyllaDB. We hook up our new and improved migrator and get a new estimate: nine days! How many machines this migrator was running on? One? :D Sounds absurdly amazing!
- dang 2y agoDiscussed (a bit) at the time: How Discord Stores Trillions of Messages - https://news.ycombinator.com/item?id=35048410 https://news.ycombinator.com/item?id=35048410 - March 2023 (10 comments)
- xyst 2y agoHaving used discord in the past. Most of the conversations were just shit posts. Nothing serious. Why even bother storing a trillion messages of garbage in the first place?
- aurareturn 2y agoHow do you differentiate shit posts vs quality ones if you’re Discord?
- adzm 2y agoBecause that is literally what Discord is for
- hypeatei 2y agoThat's why I laugh when people say discord content needs to be indexed on the web so things are more discoverable. 99% is garbage and the useful messages are scattered across channels.
- retsibsi 2y agoI'm not trying to be a smartarse but doesn't this describe the entire internet? The good stuff is rare and scattered, and that's why search is so important.
- jcgrillo 2y agoJust wait until the LLM bots start arguing with each other on discord ;)
- hypeatei 2y agoAt least with forums, there are dedicated pages for whatever is being discussed. Discord is just a collection of channels with topics being split up across multiple messages and shitposts in the middle.
- huimang 2y agoMany people within niches have discord servers for researching and discussing specific things. There is a large wealth of information locked away behind them that can be lost pretty much whenever discord decides to start pursuing different monetization strategies.
- robertclaus 2y agoVery cool that even at this scale the right vanilla SQL database just works. No fancy document store, map-reduce, or GPU implementations needed.
- hinkley 2y agoIt annoys me sometimes how effective B-trees are. Every decade has some cool breakthrough in compression, and a handful of other disciplines. But OLTP databases are still basically better B-trees.
- menaerus 2y agoLSM trees? ScyllaDB is LSM-based storage engine. RocksDB as well.
- salomonk_mur 2y agoHow is ScyllaDB (the solution used in the article) a vanilla SQL DB? Its the complete opposite!
- melodyogonna 2y agoThe syntax is SQL
- biorach 2y agoThat... doesn't necessarily mean that it's a "vanilla SQL server"
- asjfkdlf 2y agoAren’t they using a NoSQL store? They migrated from Casandra to Scylla DB
- andrewstuart 2y agoWhen you get to scale like this, I wonder if the access patterns of the application and its data might be best served by a custom data retrieval and storage application. I may be wrong but I just wonder if efficiency is lost to the generalized nature of any data storage system. The other question that comes to mind is, to what extent have the developers made a systematic effort to optimize how data is stored and retrieved? If you’re building a gigantic back end system and simply accepting that the system load is what it is then you might be missing a chance to dramatically impact the size of the task of managing that data.
- lyu07282 2y agoThey did give one example, if someone does a @everyone in a big channel, they specifically optimized their architecture to make that efficient using their custom data services.
- akimbostrawman 2y agoin cleartext
- 7bit 2y agoThe blog posts shows how great the technical expertise is at Discord. I work in IT and in my company devs are so incompetent, they don't even know how to create an M365/Azure dev tenant and constantly request *.Read write.All to our production tenant. I'm so envious! On the other hand, the HOME/END keys jump to the beginning of the input field rather than the line and the frontend devs are unable to fix this non-default behaviour for years, which makes it a fucking pain in the ass to use the Posts feature within a Discord channel. I believe the budget for the backend geniuses meant that frontend had to be juniors only.
- fastball 2y agoIn their defense, Azure is terrible.
- 7bit 2y agoI haven't found a difference to AWS, for example. They are all terrible in their own ways. But if one or the other is what you earn your money with, then at least put in the effort to be proficient with it, and not a complete dumbass. (Not you as in "you!")
- crop_rotation 2y agoHiring good is probably the most important thing for a company and also one of the hardest problem. I have seen a team of competent engineers outperform their sibling teams by 5-10x as long as each member of the team is good enough. Just 2 bad hires will slow down a team drastically. One terrible hire can do -5x work of a normal engineer.
- aaptel 2y agoThis whole problem wouldn't exist if we used distributed chat protocols which have been around for over 40 years (IRC). With the added benefit of having an open specification and multiple implementations. No walled gardens. And if you think IRC is too old for the modern world take a look at matrix or xmpp. How did we let discord take over is a mystery to me, or rather a tragedy.
- lofaszvanitt 2y agoDiscord wrapped irc in shiny paper.
- Krasnol 2y agoUsability did it. You download an exe, install it, make an account and it runs. Just like that. Everybody can do it. There are tons of useful and great software out there. Most of it is not easy for the public. Some (most?) of it doesn't even have an GUI. People rather sell their identity and even pay than suffer through too many hops.
- Intralexical 2y agoNot even a EXE. The web version is feature-complete, so you only need to click a link.
- Krasnol 2y agoYou're right. I forgot about that. I also forgot all those people who came from the TeamSpeak servers.
- maccard 2y agoIf you want to know why, look at the App Store reviews for discord and tea speak and compare them. Discord just works.
- dewey 2y agoI’m a huge IRC fan and I dislike Discord, but all these other services are way too clunky and IRC is really only usable through IRCCloud that has a relatively okay mobile app these days. Recently a very technical group I’m part of migrated from Telegram to Matrix and the user experience is just not very good. The apps are buggy, don’t look good, then in the new “Element” app SSO isn’t supported so I can’t use my account with it. There’s lots of paper cuts that are okay for someone like me who likes to figure it out but I’d never try to convince my friends to use it.
- dean2432 2y agoThey make it literally impossible to delete your old messages. It's a privacy nightmare and I wonder why the EU hasn't stepped in.
- Xen9 2y agoIn Discord culture, indeed, users usually share a shit-ton of PII in "introduction" messages from images to specific hobbies to medical information (EG "support" communities). The problem from GDPR perspective is that Discoed makes it impossible to delete those, since once thet detect your interest in trying to delete any of your accounts' data, they will try to get to "anonymisize" it. Then at least publicly your username isdisconnected from thos messages, but they can still be traced back to specific persons. Now if this also is done server side, then they would be in a situation where you'd either have to go through ton of messages or to bulk delete past messages of all to enforce the GDPR demands of an user wanting their PII deleted. EU Parliament is not a real Parliament in the sense that ONLY the Comission can propose new laws, and the elected parliament basically just votes on those. Who controls the Comission if not the people? The US State Department. Newsguard and non-Musk US bigtechs including Discord are in the same poli-financial bed of the establishment here. And they are full of previous state department workers.* Unless there is public outrage, the EU-level bodies at least will probably be owned. But Public opinion is controlled by the cyberpunk establishment that trains their LLMs & targets their campaign ads using that illegal Discord data to get political advantage. You in my view ought to "worry" about the fact that it's possible there will sooner or later no longer be escape from a permanent establishment, Orwell-style. Goes along with the theme that "cybersecurity" is the United States government level has been "war against hate speech" for years, and of course "hate speech" meaning "censorship of internal and external enemy speech." Budd Dwyers if I recall correctly shot himself in TV after writing to Biden (???) that under some conditions (that became true), the Department of Justice should have "Justice" removed from its name. --- Most of this I hold only at 50+% confidence of being broadly correct. Take with lots of salt.
- r3d0c 2y agoincoherant babbling
- dorlaor 2y agoSome additional nuggets by ScyllaDB co-founder: - Discord couldn't complete repair with Cassandra. Not the case with Scylla - Scylla has a lot in common with Cassandra, from a good reason, like the LSM tree, compaction etc. However, Scylla has a unique CPU&IO schedulers which allows us to prioritize the queries over compaction, and defer compaction to the half milisecond where we have enough idle bandwidth. We have plenty of articles about it - Scylla has a new (1.5 years) tombstone_gc=repair - a much safer mode - Scylla's new architecture of Raft and tablets was recently launched and is the next big thing for our users. Watch the cool youtube video of those tablet load balancing
- gigatexal 2y agoWhat a fun write up and a huge confidence building post for me in ScyllaDB.
- yas_hmaheshwari 2y agoDoes this article imply that don't use Cassandra. Use ScyllaDB when you think you want Cassandra
- znpy 2y agoInteresting read on one had, a bit disappointing on the other: when the solution is just "we moved to this other product" it smells of lack of serious and rigorous investigation. Also, having worked with the JVM and with GC issues I don't buy the "GC problems" point: there are a number of improvements in recent JVM release, the main being ZGC (and generational ZGC in particular). ZGC is great, I've personally witnessed sub-millisecond GC pauses (and i mean sub-millisecond stop-the-world pauses) on machines serving millions of requests per second. Garbage Collection is largely a solved problem in the industry as of today, thanks to ZGC. Other than this, also comparing latencies for machines with 9TB disks rather than 4TB disks is a bit like comparing apples and oranges: we will never know if issues at the storage layer were affecting tail latencies. Were the node having, i don't know, filesystem fragmentation issues? Does the 9TB storage configuration deliver higher iops than the previous 4TB storage configuration? Is the same kind of hardware underneat (same disk type? same disk bus? or are we talking ssd vs nvme?). As somebody that's been doing performance engineering for work, this piece is a bit appalling. Glad to see they've solved their issue though!
- ozgrakkurt 2y agoGC is a problem, and it always will be at some level. You can improve it but that doesn’t mean it is not a problem. Memory allocation and management is a problem even in c/c++ problems if you want to optimize your program, there is no universe where gc is not a problem
- dobin 2y agoSo the TL;DR is: Cassandra and ScyllaDB have bad performance when reading. So they put a cache in front.
- jhgg 2y agoNo cache. Just read coalescing. There is a big difference. Coalescing just ensures that while a query is executing if an identical query arrives, rather than sending the same query as an already executing query to the database it will wait for the existing query to complete and duplicate the result. If after this the same query arrives again, it will be issued against the database. This means we don't have to deal with cache invalidation/consistency issues while also being able to handle thundering herds, for example a large server pinging @everyone and having a bunch of people click into the channel or launch their apps in response.
- robmccoll 2y agoCassandra is essentially an append-mostly distributed fault-tolerant hash table. If you need specifically that with high write throughput, it's a good choice. I don't understand why people use it as a database. You run into it's limitations immediately and the pain of trying to use it like a database only gets worse with scale.
- mianos 2y agoAll the Casandra documentation and web site say it is a database. You can't blame anyone from getting confused. In my experience, I have never seen a project that started to use it, continue to use it after a year or so it may take a year to run into its limitations before having to replace it, with a database, like Postgres.
- LeifCarrotson 2y agoFTA: > In Cassandra, reads are more expensive than writes. This makes it insane as a message store for a chat server to me. It seems appropriate for a logging destination for a distributed system, one where you want lots of clients to dump data but most of the time you don't even need to audit the logs, so the number of reads for a given item is less than one. This is obviously not true for Discord messages.
- Squeeeez 2y agoNot too sure - I would have guessed that most of the messages are written once, read by the constant number of participants (say 1-100 or so) and then they disappear off the screen and are never accessed again, ever. Maybe a few people will scroll or search, or use some custom extension to load and export the history, but very rarely.
- atombender 2y agoThe sentence makes it sounds like Cassandra and Scylla are slow for writes, which isn't the case at all. It's just that writes require a bit less I/O. Reads are still very fast. If reads were slow, nobody would use Cassandra and Scylla for the purposes that they're being used for.
- bofaGuy 2y agoI’m lost at why a DB (Cassandra) with better write performance than read performance was ever selected for a messaging system. I feel like it’s obvious that a message will be read more than it is written (once).
- SpikeMeister 2y agoWhile it’s true that messages are read more, reading can be cached so not every read necessarily results in a DB call.
- axelthegerman 2y agoWhich seems something they added recently but was not part of the original design of using Cassandra
- remram 2y agoThe fact that it has better write speed than read speed doesn't mean that it has bad read speed. It just happens to have even better write speed. It's like how I connect my phone to my home's cable connection to send a big file. It is better at downloading than uploading, but that doesn't mean it's not the best solution for uploading.
- PaulHoule 2y agoHow is they just can’t shard the thing? Isn’t each Discord ‘server’ isolated from the others (can’t send a message from one to the other?) Why can’t they address trillions of messages by having thousands of shards that each handle billions?
- qntmfred 2y agoi usually start projects with postgres this days. i have reached the tens of millions of rows threshold without breaking a sweat, but is there any good reason postgres can't handle into the billions or trillions? any well known products at that scale that are known to use postgres?
- mxscho 2y agoJust the raw amount of data is not enough metrics to judge whether postgres is "enough". They seem to value horizontal scalability e.g. in terms of write throughput, which is easier to handle with something like their solution compared to postgres.
- bastawhiz 2y agoPostgres can pretty easily scale to billions or trillions of rows. It forces you to think carefully about how you query that data, though, and I think most beginners would find themselves in deep trouble jumping into the deep end.
- qntmfred 2y ago> most beginners would find themselves in deep trouble jumping into the deep end probably true for any database platform. postgres probably easier for beginners than cassandra
- postgresfan 2y agoThe problem with Postgres is that you have to read the doc (boring), sometimes read database books beyond chapter 2 (lmao nerds). This filters out 99% of software "engineers". So it's better to use KookaburaDB, version 0.2 just got released. It's written in Rust, and it's modern of course (whatever that fucking means: config written in YAML I guess? Complicated build and deployment?)
- consteval 2y agoThank you prostgresfan, it's nice to see a completely unbiased source on database technologies. Jokes aside, you're largely right. Postgres really does cover 99.9% of database usecases. But, I think Discord might still fall outside of that. The problem is that Discord's scale pretty much requires heavy sharding. While you can make this word with Postgres, you can tell it was never designed out of the gate for this. IMO, Discord isn't even taking it far enough. Working under the assumption every server is its own isolated pocket, I see no reason not to have 1 database (or database-like thing) per server. Then it's truly a distributed system, which matches Discord's business use cases. I often find matching business use cases to technology like this can greatly simply architecture and reduce friction.
- airocker 2y agoJust wondering if anyone considered using Postgres or another relational db. I understand it won’t do multi master replication as well but it is much more stable and predictable if you give it right amount of traffic. I guess the team had to do that part anyways for ScyllaDB
- crop_rotation 2y agoI don't think anyone runs Postgress at that scale (unless very specialized sharding setup). Given the choice between using ScyllaDB like everyone else and using Postgres in a super specialized best in the world setup, the choice becomes clear. Also keep in mind that Discord is not a huge super profitable company, so for them to develop something like vitess for Postgress would not make sense. For a small company with huge data like discord, using existing data solutions makes a lot more sense.
- airocker 2y agoThey could use vitess, citus or alloydb. They could use read replicas for read operations and single master in a shard for write. They would get many SQL features (upgrades, referential integrity etc) for free. It would allow them to extend their business logic considerably.
- crakhamster01 2y agoInteresting technical read, but I appreciated the lighthearted jokes/comments the author threw in as well. Felt like they struck the right balance - nice work!