7 ms·

Is there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.

cowthulhu 2y ago
I’m not sure if Postgres would have enough horizontal scaling to accommodate the insane volume of reads and writes. I would be super interested to be proven wrong though… anyone know of a cluster being run at that scale?
riku_iki 2y ago
> Scale certainly wouldn't be it. vanilla postgres can't scale to such size, you need some sharding solution on top, which likely will be much harder to maintain than ScyllaDB..
justnoise 2y ago
I'd guess that Discord's storage systems lean towards processing a lot more writes than reads. Postgres and other databases that use B-tree indexing are ideally suited for read heavy workloads. LSM based databases like Cassandra/Scylla are designed for write intensive workloads and have very good horizontal scaling properties built into the system.
- Aeolun 2y ago
  Would you actually have more writes than reads? Are messages read by fewer people than post them?
  - sadeshmukh 2y ago
    When you send a message, afaik it sends to all people looking at it at the time. So there is no read when in a conversation, and maybe the reads are batched when reading multiple.
- jhgg 2y ago
  Read traffic is much higher than write traffic due to mobile clients needing to sync chat history more often as their sessions are much shorter lived. Also search queries execute 1 query per result. And don't forget people doing GDPR data dump requests. It adds up.
ericvolp12 2y ago
ScyllaDB scales horizontally on a shard-per-core architecture with a ballpark throughput of 12,500 Reads and 12,500 Writes per second per shard. If you're running Scylla across a total of 64 cores (maybe on 4 VMs with 16 vCPUs each), you can get up to 800k Reads 800k Writes per sec of throughput with P99 writes of <500us and p99 reads of <2ms. You will not be able to get that performance out of Postgres and the write scaling will also be impossible on a non-sharded DB. If you're a company like Discord and are running dozens (70-something?) of ScyllaDB nodes, likely each with 32 or 64 vCPUs, you've got capacity for 50M+ reads/writes per second across the cluster assuming your read/write workloads are evenly balanced across shards.
- ryanjshaw 2y ago
  Okay but this is where I get confused. Why does Discord need a single database system when discord servers are independent, right? And the volume of traffic per Discord server must be human-processable or what would the point be? A Discord server doing 800k writes per second makes no sense. So why not a RDBMS per Discord server, and if you want to ship all that out to a warehouse for analytics you do that as a separate problem? Or is it that spinning up a Postgres instance per Discord server ends up being significantly more expensive than these mega distributed database systems?
  - jhgg 2y ago
    There are ballpark of a few hundred million discord servers... do you really want to run that many Postgres instances? And even so what do you do about DM/GDMs? Easier to just run one big mega cluster for messages.
    - ryanjshaw 2y ago
      Okay so the latter then - economies of scale. Surprised to hear that few hundred million figure - I thought it'd be 1/10th of that at most! Wow. Although I did expect there'd be a very long tail, and you might choose to host a bunch of servers on a single RDBMS, at that scale yeah it wouldn't solve much. Thanks for coming back to me, appreciate it.

I’m not sure if Postgres would have enough horizontal scaling to accommodate the insane volume of reads and writes. I would be super interested to be proven wrong though… anyone know of a cluster being run at that scale?

riku_iki 2y ago

> Scale certainly wouldn't be it. vanilla postgres can't scale to such size, you need some sharding solution on top, which likely will be much harder to maintain than ScyllaDB..

justnoise 2y ago

I'd guess that Discord's storage systems lean towards processing a lot more writes than reads. Postgres and other databases that use B-tree indexing are ideally suited for read heavy workloads. LSM based databases like Cassandra/Scylla are designed for write intensive workloads and have very good horizontal scaling properties built into the system.

Aeolun 2y ago
Would you actually have more writes than reads? Are messages read by fewer people than post them?
- sadeshmukh 2y ago
  When you send a message, afaik it sends to all people looking at it at the time. So there is no read when in a conversation, and maybe the reads are batched when reading multiple.
jhgg 2y ago
Read traffic is much higher than write traffic due to mobile clients needing to sync chat history more often as their sessions are much shorter lived. Also search queries execute 1 query per result. And don't forget people doing GDPR data dump requests. It adds up.

ericvolp12 2y ago

ScyllaDB scales horizontally on a shard-per-core architecture with a ballpark throughput of 12,500 Reads and 12,500 Writes per second per shard. If you're running Scylla across a total of 64 cores (maybe on 4 VMs with 16 vCPUs each), you can get up to 800k Reads 800k Writes per sec of throughput with P99 writes of <500us and p99 reads of <2ms. You will not be able to get that performance out of Postgres and the write scaling will also be impossible on a non-sharded DB. If you're a company like Discord and are running dozens (70-something?) of ScyllaDB nodes, likely each with 32 or 64 vCPUs, you've got capacity for 50M+ reads/writes per second across the cluster assuming your read/write workloads are evenly balanced across shards.

ryanjshaw 2y ago
Okay but this is where I get confused. Why does Discord need a single database system when discord servers are independent, right? And the volume of traffic per Discord server must be human-processable or what would the point be? A Discord server doing 800k writes per second makes no sense. So why not a RDBMS per Discord server, and if you want to ship all that out to a warehouse for analytics you do that as a separate problem? Or is it that spinning up a Postgres instance per Discord server ends up being significantly more expensive than these mega distributed database systems?
- jhgg 2y ago
  There are ballpark of a few hundred million discord servers... do you really want to run that many Postgres instances? And even so what do you do about DM/GDMs? Easier to just run one big mega cluster for messages.
  - ryanjshaw 2y ago
    Okay so the latter then - economies of scale. Surprised to hear that few hundred million figure - I thought it'd be 1/10th of that at most! Wow. Although I did expect there'd be a very long tail, and you might choose to host a bunch of servers on a single RDBMS, at that scale yeah it wouldn't solve much. Thanks for coming back to me, appreciate it.