4 ms·

4 billion messages sent through the platform by millions of people per day I wish companies would stop inflating their numbers by citing "per day" statistics.

4 billion messages sent through the platform by millions of people per day I wish companies would stop inflating their numbers by citing "per day" statistics. 4 billion messages per day is less than 50k/second; that sort of transaction volume is well within the capabilities of pgsql running on midrange hardware.

pixl97 4y ago
50k/s doesn't tell us about the burstiness of the messages. Most places in the US don't have much traffic at 2AM
bearjaws 4y ago
Absolutely not. This is an average, I bet their peak could be in the mid hundreds of thousands, imagine a hype moment in a LCS game. It can in theory work, but the real world would make this the most unstable platform of all the messaging platforms. Just one vacuum would bring this system down, even if it wasn't an exclusive lock... Also I would be curious how you would implement similar functionality to the URL deep linking and image posting / hosting. Mind you the answers to these will probably increase average message size. Which means more write bandwidth. Some bar napkin math shows this would be around 180GiB per hour, 24/7, 4.3TiB per day. Unless all messages disappeared within 20 days you would exceed pretty much any reasonable single-server NVME setup. Also have fun with trim and optimizing NVME write performance. Which is also going to diminish as all the drives fail due to write wear...
- cperciva 4y ago
  Absolutely not. This is an average, I bet their peak could be in the mid hundreds of thousands That's my point. Per-second numbers are far more useful than per-day numbers.
  - mbesto 4y ago
    > useful than per-day numbers Useful for what exactly? You're basically accusing the company of inflating their numbers (??) as if they are deliberating lying about their scale.
    - cperciva 4y ago
      As zorkian noted, "messages per day" is a marketing number. In a technical blog post, what matters is the numbers which are relevant for technical purposes, such as peak (or high-percentile) messages per second.
      - asymptotic 4y ago
        In a technical blog or dashboard I like to see 1) avg mean (for throughput estimates), 2) p50 (typical customer / typical load), 3) p90 / p99 / p99.99 (whatever you think your tail is) 4) p100 (max, always useful to see and know, maybe p0). Or throw in a histogram or kernel density estimate, sometimes there are really interesting patterns. I’ve never seen a technical blog give such traffic or latency details though. Edit: reading other comments, please do not read this as diminishing this blog post or Discord, great clear writing and impressive solution to an interesting problem.
      - mbesto 4y ago
        > blog post It's a blog...it IS marketing.
- kaliszad 4y ago
  So add a Redis node or two as cache - exactly the same kind of things they did basically.
marcinzm 4y ago
I mean they literally list the per second numbers a couple paragraph down: >Our databases were serving around 2 million requests per second (in this screenshot.) I doubt pgsql will have fun on mid-range hardware with 50k writes/sec, ~2 million reads/sec and 4 billion additional rows per day with few deletions.
- merb 4y ago
  actually 'few deletions' and 'few updates' is basically the happy case for postgres. MVCC in a append only system is where it shines. (because you generate way less dead tuples, thus vacuum is not that big of a problem)
dist1ll 4y ago
That's only messages sent. The article cites 2 million queries per second hitting their cluster, which are not served by a CDN. Considering latency requirements and burst traffic, you're looking at a pretty massive load.
mbesto 4y ago
What a myopic comment. > 50k/second Yes, 50k/second for every minute of the day 365/24/7. Very few companies can quote that. Not to mention: - Has complex threaded messages - Geo-redundancies - Those message are real-time - Global user base - Unknown told of features related to messaging (bot recations, ACL, permissions, privacy, formatting, reactions, etc.) - No/limited downtime, live updates Discord is technically impressive, not sure why you felt you had to diminish that.
- sammy2255 4y ago
  Geo redundancies? Discord is all in us-east-1 gcp. Unless you Meant AZ redundancy?
  - mbesto 4y ago
    > We are running more than 850 voice servers in 13 regions (hosted in more than 30 data centers) all over the world. This provisioning includes lots of redundancy to handle data center failures and DDoS attacks. We use a handful of providers and use physical servers in their data centers. We just recently added a South Africa region. Thanks to all our engineering efforts on both the client and server architecture, we are able to serve more than 2.6 million concurrent voice users with egress traffic of more than 220 Gbps (bits-per-second) and 120 Mpps (packets-per-second). I don't see anything on their messaging specifically, just assuming they would have something similar. https://discord.com/blog/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc https://discord.com/blog/how-discord-handles-two-and-half-mi...
    - zorkian 4y ago
      Our messaging stack is not currently multi-regional, unfortunately. This is in the works, though, but it's a fairly significant architectural evolution from where we are today. Data storage is going to be multi-regional soon, but that's just from a redundancy/"data is safe in case of us-east1 failure" scenario -- we're not yet going to be actively serving live user traffic from outside of us-east1.
deleted 4y ago
[deleted]
combyn8tor 4y ago
Is there a template for replying to hackernews posts linking to engineering articles? 1. Cherry pick a piece of info 2. Claim it's not that hard/impressive/large 3. Claim it can be done much simpler with <insert database/language/hardware>
zorkian 4y ago
(I work at Discord.) It's not even the most interesting metric about our systems anyway. If we're really going to look at the tech, the inflation of those metrics to deliver the service is where the work generally is in the system -- * 50k+ QPS (average) for new message inserts * 500k+ QPS when you factor in deletes, updates, etc * 3M+ QPS looking at db reads * 30M+ QPS looking at the gateway websockets (fanout of things happening to online users) But I hear you, we're conflating some marketing metrics with technical metrics, we'll take that feedback for next time.
- cperciva 4y ago
  Ideally I'd like to hear about messages per second at the 99.99th percentile or something similar. That number says far more about how hard it is to service the load than a per-day value ever will.
  - hsbauauvhabzb 4y ago
    Wouldn’t a single second slice be of limited value compared to 30+ second samples?

pixl97 4y ago

50k/s doesn't tell us about the burstiness of the messages. Most places in the US don't have much traffic at 2AM

bearjaws 4y ago

Absolutely not. This is an average, I bet their peak could be in the mid hundreds of thousands, imagine a hype moment in a LCS game. It can in theory work, but the real world would make this the most unstable platform of all the messaging platforms. Just one vacuum would bring this system down, even if it wasn't an exclusive lock... Also I would be curious how you would implement similar functionality to the URL deep linking and image posting / hosting. Mind you the answers to these will probably increase average message size. Which means more write bandwidth. Some bar napkin math shows this would be around 180GiB per hour, 24/7, 4.3TiB per day. Unless all messages disappeared within 20 days you would exceed pretty much any reasonable single-server NVME setup. Also have fun with trim and optimizing NVME write performance. Which is also going to diminish as all the drives fail due to write wear...

cperciva 4y ago
Absolutely not. This is an average, I bet their peak could be in the mid hundreds of thousands That's my point. Per-second numbers are far more useful than per-day numbers.
- mbesto 4y ago
  > useful than per-day numbers Useful for what exactly? You're basically accusing the company of inflating their numbers (??) as if they are deliberating lying about their scale.
  - cperciva 4y ago
    As zorkian noted, "messages per day" is a marketing number. In a technical blog post, what matters is the numbers which are relevant for technical purposes, such as peak (or high-percentile) messages per second.
    - asymptotic 4y ago
      In a technical blog or dashboard I like to see 1) avg mean (for throughput estimates), 2) p50 (typical customer / typical load), 3) p90 / p99 / p99.99 (whatever you think your tail is) 4) p100 (max, always useful to see and know, maybe p0). Or throw in a histogram or kernel density estimate, sometimes there are really interesting patterns. I’ve never seen a technical blog give such traffic or latency details though. Edit: reading other comments, please do not read this as diminishing this blog post or Discord, great clear writing and impressive solution to an interesting problem.
    - mbesto 4y ago
      > blog post It's a blog...it IS marketing.
kaliszad 4y ago
So add a Redis node or two as cache - exactly the same kind of things they did basically.

marcinzm 4y ago

I mean they literally list the per second numbers a couple paragraph down: >Our databases were serving around 2 million requests per second (in this screenshot.) I doubt pgsql will have fun on mid-range hardware with 50k writes/sec, ~2 million reads/sec and 4 billion additional rows per day with few deletions.

merb 4y ago
actually 'few deletions' and 'few updates' is basically the happy case for postgres. MVCC in a append only system is where it shines. (because you generate way less dead tuples, thus vacuum is not that big of a problem)

dist1ll 4y ago

That's only messages sent. The article cites 2 million queries per second hitting their cluster, which are not served by a CDN. Considering latency requirements and burst traffic, you're looking at a pretty massive load.

mbesto 4y ago

What a myopic comment. > 50k/second Yes, 50k/second for every minute of the day 365/24/7. Very few companies can quote that. Not to mention: - Has complex threaded messages - Geo-redundancies - Those message are real-time - Global user base - Unknown told of features related to messaging (bot recations, ACL, permissions, privacy, formatting, reactions, etc.) - No/limited downtime, live updates Discord is technically impressive, not sure why you felt you had to diminish that.

sammy2255 4y ago
Geo redundancies? Discord is all in us-east-1 gcp. Unless you Meant AZ redundancy?
- mbesto 4y ago
  > We are running more than 850 voice servers in 13 regions (hosted in more than 30 data centers) all over the world. This provisioning includes lots of redundancy to handle data center failures and DDoS attacks. We use a handful of providers and use physical servers in their data centers. We just recently added a South Africa region. Thanks to all our engineering efforts on both the client and server architecture, we are able to serve more than 2.6 million concurrent voice users with egress traffic of more than 220 Gbps (bits-per-second) and 120 Mpps (packets-per-second). I don't see anything on their messaging specifically, just assuming they would have something similar. https://discord.com/blog/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc https://discord.com/blog/how-discord-handles-two-and-half-mi...
  - zorkian 4y ago
    Our messaging stack is not currently multi-regional, unfortunately. This is in the works, though, but it's a fairly significant architectural evolution from where we are today. Data storage is going to be multi-regional soon, but that's just from a redundancy/"data is safe in case of us-east1 failure" scenario -- we're not yet going to be actively serving live user traffic from outside of us-east1.

deleted 4y ago

[deleted]

combyn8tor 4y ago

Is there a template for replying to hackernews posts linking to engineering articles? 1. Cherry pick a piece of info 2. Claim it's not that hard/impressive/large 3. Claim it can be done much simpler with <insert database/language/hardware>

zorkian 4y ago

(I work at Discord.) It's not even the most interesting metric about our systems anyway. If we're really going to look at the tech, the inflation of those metrics to deliver the service is where the work generally is in the system -- * 50k+ QPS (average) for new message inserts * 500k+ QPS when you factor in deletes, updates, etc * 3M+ QPS looking at db reads * 30M+ QPS looking at the gateway websockets (fanout of things happening to online users) But I hear you, we're conflating some marketing metrics with technical metrics, we'll take that feedback for next time.

cperciva 4y ago
Ideally I'd like to hear about messages per second at the 99.99th percentile or something similar. That number says far more about how hard it is to service the load than a per-day value ever will.
- hsbauauvhabzb 4y ago
  Wouldn’t a single second slice be of limited value compared to 30+ second samples?