5 ms·

We had a similar challenge supporting sending this many notifications at OneSignal, which we solved using Rust. We recently hit a peak of 850 Million notificat

We had a similar challenge supporting sending this many notifications at OneSignal, which we solved using Rust. We recently hit a peak of 850 Million notifications per second, and 5 billion notifications per day. Here's a blog post on how we do it. Written back when we were at "only" 2 billion notifications per week: https://onesignal.com/blog/rust-at-onesignal/ https://onesignal.com/blog/rust-at-onesignal/

sk5t 7y ago
The one-day record could have been serviced in 5.8 seconds of peak per-second throughput, is that right?
- gdeglin 7y ago
  It's not a perfect comparison because we benefit slightly by batching messages with the same content when sending to Android devices, and we have some backend features in place to reduce storage usage when the content is the same across many messages. iOS, however, is still quite challenging at scale since there's no batching mechanism for APNS, so even bandwidth becomes a bottleneck. We do web push too, which requires a lot of CPU cycles to encrypt each payload for its recipient.
  - ddorian43 7y ago
    How does sharding/partitioning works in these cases ? How do you partition persistent connections going to 1 url/load-balancer in the backend ?
broth 7y ago
I really like the diagram in this blog post. Is that diagram custom made or did you use software to create it?
bsaul 7y ago
ROFL : I was about to post a message here bragging how i achieved 2 millions push a minute with a simple kafka queues + a cluster of push servers running on go architecture, but now i feel humbled !
tempguy9999 7y ago
No offence but is this correct? > We recently hit a peak of 850 Million notifications per second per second? From your blog "OnePush is fast - We've observed sustained deliveries up to 125,000/second and spikes up to 175,000/second." I think you may have a typo. The bandwidth would be incredible too, if it were an unlikely 10 bytes per delivery that would be 8.5 GB/Sec.
- gdeglin 7y ago
  This post is now a couple years old, so we've grown a lot since then. Here's a newer (more marketing-centric) post where we announced our 850k/second milestone: https://onesignal.com/blog/throughput-record/ https://onesignal.com/blog/throughput-record/ It's generally under 4 bytes per delivery, depending on the content, and we have several delivery servers. APNS, for example, doesn't support payloads larger than 4 bytes.
  - Arkanta 7y ago
    I think you mean 4kb rather than 4 bytes
- gdeglin 7y ago
  Oops, you're right. There is a typo. It should say "850,000/second"

sk5t 7y ago

The one-day record could have been serviced in 5.8 seconds of peak per-second throughput, is that right?

gdeglin 7y ago
It's not a perfect comparison because we benefit slightly by batching messages with the same content when sending to Android devices, and we have some backend features in place to reduce storage usage when the content is the same across many messages. iOS, however, is still quite challenging at scale since there's no batching mechanism for APNS, so even bandwidth becomes a bottleneck. We do web push too, which requires a lot of CPU cycles to encrypt each payload for its recipient.
- ddorian43 7y ago
  How does sharding/partitioning works in these cases ? How do you partition persistent connections going to 1 url/load-balancer in the backend ?

broth 7y ago

I really like the diagram in this blog post. Is that diagram custom made or did you use software to create it?

bsaul 7y ago

ROFL : I was about to post a message here bragging how i achieved 2 millions push a minute with a simple kafka queues + a cluster of push servers running on go architecture, but now i feel humbled !

tempguy9999 7y ago

No offence but is this correct? > We recently hit a peak of 850 Million notifications per second per second? From your blog "OnePush is fast - We've observed sustained deliveries up to 125,000/second and spikes up to 175,000/second." I think you may have a typo. The bandwidth would be incredible too, if it were an unlikely 10 bytes per delivery that would be 8.5 GB/Sec.

gdeglin 7y ago
This post is now a couple years old, so we've grown a lot since then. Here's a newer (more marketing-centric) post where we announced our 850k/second milestone: https://onesignal.com/blog/throughput-record/ https://onesignal.com/blog/throughput-record/ It's generally under 4 bytes per delivery, depending on the content, and we have several delivery servers. APNS, for example, doesn't support payloads larger than 4 bytes.
- Arkanta 7y ago
  I think you mean 4kb rather than 4 bytes
gdeglin 7y ago
Oops, you're right. There is a typo. It should say "850,000/second"