10 ms·
How We Manage a Million Push Notifications an Hour
- srameshc 7y agoThis is cool, but I found this things called MQTT protocol and Emitter.io is one implementation of it. You can build your own notification system on top of it for a much cheaper cost.
- lachlan-sneff 7y ago1 million events an hour is just 278 a second, which is a much more surmountable number than 1 million.
- bt848 7y agoSame reaction. You could handle this kind of traffic with, just as examples, procmail or inetd on a single PC.
- messe 7y agoHell, modern microcontrollers (which run into the several hundred megahertz) could handle that load.
- fourier76 7y agoWhat if you get one push event on the first second, and then 999,999 on the last second? Still one million per hour. Don’t assume the events are evenly distributed. I see this a lot on HN. Yes, 1,000,000 divided by 3600 is indeed about 278, but why would we assume it was 278 notifications exactly every second?
- Alupis 7y agoNothing says you must handle them all at once though. You could queue and handle as your system allows... still reaching a 1 million per hour throughput.
- fourier76 7y agoSure, but just because the average is 278 per second doesn’t mean the events happen at that rate. Bursts are way harder to process, which is much more typical. But someone writes an article like “How We Handle X things every Y”, and inevitably someone does basic division and surmises that it shouldn’t be so hard.
- RhodesianHunter 7y ago> Bursts are way harder to process They aren't though. There are very few use cases that aren't bursty, and a plethora of reliable queueing systems exist.
- gnaritas 7y agoQueues eliminate bursts, bursts are not harder to process unless the response must by synchronous.
- deleted 7y ago[deleted]
- pkaye 7y agoThen they should have said they can handle a million push notifications a second.
- fourier76 7y agoThat doesn’t make sense?
- basilgohar 7y agoI think very few people, even on Hacker News, are so pedantic to argue strongly that someone is lying when they call 999,999 notifications in one second "a million".
- nwallin 7y agoQueue it. It's not particularly urgent that push notifications go out instantaneously. A 99th percentile latency of 5 minutes is fine.
- prebrov 7y agoNot if it’s your ride waiting for you in short-term parking space, or a food delivery person at your doorstep.
- ModernMech 7y ago> or a food delivery person at your doorstep. That's what doorbells are for.
- mustardo 7y agoHaha I look forward to an IoT doorbell company blogging about how they send 2 million door bell rings per second
- lwansbrough 7y agoI'm currently implementing a notification system for our network of sites. Aside from the HTTP microservice, my implementation is about the same. Which isn't much of a surprise - I expect this is a fairly common way of doing it. With that said, I was surprised to learn how complicated it can be to send notifications cross platform. Fortunately we're targeting the web, so I really only have to worry about two push provider implementations: APNs and VAPID. I really hope Apple agrees to implement VAPID sooner than later so developers can stop wasting their time. We have used OneSignal in the past, but there's something so satisfying about delivering the notification yourself. Also, a word of caution for people new to push notifications: we discovered early on that sending a million push notifications all at once is a really good way of crashing your site, as they have a surprisingly high click through rate in the first minute of sending!
- philjohn 7y agoWhy not use Amazon SNS? They abstract away the differences between APNS and whatever Google are calling their push infrastructure these days.
- deleted 7y ago[deleted]
- kondro 7y agoI don’t mean to discount the work done here, but why would a dev team build this themselves these days? There are many practically infinitely scalable services that do just this with various levels of sophistication. From the simple AWS SNS ($0.50/million messages) or Google Firebase (free) solutions that do multi-device push messaging whilst managing keys, redelivery and delivery responses to the more managed services like Urban Airship and OneSignal. The actual Android delivery component has to end up going through Firebase anyway. The only part of this solution SNS and Firebase don’t provide are the user fan-out functionality. Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications.
- thewarrior 7y agoPretty soon your app is just a bunch of proprietary cloud provider products glued together and now there’s vendor lock-in and much less flexibility. You might want to throttle messages based on user settings or say ML models to avoid too many annoying messages. For critical stuff you might want to switch over to email or SMS. These are startups that have the funding to do this kind of thing. This is a core part of the stack for any such company.
- kondro 7y agoIf that's your concern, lean on Firebase. You have to use Firebase to deliver messages to Android anyway, having it deliver to APNS as well you get for free with that implementation. Maybe you do want to do those things, but this implementation doesn't, nor do they mention the intention to do that. But if you did, SNS also provides implementations for delivering SMS and Email (preferably through SES). Startups shouldn't waste their money on this type of thing, especially if you consider most startups will fail. Vendor lock-in is a pretty stupid thing to worry about for an early-stage start-up (and I argue it's stupid to worry about long-term too if that vendor is AWS, Google or Azure who are all very competitive with each other).
- thewarrior 7y ago
- gdeglin 7y agoWe had a similar challenge supporting sending this many notifications at OneSignal, which we solved using Rust. We recently hit a peak of 850 Million notifications per second, and 5 billion notifications per day. Here's a blog post on how we do it. Written back when we were at "only" 2 billion notifications per week: https://onesignal.com/blog/rust-at-onesignal/ https://onesignal.com/blog/rust-at-onesignal/
- sk5t 7y agoThe one-day record could have been serviced in 5.8 seconds of peak per-second throughput, is that right?
- gdeglin 7y agoIt's not a perfect comparison because we benefit slightly by batching messages with the same content when sending to Android devices, and we have some backend features in place to reduce storage usage when the content is the same across many messages. iOS, however, is still quite challenging at scale since there's no batching mechanism for APNS, so even bandwidth becomes a bottleneck. We do web push too, which requires a lot of CPU cycles to encrypt each payload for its recipient.
- ddorian43 7y agoHow does sharding/partitioning works in these cases ? How do you partition persistent connections going to 1 url/load-balancer in the backend ?
- broth 7y agoI really like the diagram in this blog post. Is that diagram custom made or did you use software to create it?
- bsaul 7y agoROFL : I was about to post a message here bragging how i achieved 2 millions push a minute with a simple kafka queues + a cluster of push servers running on go architecture, but now i feel humbled !
- ian0 7y agoI used to have pretty much the same reaction as most of the comments here on gojek engineering posts. Coming from a telco background the RPS figures were quite low while the complexity of architecture seemed completely over the top. However, I now know more about the company. They have one of the most lean engineering teams of comparable companies here. Their CTO & VP Eng are incredibly practical guys whose advice would resonate with pretty much everyone here (Eg interview here[0] where they repeat the dangers of scaling too fast). They have small multifunctional product teams and the microservices architecture fits to this (as opposed to building it for fanciness / resumes sake). And they did it. Theres dozens of "features" in the app which are entire giant businesses in their own right. So Id imagine the shortest path to getting this up and running this was definitely considered, would have been nice to go into more detail as to why simpler / outsourced solutions would work but id guess theres a good reason. [0] https://www.youtube.com/watch?v=He0XBBfCEVk https://www.youtube.com/watch?v=He0XBBfCEVk
- mkagenius 7y ago> but id guess theres a good reason. Sometimes there are no good reasons, and we should have the ability to question it sometimes, that's how progress is made, in science and in engineering.
- didibus 7y agoI've seen a few good example blog posts discussing this scale, the 10 to 500 transactions per second and learned a lot from all of them. Does anyone know of any description of architectures scaling to the next magnitude? 1000 to 10 000 TPS ? And even higher to say 100 000 to 1 000 000 TPS ?
- random42 7y agoIts difficult to "guess" ahead of the fact how you are going to scale for the next 10x, because you need to know. a.) Which component is going to start breaking, and that depends on the usage pattern. b.) Which business/tech compromises are ok to make, and that depends on a.) Generally speaking though, you'd try to benchmark the system to find the bottleneck component, and based on the nature of it, either try to throw more hardware (horizontal or vertical scaling) or optimize the software.
- blackoil 7y agoWe handle peaks of 800k tps in few systems. It is for an analytical platform. Partition in Kafka by some evenly distributed key, Create simple apps that read from a partition and process it, commit offset. Avoid communication between process/threads. Repartition using kafka only. For some cases we had to implement sampling wherein usecase required highly skewed partitions.
- jzl 7y agoI appreciate the positive effort to communicate something useful, but this basically came down to "We use RabbitMQ".
- kaycebasques 7y agoI had a blast using Go-Jek and Grab when I visited Indonesia. It’s basically Uber/Lyft, except you sit on the backseat of a scooter rather than a car. https://www.instagram.com/p/BZHty8NnTVZ/?igshid=5qg5ndm9r9ui https://www.instagram.com/p/BZHty8NnTVZ/?igshid=5qg5ndm9r9ui Of course as others mentioned it’s a super app so they do many other things, but I believe this is one of their main businesses.
- thelittleone 7y agoGo-Jek has been huge for us as holiday rental managers in Bali. But lately there seems to be some sort of racket by drivers: Example for Go-send (sending a package): "I have motorbike problem please can you cancel". A minute after cancelling I get a message on WhatsApp: "Hey fixed my motorbike, still need?". Same thing with food orders. Example for the Go-food: "Hey I'm at the restaurant, the phone is down. Can you cancel order? I will do manually". Obviously the restaurant and the driver gets the full customer payment this way. Go-jek gets zero. This depends on the customer cancelling as otherwise the drivers rating takes a hit.
- reportgunner 7y agoWow that's really clever!
- thelittleone 7y agoThe average monthly full time salary in Bali for a local is around $270 USD. A Gofood order could easily be $100. Last time I checked Gojek takes 20% of the order value. On a $100 order that could get them $10 (assuming they split the $20 evenly with the restaurant). One a day for a month and you've made an additional average monthly salary. If you think about that in terms of a Western IT annual income (assume $60k) that's an extra $5k a month.
- uvu 7y agoI have done similar like. I think I do much better than those. But, I don't pass their interview haha.
- jbverschoor 7y agoWhy is less than 300tx/sec a such a big deal?
- finchisko 7y agoI thought this would be about implementing custom notification server and not using google's or apple's.
- reportgunner 7y agoIs that a lot of notifications ?
- londons_explore 7y ago1 Million per hour is only 300 per second. On a 1.5 Ghz 4 core raspberry pi, that gives you 21 million clock cycles to deal with each message. The architecture seems rather overengineered considering a single raspberry pi could do the job, even after 100x scaling!
- lmilcin 7y agoI wrote an algorithmic trading framework to work on Warsaw Stock Exchange. WSE back then had a limit of 10k messages per second (as in one message every 1/10000th of a second). Messages came by two separate network operators so that was 20k messages to be deduplicated to 10k operations. Incomming messages came in compressed so they required uncompressing. Responding to the message required complex processing and then may have resulted in an order to market which, again, had to be constructed and validated. All this worked on a single server (regular two-socket Xeon-based server). There was 10us (microseconds..) time budget to send response to the market and it had to work every time even under maximum possible load (10k messages per second). Routing and forwarding 300 messages per second doesn't seem like something to brag about...
- cristaloleg 7y agoHi from Warsaw, care to tell something more about architecture, used technologies, numbers? Thanks in advance.
- thewarrior 7y agoAnd what happens if the raspberry pi goes down for an hour ?
- londons_explore 7y agoYou spend $35 on a spare.
- nurettin 7y agoOdroids have better price to performance in my opinion.
- dis-sys 7y ago1 million notifications per hour, or ~300 per second, seriously?
- shanipribadi 7y agoThe title of the blog is actually a click-bait. The point of the article was more that when you have multiple teams, each owning multiple products, having a single well-defined abstractions over external dependencies provided as a service that you control is important to manage risks posed by those external dependencies and to make overall maintenance easier.