5 ms·
NSQ: Open-source realtime distributed messaging, billions of messages / day
- esafak 3y agoWhen you would use this over Kafka, Pulsar, Kinesis, NATS, RedPanda, Fluvio, etc.? I found this comparison from 2023 https://gcore.com/learning/nats-rabbitmq-nsq-kafka-comparison/ https://gcore.com/learning/nats-rabbitmq-nsq-kafka-compariso...
- skyde 3y agoI think those product can be sliced based on the typical use case. Kafka and Azure EventHub and RedPanda are EventStream system use Retention policy. They can handle very high throughput efficiently but are not transactional and cannot be safely used when data consistency is the priority. While system like ActiveMQ, Azure service bus, RabbitMQ are transactional message broker each subscriber to a topic has its own mailbox and message are removed from the mailbox when the client acknowledge that the message has been successfully processed. And deleted from the broker when no mailbox still contain the message. They can be used for workflow involving financial transaction (money) and are very safe. But they sometime dont maintain ordering, client might receive message in different order than the publisher put them inside the topic.
- ko_pivot 3y agoNATS especially seems very similar but more advanced.
- natbobc 3y agoProbably that’s the scenario. You want something simple and narrowly focused. Advanced and infinitely configurable isn’t always a virtue.
- deleted 3y ago[deleted]
- bob1029 3y agoBillions of messages per day is 10-100k per second on average. If you do this in a vertical on one box with something like a ring buffer abstraction, you can achieve MPSC messaging rates in excess of millions per second without trying very hard across any arbitrary # of clients. E.g. https://aeron.io/aeron-open-source/#aeron_transport https://aeron.io/aeron-open-source/#aeron_transport
- natbobc 3y agoI find it mildly devious when numbers are inflated by changing the period from seconds to days. Aeron and NSQ have slightly different design principles. Which can easily be identified in their feature list. Aeron’s origins are exchanges for fintechs and focus on predictable low latency with a tight standard deviation. NSQ puts heavy emphasis on being a distributed broker less message queue. Performance alone probably isn’t a good basis to measure their utility.
- zerotolerance 3y agoThis project has been around for a long time. I made a few contributions in like 2014 or 2015. It is a great distributed stack and a semi-comfortable alternative to things like RabbitMQ. Did something new happen?
- natbobc 3y agoLooks like a release about 2 weeks ago but otherwise not sure.
- aetherson 3y agoUsed NSQ a bunch at an old job, it was simple and reliable, never gave us any trouble, at the scale of, yes, billions of messages per day.
- vivzkestrel 3y agohow is this different from zeromq, rabbitmq?