5 ms·
NSQ A realtime distributed messaging platform
- jitl 6y agoWhat is this used for? Is this kinda sorta like a Kafka? Do you implement message bus with this? Queue task workers? Change data capture stuff like secondary indexes or materialized views? As an outsider without deep experience in this space, I struggle to evaluate this kind of software. When would I want this VS things-that-look-like-it? What about hosted offerings like SQS? (Maybe I should read a book)
- lkramer 6y agoWe use it for internal messaging between services. Particular between systems owned by different teams. It's reliable, easy to configure, have good out of the box backoff. We have used Kafka in the past and it just felt too heavy for what we needed.
- mnutt 6y agoIt's good for simple message passing between services, but has some caveats at scale. One nice thing is that you can read through the source in an afternoon and get a decent understanding of how it works. Segment has a good blog post about its pros: https://segment.com/blog/scaling-nsq/ https://segment.com/blog/scaling-nsq/ and later, some of the issues they faced at scale that caused them to move off of it: https://segment.com/blog/the-10m-engineering-problem/ https://segment.com/blog/the-10m-engineering-problem/ My company has used nsq in a very similar way with a similar number of messages over almost a decade at this point, and have come to the same conclusions that Segment did.
- synchrone 6y agoI searched all over the web for a reliable and scalable plain message queue that could work in a full disk-backed mode, not requiring the data to fit in memory. Was very impressed with how little RAM and CPU it consumed under load.
- NicoJuicy 6y agoDid you check out NATS?
- tapirl 6y agoBoth nsq and nats are very memory saving.
- synchrone 6y agoWas using it in production before. NATS itself is great and reliable. However disk persistency is done with nats-streaminh and i had big issues with it's Raft file based store, where it would stop accepting messages after a short network issue, and not get unstuck unless I destroy the whole nats-streaming data storage amd start from scratch. Also raft means 3x disk usage for one queue, since its fully replicated
- polskibus 6y agoDoes it offer any delivery guarantees?
- synchrone 6y agoAs far as I read the docs, if you use 0 for memory storage size - it will flush to disk before responding that the message was published OK. Delivery is "at least once".
- 02020202 6y agoi prefer kvdb-backed grpc for messaging to these pubsub projects because there is always some caveat. either it has at-least-once delivery or at-most-once delivery and other times it will disconnect clients if they are too slow or producers are too fast. there is always something. so tailor-made solution works out the best for me. easy and fast to write. works as one wants it to work. and with go, it's a piece of cake and also part of single binary, no external dependencies.
- est 6y agonsq is a solid piece of technology. 20k conections on a single nsqd, no complaints.