7 ms·
NSQ – A realtime distributed messaging platform designed to operate at scale
- rjeli 9y agoSegment is probably the biggest NSQ user right now, and they're moving to Kafka - any employees want to weigh in? :)
- caffeineninja 9y agoWe run 1.5 million messages per minute through our NSQ framework and we're starting to run into architectural limitations i.r.t to the number of workers in each producer/consumer pool, and are now testing/benchmarking Kafka in our staging environment.
- Roritharr 9y agoI think the biggest question is, do you think its feasible to directly start with Kafka instead of NSQ or does Kafka just require a much stronger/larger team to operate than NSQ?
- wand3r 9y agoIf you are starting off just use what you/the team is comfortable with. I mean, unless you think you will need capacity for over 25,000 messages per second.
- kasey_junk 9y agoKafka and NSQ have widely variant promises around things like durability, order, etc. In most use cases you can get NSQ like behavior out of Kafka and the inverse isn't true. Kafka's performance and added gaurantees come at the expense of being harder to operate.
- StavrosK 9y agoCan someone summarize the promises? Specifically, would NSQ work well as an easier-to-operate, Kafka alternative, or are there low-throughput use cases it's just not suitable for?
- rthille 9y agoI think it's less about throughput and more about durability guarantees. Kafka producers can specify the number of 'acks' (brokers which have written the message to disk) when they produce a message and the request will only return successfully if that can be done.
- StavrosK 9y agoI see, thank you. Can't NSQ do that, or at least get probabilistically close to it with multiple nodes (i.e. doesn't having more nodes reduce the probability of data loss)?
- be_erik 9y agoAbsolutely, but you can't rely on it to be persistent like Kafka. We use it extensively and are incredibly happy with it, but we follow best practices around not putting state in messages, making changes idempotent, and ensuring that we can always replay a message if needed. We've yet to lose any messages in production, but it could happen and we're okay with the tradeoffs between that and the operational complexity of kafka.
- StavrosK 9y ago
- be_erik 9y agoWe debated the same question and went with NSQ for now. We might need some of the guarantees that Kafka makes in the longterm or for some specific use cases, but for a no thrills distributed messaging platform that is incredibly simple to operate at scale, NSQ is pretty fantastic. Building client libraries is also a joy. I blogged about it a bit here: https://product.reverb.com/how-to-write-an-nsq-consumer-in-go-96ed8bde29ef https://product.reverb.com/how-to-write-an-nsq-consumer-in-g...
- rafaeljesus 9y agoNice one! Just as a info, I wrapped producer/consumer in a pkg https://github.com/rafaeljesus/nsq-event-bus https://github.com/rafaeljesus/nsq-event-bus also it exposes request-reply/rpc like
- titobrown 9y agoKafka is much harder to operate in a production environment, I would only start there if you have a specific reason to.
- TheHydroImpulse 9y agoEngineer @ Segment NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem. Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up your services and be done. With Kafka we had to plan how many partitions we needed and autoscaling has become a bit trickier. We now have critical services running against Kafka and started moving our whole pipeline to it as well. It's a slow process but we're getting there. We've had to build some tooling to operate Kafka and ramp up everyone else on how to use it. To be fair, we've also had to build tooling for NSQ, specifically nsq-lookup to allow us to scale up. We have an nsq-go library that we use in production along with some tooling: https://github.com/segmentio/nsq-go https://github.com/segmentio/nsq-go
- ah- 9y agoOut of interest, what kind of tooling did you build for Kafka?
- TheHydroImpulse 9y agoWe started deploying our Kafka cluster as a set of N EC2 instances but we started running into a bunch of issues (rolling the cluster, rolling an instance without moving partitions around, moving partitions around, etc...) Now we run Kafka through ECS and wrote some tooling to manage rolling the cluster and replacing brokers. krollout(1) (currently private) basically prevents partitions from becoming unavailable while rolling. Now that multiple teams are using Kakfa we started exploring how to scale up. Each team may have different requirements and isolation can become an issue. Likely more tooling will need to be built around this.
- doh 9y agoHave you ever looked at any proprietary solutions like Google's PubSub? We're running on PubSub for over year now and outside of some unplanned downtimes it's scaling very well. But as we're looking to branch out out of GCP we are looking at Kafka as an alternative. Could you comment on particular problems and challenges that you ran into? For the context, we're currently sending around 60k messages/sec and around 1k of them contains data larger than 10kb.
- agentultra 9y agoThere is quite a bit of documentation on the design but I haven't seen anything more specific along the lines of a TLA+, Lean, etc specification. There are plenty of projects like this and I'm curious how they go about creating specifications, checking their designs, etc. Would the project benefit from a formal model or proofs? A colleague and I started a side project to provide specifications for core Openstack components but we're keeping our minds open to other projects as well.
- je42 9y agowhat are the typical use cases for NSQ ?
- justinsaccount 9y agoprobably better off looking at http://word.bitly.com/post/33232969144/nsq http://word.bitly.com/post/33232969144/nsq http://nsq.io/ http://nsq.io/
- TheHydroImpulse 9y agoA bit of an older article but still useful: https://segment.com/blog/scaling-nsq/ https://segment.com/blog/scaling-nsq/
- jud_white 9y ago* Disclosure: I sometimes contribute to NSQ. We use NSQ at Dell for the commercial side of dell.com. We've been in Production with it for about 2 years. > what are the typical use cases for NSQ ? In the abstract, anything which can tolerate near real-time, at-least-once delivery, and does not need order guarantees. It also features retries and manual requeuing. It's typical to think order and exactly-once semantics are important because that's how we tend to think when we write code and work with (most) databases, and having order allows you to make more assumptions and simplify your approach. It typically comes at the cost of coordination or a bounded window of guarantees. Depending on your workload or how you frame the problem you may find order and exactly-once semantics are not that important, or it can be made unimportant (for example, making messages idempotent). In other cases order is important and it's worth the tradeoff; our Data Science team uses Kafka for these cases, but I'm not familiar with the details. Here are some concrete examples of things we built using NSQ, roughly in the order they were deployed to PROD: - Batch jobs which query services and databases to transform and store denormalized data. We process tens of millions of messages in a relatively short amount of time overnight. The queue is never the bottleneck; it's either our own code, services, or reading/writing to the database. Retries are surprisingly useful in this scenario. - Eventing from other applications to notify a pinpoint refresh is needed for some data into the denormalized store (for example, a user updated a setting in their store, which causes a JSON model to update). - Purchase order message queue, both for the purpose of retry and simulating what would happen if a customer on a legacy version of the backend was migrated to the new backend; also verifying a set of known 'good' orders continue to be good as business logic evolves (regression testing). - Async invoice/email generation. This is a case where you have to be careful of at-least-once delivery and need to use a correlation ID and persistence layer to define a 'point of no return' (can't process the message again beyond this point even if it fails). We don't want to email (or bill) customers twice. - Build system for distributing requests to our build farm. - Pre-fetching data and hydrating a cache when a user logs in or browses certain pages, anticipating the likely next page to avoid having the user wait on these pages for an expensive service call. The client in this case is another decoupled web application; the application emitting the event is completely separate and likely on a different deployment schedule from the emitting application. The event emitted tells us what the user did, and it's the consumer's responsibility to determine what to do. This is an interesting case where we use #ephemeral channels, which disappear when the last client disconnects. We append the application's version to the channel name so multiple running versions in the same environment will each get their own copy of the message, and process it according to that binary's logic. This is useful for blue/green/canary testing and also when we're mid-deployment and have different versions running in PROD, one customer facing and one internal still being tested. I think I refer to this image more than any other when explaining NSQ's topics and channels: https://f.cloud.github.com/assets/187441/1700696/f1434dc8-6029-11e3-8a66-18ca4ea10aca.gif https://f.cloud.github.com/assets/187441/1700696/f1434dc8-60... (from http://nsq.io/overview/design.html http://nsq.io/overview/design.html). Operationally, NSQ has been not just a pleasure to work with but inspirational to how we develop our own systems. Being operator friendly cannot be overrated. Last thing, if you do monitoring with Prometheus I recommend https://github.com/lovoo/nsq_exporter https://github.com/lovoo/nsq_exporter.
- deleted 9y ago[deleted]
- matticakes 9y agoI'm one of the original authors, happy to answer any questions.
- est 9y agoThe nsqadmin was written in backbone, it also requires statsd and graphite which are kinda obsolete these days.