4 ms·

Scaling NSQ to 750B Messages (2016)

60 points by tepidandroid 7y ago

bruth 7y ago
Note this is from 2016
- tepidandroid 7y ago
  thanks, I updated the title
tgtweak 7y ago
Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny. Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.
- jonathanoliver 7y ago
  As of 2017 they were migrating everything over to Kafka: https://news.ycombinator.com/item?id=14451148 https://news.ycombinator.com/item?id=14451148
- wcdolphin 7y ago
  Out of curiosity, are you using the durable, NATS-Streaming server, or just the ephemeral NATS-server? I’d be interested to hear more about the performance differences you see.
  - tgtweak 7y ago
    Using a single ephemeral instance with request/response mechanics as a kind of load balancer for messages. Publishers have persistency so it is not needed in the queue. 120Mb of memory usage and <2% of one core handling 10000 messages/s (as per nats-top) and it's been up for 2 years without and increase in memory size or failed messages. The real allure was the guarantee that only one subscriber would get a request much like a traditional load balancer. The performance was an added benefit. When we put it together a few years ago the cloistering wasn't very mature and the streaming version with persistence wasn't completed. Would look into this over kafka if only for the simplicity in management and track record of go for performance and reliability.
quxbar 7y ago
Jehiah and Matt (the original creators of NSQ) are really talented engineers, I'm glad their work is getting some love.
caust1c 7y ago
I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.
- coulix 7y ago
  What pushed you to move to Kafka?
  - caust1c 7y ago
    Reliability, scalability and tooling (in that order). Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html https://nsq.io/overview/features_and_guarantees.html Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent. NSQ has been great honestly, it was just designed with different use cases in mind.
    - nik736 7y ago
      Was language a deciding factor? NSQ being Go while Kafka is Java.
      - caust1c 7y ago
        Not at all. We mostly use Go internally for backend services, so if anything that was a point for NSQ.
        zeroc8 7y ago
        What about NATS Streaming? Is it a viable alternative (now at version 2.0) to Kafka?
heipei 7y ago
Despite Segment using Kafka nowadays, I would encourage everyone to give NSQ a try and see if it might not already fit your use-case. For me it's been one of those do-one-thing-well pieces of software that's really well-engineered, where I can clearly understand how it works and what the scope of its features are. NSQ is my go-to queue whenever I'm working on any kind of service, even more so when that service might be standalone. Consider deploying a service on an isolated box which frequently pushes data into the cloud. Connectivity to the cloud might be intermittent, so you just store every work item to the locally running nsq and you're protected against network outages as well as your data-producing process crashing. Since NSQ is so easy to spin up and easy to interact with (Publish works via plain HTTP), it's not a lot of overhead to add it to your process.

60 points by tepidandroid 7y ago

bruth 7y ago

Note this is from 2016

tepidandroid 7y ago
thanks, I updated the title

tgtweak 7y ago

Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny. Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.

jonathanoliver 7y ago
As of 2017 they were migrating everything over to Kafka: https://news.ycombinator.com/item?id=14451148 https://news.ycombinator.com/item?id=14451148
wcdolphin 7y ago
Out of curiosity, are you using the durable, NATS-Streaming server, or just the ephemeral NATS-server? I’d be interested to hear more about the performance differences you see.
- tgtweak 7y ago
  Using a single ephemeral instance with request/response mechanics as a kind of load balancer for messages. Publishers have persistency so it is not needed in the queue. 120Mb of memory usage and <2% of one core handling 10000 messages/s (as per nats-top) and it's been up for 2 years without and increase in memory size or failed messages. The real allure was the guarantee that only one subscriber would get a request much like a traditional load balancer. The performance was an added benefit. When we put it together a few years ago the cloistering wasn't very mature and the streaming version with persistence wasn't completed. Would look into this over kafka if only for the simplicity in management and track record of go for performance and reliability.

quxbar 7y ago

Jehiah and Matt (the original creators of NSQ) are really talented engineers, I'm glad their work is getting some love.

caust1c 7y ago

I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.

coulix 7y ago
What pushed you to move to Kafka?
- caust1c 7y ago
  Reliability, scalability and tooling (in that order). Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html https://nsq.io/overview/features_and_guarantees.html Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent. NSQ has been great honestly, it was just designed with different use cases in mind.
  - nik736 7y ago
    Was language a deciding factor? NSQ being Go while Kafka is Java.
    - caust1c 7y ago
      Not at all. We mostly use Go internally for backend services, so if anything that was a point for NSQ.
      - zeroc8 7y ago
        What about NATS Streaming? Is it a viable alternative (now at version 2.0) to Kafka?

heipei 7y ago

Despite Segment using Kafka nowadays, I would encourage everyone to give NSQ a try and see if it might not already fit your use-case. For me it's been one of those do-one-thing-well pieces of software that's really well-engineered, where I can clearly understand how it works and what the scope of its features are. NSQ is my go-to queue whenever I'm working on any kind of service, even more so when that service might be standalone. Consider deploying a service on an isolated box which frequently pushes data into the cloud. Connectivity to the cloud might be intermittent, so you just store every work item to the locally running nsq and you're protected against network outages as well as your data-producing process crashing. Since NSQ is so easy to spin up and easy to interact with (Publish works via plain HTTP), it's not a lot of overhead to add it to your process.