4 ms·

We are using NSQ at Hailo for sending and receiving billions of messages every day. It scales incredibly well and is extremely reliable. We'd be happy to speak

by chuhnk 12y ago

We are using NSQ at Hailo for sending and receiving billions of messages every day. It scales incredibly well and is extremely reliable. We'd be happy to speak more about this with anyone who's interested.

nivertech 12y ago
Can one use NSQ as a replacement for Kafka?
- DAddYE 12y ago
  Depends on how you use Kafka, but yes overall you can.
- chuhnk 12y ago
  Posed this question to the creator of NSQ, Matt Reiferson just a few minutes ago. His response https://twitter.com/imsnakes/status/559827559806808069 https://twitter.com/imsnakes/status/559827559806808069
- walls 12y ago
  Kafka offers persistence, which seems extremely rare in distributable queue. (Is it even available outside of Kafka?) On that note, what is everyone using these queues for that they can ignore durability?
  - retr0h 12y ago
    Also, Kafka guarantees message order, where NSQ does not. This can be dealt with in the client, but something to note.
  - yeukhon 12y ago
    Rabbit offers persistence.
  - sytringy05 12y ago
    The standard JMS spec supports durable queues and topics. Generally a nightmare to manage though - trying to track down a unconsumed message on a clustered durable topic with lots of subscribers is like trying to find a needle in a stack of needles.
- lxfontes 12y ago
  At Uken Studios we use NSQ as a chain of queues for log processing. rails -> fluentd -> NSQ (firehose) -> log splitter (python script) -> NSQ (elasticsearch, s3, hadoop) We have this setup live since November and processed more than 25b messages without hiccups.
akerl_ 12y ago
How do you handle reliability? I'm reading http://nsq.io/overview/features_and_guarantees.html http://nsq.io/overview/features_and_guarantees.html and I'm curious how you've set up your architecture to handle node failures without losing messages, or if losing messages is acceptable in your use case. I'm also curious if you've had issues with duplicated delivery, where a message is delivered more than once, and how you've handled that case.
- chuhnk 12y ago
  Hi. Good question. We actually have both use cases. In the case where we require reliability and guaranteed delivery we publish to multiple nodes and standard channels that persist messages in the case of subscribers disconnecting or overflow in what can't be buffered in memory. In the other case we just publish once and can use ephemeral channels. We scale clusters depending on usage but start with a minimum of 3 and use nsqlookupd to find topics in the cluster. In the case of things that can't handle duplicate deliveries we offload deduping to simple things memcached.

by chuhnk 12y ago

nivertech 12y ago

Can one use NSQ as a replacement for Kafka?

DAddYE 12y ago
Depends on how you use Kafka, but yes overall you can.
chuhnk 12y ago
Posed this question to the creator of NSQ, Matt Reiferson just a few minutes ago. His response https://twitter.com/imsnakes/status/559827559806808069 https://twitter.com/imsnakes/status/559827559806808069
walls 12y ago
Kafka offers persistence, which seems extremely rare in distributable queue. (Is it even available outside of Kafka?) On that note, what is everyone using these queues for that they can ignore durability?
- retr0h 12y ago
  Also, Kafka guarantees message order, where NSQ does not. This can be dealt with in the client, but something to note.
- yeukhon 12y ago
  Rabbit offers persistence.
- sytringy05 12y ago
  The standard JMS spec supports durable queues and topics. Generally a nightmare to manage though - trying to track down a unconsumed message on a clustered durable topic with lots of subscribers is like trying to find a needle in a stack of needles.
lxfontes 12y ago
At Uken Studios we use NSQ as a chain of queues for log processing. rails -> fluentd -> NSQ (firehose) -> log splitter (python script) -> NSQ (elasticsearch, s3, hadoop) We have this setup live since November and processed more than 25b messages without hiccups.

akerl_ 12y ago

How do you handle reliability? I'm reading http://nsq.io/overview/features_and_guarantees.html http://nsq.io/overview/features_and_guarantees.html and I'm curious how you've set up your architecture to handle node failures without losing messages, or if losing messages is acceptable in your use case. I'm also curious if you've had issues with duplicated delivery, where a message is delivered more than once, and how you've handled that case.

chuhnk 12y ago
Hi. Good question. We actually have both use cases. In the case where we require reliability and guaranteed delivery we publish to multiple nodes and standard channels that persist messages in the case of subscribers disconnecting or overflow in what can't be buffered in memory. In the other case we just publish once and can use ephemeral channels. We scale clusters depending on usage but start with a minimum of 3 and use nsqlookupd to find topics in the cluster. In the case of things that can't handle duplicate deliveries we offload deduping to simple things memcached.