11 ms·

I think this has a lot of roots from NSQ. But, NSQ has no replication support. I think built in replication is very nice to have. Would like to try once this

by arunoda 12y ago

I think this has a lot of roots from NSQ. But, NSQ has no replication support. I think built in replication is very nice to have. Would like to try once this arrives.

politician 12y ago
NSQ writes messages sent to a topic onto all channels subscribing to the topic which can be used as a form of replication in a way that meshes well with its at-least-once semantics.
- arunoda 12y ago
  But, that all happens inside one NSQ. I mean what if that NSQ server goes down. We don't have that kind of guarantee in NSQ. To get rid of that, we need to maintain the replication our own by publishing the message to two different NSQ servers.
  - politician 12y ago
    Good points.
  - mnutt 12y ago
    True, but with NSQ being brokerless, the lines are somewhat blurred between client and server. Each of our app servers has its own local nsqd. If that nsqd stops responding, we take the app server out of the load balancer. The local nsqd publishes to multiple channels, which get consumed by hosts in different datacenters. There are still potential failure cases: nsqd loses connection to consumers, messages start building up, then the machine somehow goes away. The only way to prevent it is to take the machine out of the load balancer any time nsqd messages start backing up, but we prioritize serving requests over sending messages.
    - arunoda 12y ago
      Yes. I didn't want to rant our NSQ. This is the reason we choose NSQ because of it's simplicity. We maintain NSQ close to the consumer. We don't our publisher to take responsibility to the message processing. Once it's push to the queue, we need to make sure it's getting process anyway. That's why we publish same message to multiple queues. All our DB operations are idempotent. So, we are okay with processing the same message multiple times.

11 ms·

I think this has a lot of roots from NSQ. But, NSQ has no replication support. I think built in replication is very nice to have. Would like to try once this

by arunoda 12y ago

I think this has a lot of roots from NSQ. But, NSQ has no replication support. I think built in replication is very nice to have. Would like to try once this arrives.

politician 12y ago
NSQ writes messages sent to a topic onto all channels subscribing to the topic which can be used as a form of replication in a way that meshes well with its at-least-once semantics.
- arunoda 12y ago
  But, that all happens inside one NSQ. I mean what if that NSQ server goes down. We don't have that kind of guarantee in NSQ. To get rid of that, we need to maintain the replication our own by publishing the message to two different NSQ servers.
  - politician 12y ago
    Good points.
  - mnutt 12y ago
    True, but with NSQ being brokerless, the lines are somewhat blurred between client and server. Each of our app servers has its own local nsqd. If that nsqd stops responding, we take the app server out of the load balancer. The local nsqd publishes to multiple channels, which get consumed by hosts in different datacenters. There are still potential failure cases: nsqd loses connection to consumers, messages start building up, then the machine somehow goes away. The only way to prevent it is to take the machine out of the load balancer any time nsqd messages start backing up, but we prioritize serving requests over sending messages.
    - arunoda 12y ago
      Yes. I didn't want to rant our NSQ. This is the reason we choose NSQ because of it's simplicity. We maintain NSQ close to the consumer. We don't our publisher to take responsibility to the message processing. Once it's push to the queue, we need to make sure it's getting process anyway. That's why we publish same message to multiple queues. All our DB operations are idempotent. So, we are okay with processing the same message multiple times.