6 ms·

RSMQ: A lightweight Redis based message queue for Node.js

36 points by Exinferis 11y ago

vkjv 11y ago
First off, this looks great! It has a simple interface and could compliment some simple projects very well. However, this quote, "If you run a Redis server and currently use Amazon SQS or a similar message queue you might as well use this fast little replacement" is greatly over simplifying things. Amazon SQS gives you acknowledgement guarantees that are not possible with Redis. Don't get me wrong, I love Redis, but you can't reliably trust that writes happen. https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis
rubiquity 11y ago
> Guaranteed delivery of a message to exactly one recipient within a messages visibility timeout. I'll just leave this here for you: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ http://bravenewgeek.com/you-cannot-have-exactly-once-deliver... Welcome to message queues. Where the network is both your best friend and enemy.
- Smrchy 11y ago
  It's not exactly "once". It's exactly "one" within a defined period of time. So in that articles context it would be the "at-least-once" variant. If the recipient fails to process the message for some reason it will reappear after the visibility timeout.
  - zwily 11y ago
    I don't think you can provide exactly once even with that constraint. What if the defined period of time was 100 years?
    - kasey_junk 11y ago
      Then you need a very big buffer.
- kitd 11y ago
  MQTT uses a 2PC-like mechanism for exactly-once delivery [1]. The sender sends and the receiver acks as usual. But the message is only delivered once the sender then sends a release message, which the receiver acks. End result: sender can assume receiver distributed the message only once. [1] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718102 http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-o...
  - kasey_junk 11y ago
    That's not exactly once. There are several failure modes that can cause duplication. You can detect duplication but not prevent it.
    - kitd 11y ago
      It depends on how you define the scope of the delivery, which is mentioned in the article. MQTT's exactly-once ensures that the receiver delivers the message to the end application exactly once, even if there had been duplicated messages between the sender and the receiver.
      - kasey_junk 11y ago
        Sure. But that is true of all messaging systems. When people are discussing whether something is at-most-once or at-least-once they are talking about independent of the protocol. The reason that is important is because it has large ramifications for other parts of the delivery promise (such as order guarantees) or the implementation of the protocol (such as how chatty MQTT is). It is relatively trivial to implement a system with exactly once delivery if you allow for client filtering and don't have performance or delivery time constraints.
  - deleted 11y ago
    [deleted]
hoggle 11y ago
Did anybody here use the corresponding worker module too? How does it compare with kue (in production)?
- Exinferis 11y ago
  We're have been using this in production for a few months now. We never tried kue but used the rsmq-worker module extensively. Currently we're processing about 30 million messages per day on a single redis server with two worker instances.
  - hoggle 11y ago
    Thanks, I will give it a try - even though I have been very happy with kue.
juliangregorian 11y ago
What does this do that I can't get from plain Redis Pub Sub?
- Exinferis 11y ago
  This is a lot more than pub/sub. A message queue stores messages so that nothing gets lost, when there are no subscribers listening. A new message will be delivered to only one recipient at a time while in a pub/sub system all subscribers would receive a message and would either all work on the same message/job or would have to decide who does what. Whoever receives a message will "work" on that message and after success will delete the message. Usually within seconds. So no one will ever receive that message again. If a receiving process crashes or some error happens the message will just pop up again and will be received again after a set invisibility time (default is 30s). So dropping messages in this message queue keeps them there until some receiver(s) delete them. It does not matter how many message producers / consumers there are. The queue handles the problem of not delivering a message to two consumers within a set time.
vlucas 11y ago
Nice library. This is unrelated, but what are you using for the docs? I love how it looks and scrolls with the topics and code examples right beside the methods.
- skinofstars 11y ago
  Shame they're unviewable with the HTTPS Everywhere plugin enabled.
- o_____________o 11y ago
  https://github.com/rstacruz/flatdoc https://github.com/rstacruz/flatdoc
vasquque 11y ago
Simply queues in Lua https://github.com/tarantool/queue https://github.com/tarantool/queue i don't know what's reason to use redis+node.js, if exist tarantool queues

36 points by Exinferis 11y ago

vkjv 11y ago

First off, this looks great! It has a simple interface and could compliment some simple projects very well. However, this quote, "If you run a Redis server and currently use Amazon SQS or a similar message queue you might as well use this fast little replacement" is greatly over simplifying things. Amazon SQS gives you acknowledgement guarantees that are not possible with Redis. Don't get me wrong, I love Redis, but you can't reliably trust that writes happen. https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis

rubiquity 11y ago

> Guaranteed delivery of a message to exactly one recipient within a messages visibility timeout. I'll just leave this here for you: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ http://bravenewgeek.com/you-cannot-have-exactly-once-deliver... Welcome to message queues. Where the network is both your best friend and enemy.

Smrchy 11y ago
It's not exactly "once". It's exactly "one" within a defined period of time. So in that articles context it would be the "at-least-once" variant. If the recipient fails to process the message for some reason it will reappear after the visibility timeout.
- zwily 11y ago
  I don't think you can provide exactly once even with that constraint. What if the defined period of time was 100 years?
  - kasey_junk 11y ago
    Then you need a very big buffer.
kitd 11y ago
MQTT uses a 2PC-like mechanism for exactly-once delivery [1]. The sender sends and the receiver acks as usual. But the message is only delivered once the sender then sends a release message, which the receiver acks. End result: sender can assume receiver distributed the message only once. [1] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718102 http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-o...
- kasey_junk 11y ago
  That's not exactly once. There are several failure modes that can cause duplication. You can detect duplication but not prevent it.
  - kitd 11y ago
    It depends on how you define the scope of the delivery, which is mentioned in the article. MQTT's exactly-once ensures that the receiver delivers the message to the end application exactly once, even if there had been duplicated messages between the sender and the receiver.
    - kasey_junk 11y ago
      Sure. But that is true of all messaging systems. When people are discussing whether something is at-most-once or at-least-once they are talking about independent of the protocol. The reason that is important is because it has large ramifications for other parts of the delivery promise (such as order guarantees) or the implementation of the protocol (such as how chatty MQTT is). It is relatively trivial to implement a system with exactly once delivery if you allow for client filtering and don't have performance or delivery time constraints.
- deleted 11y ago
  [deleted]

hoggle 11y ago

Did anybody here use the corresponding worker module too? How does it compare with kue (in production)?

Exinferis 11y ago
We're have been using this in production for a few months now. We never tried kue but used the rsmq-worker module extensively. Currently we're processing about 30 million messages per day on a single redis server with two worker instances.
- hoggle 11y ago
  Thanks, I will give it a try - even though I have been very happy with kue.

juliangregorian 11y ago

What does this do that I can't get from plain Redis Pub Sub?

Exinferis 11y ago
This is a lot more than pub/sub. A message queue stores messages so that nothing gets lost, when there are no subscribers listening. A new message will be delivered to only one recipient at a time while in a pub/sub system all subscribers would receive a message and would either all work on the same message/job or would have to decide who does what. Whoever receives a message will "work" on that message and after success will delete the message. Usually within seconds. So no one will ever receive that message again. If a receiving process crashes or some error happens the message will just pop up again and will be received again after a set invisibility time (default is 30s). So dropping messages in this message queue keeps them there until some receiver(s) delete them. It does not matter how many message producers / consumers there are. The queue handles the problem of not delivering a message to two consumers within a set time.

vlucas 11y ago

Nice library. This is unrelated, but what are you using for the docs? I love how it looks and scrolls with the topics and code examples right beside the methods.

skinofstars 11y ago
Shame they're unviewable with the HTTPS Everywhere plugin enabled.
o_____________o 11y ago
https://github.com/rstacruz/flatdoc https://github.com/rstacruz/flatdoc

vasquque 11y ago

Simply queues in Lua https://github.com/tarantool/queue https://github.com/tarantool/queue i don't know what's reason to use redis+node.js, if exist tarantool queues