16 ms·

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI t

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools. The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on behalf of workers? Then the topic/channel distinction seems artificial when wildcards would suffice (and provide much more flexibility), eg topic/*/channels or topic/channels/# in the MQTT parlance. MQTT also has more fine-grained delivery guarantees via its QoS levels. All this with a header structure that's as small as two bytes. edit: While MQTT is pubsub, as long as the system is under your control, you're free to change the semantics at the broker side from "broadcast messages to all consumers" to "rotate messages among consumers."

retrovirus 14y ago
Yes, it would be nice if someone from the team behind NSQ elaborates a bit on why they didn't go with an existing protocol like MQTT.
matticakes 14y ago
one of the developers here... The protocols that exist in NSQ now are designed to be the simplest implementation that worked. You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP based publishing protocol. Re: your point on sending metadata with SUB commands. I agree its a bit ugly. We actually intend on improving that aspect by instead sending the data in the form of an IDENTIFY type command upon initial connection. That information is used in the various administrative UIs and endpoints. I'm going to do a bit more reading on MQTT, thanks for the link.
- old_sound 14y ago
  I'm really curious to know why would you build this tool instead of using, say, RabbitMQ? DISCLAIMER: I wrote the RabbitMQ in Action book.
  - matticakes 14y ago
    We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages. We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures. That said, I think RabbitMQ is a good tool depending on your requirements. I can imagine a broker proving useful in situations where you may want strict ordering or no-duplication. Those were tradeoffs we were willing to make for fault tolerance and availability. Also, given the fact that we were already operating infrastructure built on simplequeue (which is also distributed and brokerless) we found it more appealing to evolve rather than replace.
    - jallmann 14y ago
      A broker is an implementation detail; there's nothing stopping you from going peer-to-peer with $protocol and having publishers manage their own subscribers. Some sort of registry is still needed though; nslookupd in your case. There is a bit of an impedance mismatch since now you have to go out-of-band to fetch information. Not saying it's a bad thing; that's essentially also the role of DNS and LDAP, among others, so it is a fairly common pattern.
    - rabbitmq 14y ago
      Master-slave etc are HA techniques for replicating state eg queues (or "channels" as nsq calls them?). "Decentralized and distributed" has nothing to do with state, master/slave or any of that; it's a topological property of your system. You can certainly build a decentralized and distributed system using a set of brokers, regardless of whether they support HA.
      - matticakes 14y ago
rabbitmq 14y ago
shameless rabbitmq mqtt plug here :-) http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/ http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/

retrovirus 14y ago

Yes, it would be nice if someone from the team behind NSQ elaborates a bit on why they didn't go with an existing protocol like MQTT.

matticakes 14y ago

one of the developers here... The protocols that exist in NSQ now are designed to be the simplest implementation that worked. You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP based publishing protocol. Re: your point on sending metadata with SUB commands. I agree its a bit ugly. We actually intend on improving that aspect by instead sending the data in the form of an IDENTIFY type command upon initial connection. That information is used in the various administrative UIs and endpoints. I'm going to do a bit more reading on MQTT, thanks for the link.

old_sound 14y ago
I'm really curious to know why would you build this tool instead of using, say, RabbitMQ? DISCLAIMER: I wrote the RabbitMQ in Action book.
- matticakes 14y ago
  We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages. We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures. That said, I think RabbitMQ is a good tool depending on your requirements. I can imagine a broker proving useful in situations where you may want strict ordering or no-duplication. Those were tradeoffs we were willing to make for fault tolerance and availability. Also, given the fact that we were already operating infrastructure built on simplequeue (which is also distributed and brokerless) we found it more appealing to evolve rather than replace.
  - jallmann 14y ago
    A broker is an implementation detail; there's nothing stopping you from going peer-to-peer with $protocol and having publishers manage their own subscribers. Some sort of registry is still needed though; nslookupd in your case. There is a bit of an impedance mismatch since now you have to go out-of-band to fetch information. Not saying it's a bad thing; that's essentially also the role of DNS and LDAP, among others, so it is a fairly common pattern.
  - rabbitmq 14y ago
    Master-slave etc are HA techniques for replicating state eg queues (or "channels" as nsq calls them?). "Decentralized and distributed" has nothing to do with state, master/slave or any of that; it's a topological property of your system. You can certainly build a decentralized and distributed system using a set of brokers, regardless of whether they support HA.
    - matticakes 14y ago

rabbitmq 14y ago

shameless rabbitmq mqtt plug here :-) http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/ http://www.rabbitmq.com/blog/2012/09/12/mqtt-adapter/