4 ms·

I'm glad these new projects are coming on, and this one seems to be very forthright about its limitations, but I'm just putting this in the bucket with all of t

by ibejoeb 12y ago

I'm glad these new projects are coming on, and this one seems to be very forthright about its limitations, but I'm just putting this in the bucket with all of the other messaging systems that provide a minimum feature set. I haven't seen much on the market recently that offers things like (first-class) persistence, guaranteed ordering, guaranteed delivery, or any of the other more complex distribution patterns. That's why I'm still using ActiveMQ. I'm pretty happy with it, so I suppose I'm not really looking for a replacement, but I still wonder what benefit people are getting from these newer systems.

cpitman 12y ago
I would assume its the ease of making the systems highly available and/or scalable. Scaling ActiveMQ can be a pain once you exceed the capacity of a single broker, and the standard HA setup requires shared file systems with non-broken file locks which are fairly complicated to get in cloud environments.
- sytringy05 12y ago
  +1 agree with that (although you can use the DB based persistent store). The other obvious problem with AMQ is that it supports XA transactions which means people are tempted to use it, which in turn leads to much pain and suffering for all concerned..
matticakes 12y ago
(project author) NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1]. Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools). When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees. [1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.
- ibejoeb 12y ago
  Yeah, don't get me wrong, it looks good for what it is, and certainly some of the best products come from skunkworks and hobby. Is there anywhere I can read about the specific problem at bitly you built it to facilitate? I mean, I've seen a lot of these systems where there are integration and timing concerns, so some kind of message bus is useful. But if they do get to a point where things needs to be distributed, sending messages without order and delivery guarantees just seems to be passing the buck to downstream systems to not mess up.
  - matticakes 12y ago
    Yea, in the original announcement blog post (which became the design doc): http://nsq.io/overview/design.html http://nsq.io/overview/design.html
lobster_johnson 12y ago
I would also like to see a queue system with better guarantees. We've been running RabbitMQ for a couple of years and are fed up with its lack of safety in the face of minor cluster failures. I don't know about NSQ — it looks nice, but the lack of true persistence and ordering means it's not something I can use. For example, lack of ordering means you can't use it as an event bus for document-oriented changes. I know it's not a popular sentiment these days, but I prefer to err on the side of correctness. I have had enough of systems that are supposedly SPOF-free but still fail for mysterious reasons and are nigh-impossible to debug once they do (ElasticSearch, ugh). RabbitMQ's lack of transparency means it's harder to work with the contents of a queue. For example, if the queue processing has stalled for some mysterious reason, what's clogging it? With RabbitMQ, the only way to peek into a queue is to actually pop messages from it (using a tool such as rabbitmqadmin or amqp-dequeue). Another problem with RabbitMQ is that messages go away after being acknowledged. I, for one, would like to see a bit of history. Sure, you can build this into individual apps, but it turns out this kind of history is beneficial for many kinds of apps. There's DLX, but you can't (afaik) set up rules to automatically copy acked messages into a history queue. I can't say I like AMQP. Binary protocol, requires several layers of client tooling to work with. If anything goes weird, strace or tcpdump are of no use. It's also a very complex protocol whose interpretation and matrix of support features seems to change a lot. I'm currently writing a small job manager in Go that's backed by PostgreSQL (at least initially) because I want something safer, more stable and more transparent. It's based on my realization that jobs are different from events, and should be treated differently. For example, RabbitMQ has no conception of scheduling or prioritization, and messages are ephemeral objects that go away once they've been acknowledged, as opposed to jobs, which have a life before, during and after processing. So far, it looks pretty good. Postgres can't process a bazillion messages per second, and it's not going to work for "big data", but it seems to scale decently enough.
antirez 12y ago
Hello, in the last couple of months I've been working on a message queue of the kind you describe, which is, one that is more biased about providing a number of features already built in inside the broker itself, instead of delegating it to the client. I understand the case for the other approach taken by NSQ and other systems, it's just a matter of what you want to do. However while my queue project supports persistence, synchronous replication, delayed jobs, at least once and at most once delivery semantics, automatic federation, there is one thing I don't support in the list of features you mentioned: ordering. And I believe there is a strong argument for not supporting ordering in certain kinds of message queues. By implementing only a weak form of ordering (approximated wall-clock ordering, so that, usually jobs are served in roughly insertion order) we gain: availability (as in CAP, the system can continue with a single node), latency (even in the case of synchronous replication, you need to care just for a message to be replicated into N nodes, regardless of what those N nodes are), and functionality (the queue can auto-reissue messages after a specified retry-time, so at-least-once delivery is trivial to accomplish for consumers and producers). Depending on how the system is designed, to give up strict ordering also wins you scalability. So I agree about your reasoning but my feeling is that message ordering is a big point that really changes how a message queue is shaped. My bet is that there are many problems where ordering is not needed but all the other features are.
- ryanjshaw 12y ago
  Is your queue project published anywhere? Why did you choose to support persistence? I'm dying for a messaging library that supports only brokerless publish/subscribe with QOS support, request/reply and discovery. On top of that I want to build a persistence layer supporting 'retry' and 'replay' operations; this would allow me to build highly decoupled components, with the ability to do maintenance operations (retry) and rebuild/debug/analyse/spike environments easily (replay), a model which I find supports the type of back-end business applications I build very well. DDS supports all of this including the QOS layer, but no request/reply unfortunately. The existing open source DDS implementations are massive code bases that I wouldn't dare go near to attempt extending, and DDS is heavily based on IDL, which I'd prefer avoiding. If I had the messaging library I could do the QOS + persistence layer, but so far I haven't seen anything that ticks all the boxes (ZMQ has the raw capabilities but I'd need to build out the publish/subscribe and request/reply protocol, which I honestly don't have the time/skill to do.)

by ibejoeb 12y ago

cpitman 12y ago

I would assume its the ease of making the systems highly available and/or scalable. Scaling ActiveMQ can be a pain once you exceed the capacity of a single broker, and the standard HA setup requires shared file systems with non-broken file locks which are fairly complicated to get in cloud environments.

sytringy05 12y ago
+1 agree with that (although you can use the DB based persistent store). The other obvious problem with AMQ is that it supports XA transactions which means people are tempted to use it, which in turn leads to much pain and suffering for all concerned..

matticakes 12y ago

(project author) NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1]. Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools). When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees. [1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.

ibejoeb 12y ago
Yeah, don't get me wrong, it looks good for what it is, and certainly some of the best products come from skunkworks and hobby. Is there anywhere I can read about the specific problem at bitly you built it to facilitate? I mean, I've seen a lot of these systems where there are integration and timing concerns, so some kind of message bus is useful. But if they do get to a point where things needs to be distributed, sending messages without order and delivery guarantees just seems to be passing the buck to downstream systems to not mess up.
- matticakes 12y ago
  Yea, in the original announcement blog post (which became the design doc): http://nsq.io/overview/design.html http://nsq.io/overview/design.html

lobster_johnson 12y ago

I would also like to see a queue system with better guarantees. We've been running RabbitMQ for a couple of years and are fed up with its lack of safety in the face of minor cluster failures. I don't know about NSQ — it looks nice, but the lack of true persistence and ordering means it's not something I can use. For example, lack of ordering means you can't use it as an event bus for document-oriented changes. I know it's not a popular sentiment these days, but I prefer to err on the side of correctness. I have had enough of systems that are supposedly SPOF-free but still fail for mysterious reasons and are nigh-impossible to debug once they do (ElasticSearch, ugh). RabbitMQ's lack of transparency means it's harder to work with the contents of a queue. For example, if the queue processing has stalled for some mysterious reason, what's clogging it? With RabbitMQ, the only way to peek into a queue is to actually pop messages from it (using a tool such as rabbitmqadmin or amqp-dequeue). Another problem with RabbitMQ is that messages go away after being acknowledged. I, for one, would like to see a bit of history. Sure, you can build this into individual apps, but it turns out this kind of history is beneficial for many kinds of apps. There's DLX, but you can't (afaik) set up rules to automatically copy acked messages into a history queue. I can't say I like AMQP. Binary protocol, requires several layers of client tooling to work with. If anything goes weird, strace or tcpdump are of no use. It's also a very complex protocol whose interpretation and matrix of support features seems to change a lot. I'm currently writing a small job manager in Go that's backed by PostgreSQL (at least initially) because I want something safer, more stable and more transparent. It's based on my realization that jobs are different from events, and should be treated differently. For example, RabbitMQ has no conception of scheduling or prioritization, and messages are ephemeral objects that go away once they've been acknowledged, as opposed to jobs, which have a life before, during and after processing. So far, it looks pretty good. Postgres can't process a bazillion messages per second, and it's not going to work for "big data", but it seems to scale decently enough.

antirez 12y ago

Hello, in the last couple of months I've been working on a message queue of the kind you describe, which is, one that is more biased about providing a number of features already built in inside the broker itself, instead of delegating it to the client. I understand the case for the other approach taken by NSQ and other systems, it's just a matter of what you want to do. However while my queue project supports persistence, synchronous replication, delayed jobs, at least once and at most once delivery semantics, automatic federation, there is one thing I don't support in the list of features you mentioned: ordering. And I believe there is a strong argument for not supporting ordering in certain kinds of message queues. By implementing only a weak form of ordering (approximated wall-clock ordering, so that, usually jobs are served in roughly insertion order) we gain: availability (as in CAP, the system can continue with a single node), latency (even in the case of synchronous replication, you need to care just for a message to be replicated into N nodes, regardless of what those N nodes are), and functionality (the queue can auto-reissue messages after a specified retry-time, so at-least-once delivery is trivial to accomplish for consumers and producers). Depending on how the system is designed, to give up strict ordering also wins you scalability. So I agree about your reasoning but my feeling is that message ordering is a big point that really changes how a message queue is shaped. My bet is that there are many problems where ordering is not needed but all the other features are.

ryanjshaw 12y ago
Is your queue project published anywhere? Why did you choose to support persistence? I'm dying for a messaging library that supports only brokerless publish/subscribe with QOS support, request/reply and discovery. On top of that I want to build a persistence layer supporting 'retry' and 'replay' operations; this would allow me to build highly decoupled components, with the ability to do maintenance operations (retry) and rebuild/debug/analyse/spike environments easily (replay), a model which I find supports the type of back-end business applications I build very well. DDS supports all of this including the QOS layer, but no request/reply unfortunately. The existing open source DDS implementations are massive code bases that I wouldn't dare go near to attempt extending, and DDS is heavily based on IDL, which I'd prefer avoiding. If I had the messaging library I could do the QOS + persistence layer, but so far I haven't seen anything that ticks all the boxes (ZMQ has the raw capabilities but I'd need to build out the publish/subscribe and request/reply protocol, which I honestly don't have the time/skill to do.)