4 ms·

Anyone here use NSQ in production (https://github.com/bitly/nsq https://github.com/bitly/nsq). If so what are your thoughts in comparison to the article?

by knodi 14y ago

matticakes 14y ago
there are a few (growing) number of production installations of NSQ... our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second. I can answer any questions you have (one of the authors)
- daemon13 14y ago
  Any advice on dealing with de-duping? Also, how was performance using JSON data format compared with ProtoBuffers & MsgPack?
  - matticakes 14y ago
    NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously. re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/weaker/cidr07p15.pdf http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency. What sort of use case are you thinking of? (context would help answer your de-dupe question)
mh- 14y ago
we do; rather happy with it. do you have any specific questions about it? the article's approach is.. strange, so I'm not sure how I can meaningfully contrast from a high-level. (disclosure: i've contributed to nsq, after we began using it in production)
- daemon13 14y ago
  Any gotchas moving from RabbitMQ to nsq? Also, are you missing any features in nsq?
  - mh- 14y ago
    didn't run into any gotchas, but I tested things very thoroughly before going into production. to the contrary- deploying static binaries (instead of the erlang environment) simplified things nicely, in my opinion. to be fair, i (personally) don't have the requisite experience tuning BEAM which probably biases my preference. while relatively high volume, our usage of RabbitMQ was straightforward and covered by the functionality offered in NSQ. like most, we were using AMQP.. so the switch to NSQ's concise wire protocol (and the associated reduction in pkt/s) saved us a lot of pain given the highly-variable performance we see in the EC2 network.

Anyone here use NSQ in production (https://github.com/bitly/nsq https://github.com/bitly/nsq). If so what are your thoughts in comparison to the article?

by knodi 14y ago

Anyone here use NSQ in production (https://github.com/bitly/nsq https://github.com/bitly/nsq). If so what are your thoughts in comparison to the article?

matticakes 14y ago

there are a few (growing) number of production installations of NSQ... our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second. I can answer any questions you have (one of the authors)

daemon13 14y ago
Any advice on dealing with de-duping? Also, how was performance using JSON data format compared with ProtoBuffers & MsgPack?
- matticakes 14y ago
  NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously. re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/weaker/cidr07p15.pdf http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency. What sort of use case are you thinking of? (context would help answer your de-dupe question)

mh- 14y ago

we do; rather happy with it. do you have any specific questions about it? the article's approach is.. strange, so I'm not sure how I can meaningfully contrast from a high-level. (disclosure: i've contributed to nsq, after we began using it in production)

daemon13 14y ago
Any gotchas moving from RabbitMQ to nsq? Also, are you missing any features in nsq?
- mh- 14y ago
  didn't run into any gotchas, but I tested things very thoroughly before going into production. to the contrary- deploying static binaries (instead of the erlang environment) simplified things nicely, in my opinion. to be fair, i (personally) don't have the requisite experience tuning BEAM which probably biases my preference. while relatively high volume, our usage of RabbitMQ was straightforward and covered by the functionality offered in NSQ. like most, we were using AMQP.. so the switch to NSQ's concise wire protocol (and the associated reduction in pkt/s) saved us a lot of pain given the highly-variable performance we see in the EC2 network.