4 ms·

I'm still hoping for a simple protocol with strong guarantees and persistence. RabbitMQ seems to be a very good backend implementation but its the choice of AMQ

by moe 17y ago

I'm still hoping for a simple protocol with strong guarantees and persistence. RabbitMQ seems to be a very good backend implementation but its the choice of AMQP for the main protocol is a pity. AMQP is a ridiculously over-engineered monstrosity and a saturn-sized ball on a chain for any project.

sophacles 17y ago
Look into the DDS protocol -- various implementations exist, and many of the features you want are available. I believe the open source implementation is called OpenSplice
mahmud 17y ago
Seriously. Roll your own over Redis. Redis is faster than BeanStalkd, a tiny message queue server written in C that uses libevent and a simple memcached-like protocol. Redis kicks ass, specially the beta versions.
- dlsspy 17y ago
  By what metric is redis faster than beanstalkd? Does redis has any job queue facilities for making a reliable, low-latency, safe job queue? http://www.rockstarprogrammer.org/post/2008/oct/04/what-matters-asynchronous-job-queue/ http://www.rockstarprogrammer.org/post/2008/oct/04/what-matt...
  - z8000 17y ago
    Yes, it does. See the LPOPRPUSH and BLPOP protocol commands. There's a number of threads on the Redis Google Group about using Redis as a job queue. There's also Resque by a githubber. http://github.com/defunkt/resque http://github.com/defunkt/resque
    - dlsspy 17y ago
      RPOPLPUSH does about half of what's needed and BLPOP does the other half, but neither does the whole job. It's not interesting to discuss what you can use to hack a queue together. Lots and lots of really badly thought out queue services exist today built on top of memcached or a relational database or other things that appear to work in normal cases, but can go horribly wrong and start losing data when bad things happen.
- sophacles 17y ago
  beanstalkd is not really a message queue, its a job queue. I understand that job vs message can get into semantic nonsense, but in essence a message queue can and will send any given message to n people. A job queue will only send it to the first requester. Redis does kick butt, but a k/v store is not the same as a broacast (or 1toMany, or many to one) meassage store, is not the same as a 'give me the next job' (1toAny) server either.
- moe 17y ago
  Redis is a k/v store, not a queue.
  - mahmud 17y ago
    Key == mailbox/slot, and value == message. All you need are concurrency primitives, and broadcasting (in the case of a distributed queue) People have modeled message queues with far more primitive solutions: http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.html http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.h... A Redis based solution might not be as elaborate as RabbitMQ, but it will be very clean and straightforward, not to mention blindingly fast, and will get the job done.
    - moe 17y ago
      Well, I'm quite skeptical about the "blindingly fast" part (network overhead?) but agree that in the light of the current mq-options that's indeed one of the more appealing ideas. That said, I'm not really in the mood to reinvent that particular wheel myself here, but would probably give someone else's shot a try.
  - mattrepl 17y ago
    Actually Redis is a bit more than a k/v store these days. It provides implementations of a variety of basic data structures, including lists that support queue operations. There's BLPOP[1] and BRPOP for blocking job/message requests and LPUSH and RPUSH for placing a job/message on the queue. That said, I'd still like to see benchmarks showing Redis beating Beanstalkd. Of the two, I've only used Beanstalkd, but it's a great example of keeping things simple while still providing advanced functionality. _ 1 http://code.google.com/p/redis/wiki/BlpopCommand http://code.google.com/p/redis/wiki/BlpopCommand
- z8000 17y ago
  How exactly would you do this with Redis? Redis does not push out data to N > 1 listeners/observers/subscribers. I suppose you could use either SYNC and pretend to be a replication slave, MONITOR and watch for RPUSH, or BLPOP. BLPOP hangs/blocks the client though. Sure, the Redis C codebase is fairly approachable. Perhaps you were suggesting adding new protocol commands?
- JulianMorrison 17y ago
  If Redis uses TCP it will bottleneck long before 0MQ does, because 0MQ uses a reliable-multicast protocol for communication over the LAN. As discussed in the linked article, in the comments. Also, Redis is a centralized server and a single point of failure - 0MQ doesn't force you into that topology.
yummyfajitas 17y ago
I'm curious, what do you feel is overengineered about AMQP? I've been using it for a while, and I find it to be quite simple. I send messages based on a routing key. They go to the right place, and reasonably fast (though not as fast as zeromq). What's overengineered about it?
- moe 17y ago
  Well, for starters try to find a good AMQP tutorial. There's Rabbits & Warrens (which is good but still quite rough) and that's about it. Large parts of the semantics (exchange types, interactions between the various flags) are needlessly complex. Most of the same could have been achieved with much simpler primitives (see other protocols like spread, stomp etc.). Even one of the authors in this article admits dumb decisions like making it a binary protocol and generally making it extra-ordinarily difficult to create a working protocol endpoint. What's up with this whole XML descriptor thing... In summary AMQP has been the most painful protocol I had the misfortune to work with since leaving java-land. (Admittedly partly because the python drivers are especially messy)
  - gcampbell 17y ago
    That's interesting - my experience has been the opposite (that is, AMQP has been quite easy to work with), but I wonder if it's because I primarily use the Ruby clients (http://github.com/tmm1/amqp http://github.com/tmm1/amqp for asynchronous, http://github.com/celldee/bunny http://github.com/celldee/bunny for synchronous)? I've also used the Erlang client a bit, which was somewhat trickier. Maybe this should inspire me to start a blog and write up a tutorial.
    - yummyfajitas 17y ago
      It isn't just ruby clients, I've found the python client to be similarly simple. I get the impression that there can be a lot of complexity to AMQP - but so far the c and python client libraries have hidden nearly all of it from me.

by moe 17y ago

sophacles 17y ago

Look into the DDS protocol -- various implementations exist, and many of the features you want are available. I believe the open source implementation is called OpenSplice

mahmud 17y ago

Seriously. Roll your own over Redis. Redis is faster than BeanStalkd, a tiny message queue server written in C that uses libevent and a simple memcached-like protocol. Redis kicks ass, specially the beta versions.

dlsspy 17y ago
By what metric is redis faster than beanstalkd? Does redis has any job queue facilities for making a reliable, low-latency, safe job queue? http://www.rockstarprogrammer.org/post/2008/oct/04/what-matters-asynchronous-job-queue/ http://www.rockstarprogrammer.org/post/2008/oct/04/what-matt...
- z8000 17y ago
  Yes, it does. See the LPOPRPUSH and BLPOP protocol commands. There's a number of threads on the Redis Google Group about using Redis as a job queue. There's also Resque by a githubber. http://github.com/defunkt/resque http://github.com/defunkt/resque
  - dlsspy 17y ago
    RPOPLPUSH does about half of what's needed and BLPOP does the other half, but neither does the whole job. It's not interesting to discuss what you can use to hack a queue together. Lots and lots of really badly thought out queue services exist today built on top of memcached or a relational database or other things that appear to work in normal cases, but can go horribly wrong and start losing data when bad things happen.
sophacles 17y ago
beanstalkd is not really a message queue, its a job queue. I understand that job vs message can get into semantic nonsense, but in essence a message queue can and will send any given message to n people. A job queue will only send it to the first requester. Redis does kick butt, but a k/v store is not the same as a broacast (or 1toMany, or many to one) meassage store, is not the same as a 'give me the next job' (1toAny) server either.
moe 17y ago
Redis is a k/v store, not a queue.
- mahmud 17y ago
  Key == mailbox/slot, and value == message. All you need are concurrency primitives, and broadcasting (in the case of a distributed queue) People have modeled message queues with far more primitive solutions: http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.html http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.h... A Redis based solution might not be as elaborate as RabbitMQ, but it will be very clean and straightforward, not to mention blindingly fast, and will get the job done.
  - moe 17y ago
    Well, I'm quite skeptical about the "blindingly fast" part (network overhead?) but agree that in the light of the current mq-options that's indeed one of the more appealing ideas. That said, I'm not really in the mood to reinvent that particular wheel myself here, but would probably give someone else's shot a try.
- mattrepl 17y ago
  Actually Redis is a bit more than a k/v store these days. It provides implementations of a variety of basic data structures, including lists that support queue operations. There's BLPOP[1] and BRPOP for blocking job/message requests and LPUSH and RPUSH for placing a job/message on the queue. That said, I'd still like to see benchmarks showing Redis beating Beanstalkd. Of the two, I've only used Beanstalkd, but it's a great example of keeping things simple while still providing advanced functionality. _ 1 http://code.google.com/p/redis/wiki/BlpopCommand http://code.google.com/p/redis/wiki/BlpopCommand
z8000 17y ago
How exactly would you do this with Redis? Redis does not push out data to N > 1 listeners/observers/subscribers. I suppose you could use either SYNC and pretend to be a replication slave, MONITOR and watch for RPUSH, or BLPOP. BLPOP hangs/blocks the client though. Sure, the Redis C codebase is fairly approachable. Perhaps you were suggesting adding new protocol commands?
JulianMorrison 17y ago
If Redis uses TCP it will bottleneck long before 0MQ does, because 0MQ uses a reliable-multicast protocol for communication over the LAN. As discussed in the linked article, in the comments. Also, Redis is a centralized server and a single point of failure - 0MQ doesn't force you into that topology.

yummyfajitas 17y ago

I'm curious, what do you feel is overengineered about AMQP? I've been using it for a while, and I find it to be quite simple. I send messages based on a routing key. They go to the right place, and reasonably fast (though not as fast as zeromq). What's overengineered about it?

moe 17y ago
Well, for starters try to find a good AMQP tutorial. There's Rabbits & Warrens (which is good but still quite rough) and that's about it. Large parts of the semantics (exchange types, interactions between the various flags) are needlessly complex. Most of the same could have been achieved with much simpler primitives (see other protocols like spread, stomp etc.). Even one of the authors in this article admits dumb decisions like making it a binary protocol and generally making it extra-ordinarily difficult to create a working protocol endpoint. What's up with this whole XML descriptor thing... In summary AMQP has been the most painful protocol I had the misfortune to work with since leaving java-land. (Admittedly partly because the python drivers are especially messy)
- gcampbell 17y ago
  That's interesting - my experience has been the opposite (that is, AMQP has been quite easy to work with), but I wonder if it's because I primarily use the Ruby clients (http://github.com/tmm1/amqp http://github.com/tmm1/amqp for asynchronous, http://github.com/celldee/bunny http://github.com/celldee/bunny for synchronous)? I've also used the Erlang client a bit, which was somewhat trickier. Maybe this should inspire me to start a blog and write up a tutorial.
  - yummyfajitas 17y ago
    It isn't just ruby clients, I've found the python client to be similarly simple. I get the impression that there can be a lot of complexity to AMQP - but so far the c and python client libraries have hidden nearly all of it from me.