5 ms·

What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.

by yayitswei 12y ago

coolsunglasses 12y ago
I wouldn't use either of those if you need durability.
- arjie 12y ago
  What would you use? And why not RabbitMQ durable queues?
  - tensor 12y ago
    Probably referring to: http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq But all systems like this have weak points, just look at the other systems aphyr has analyzed.
  - erichmond 12y ago
    Kafka may be worth a look. It's the best message delivery system I've seen in the wild.
    - hkarthik 12y ago
      Anyone using Kafka with AWS? I'm concerned that it won't be partition tolerant enough to use it there. http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.
- imperialWicket 12y ago
  This isn't helpful. It's a perfectly valid question that even provides insight into some possible candidates. The candidates might not be the best options, but they commonly serve this purpose. What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?
- tensor 12y ago
  I've looked and have not found any alternatives that are better.
  - coolsunglasses 12y ago
    Did you rule out Kafka? If so, on what basis?
lgierth 12y ago
Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability. Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105 http://vimeo.com/69960105 Edit: usually with durable queues on SSDs Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-rabbitmq-soundcloud http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...
Scuds 12y ago
and here I am, staring down the barrel of Erlang. again
- therockhead 12y ago
  What type of issues are you having with Erlang?

What are you guys using for durable queues? I'm currently evaluating Redis and RabbitMQ.

coolsunglasses 12y ago

I wouldn't use either of those if you need durability.

arjie 12y ago
What would you use? And why not RabbitMQ durable queues?
- tensor 12y ago
  Probably referring to: http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq But all systems like this have weak points, just look at the other systems aphyr has analyzed.
- erichmond 12y ago
  Kafka may be worth a look. It's the best message delivery system I've seen in the wild.
  - hkarthik 12y ago
    Anyone using Kafka with AWS? I'm concerned that it won't be partition tolerant enough to use it there. http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka We chose RabbitMQ, but we're achieving durability by storing messages in a backup database (CouchBase) that appears to handle partitions better. Still not ideal, and more complicated than we'd like.
imperialWicket 12y ago
This isn't helpful. It's a perfectly valid question that even provides insight into some possible candidates. The candidates might not be the best options, but they commonly serve this purpose. What is it about RabbitMQ or Redis durability that you find lacking? What alternative do you recommend?
tensor 12y ago
I've looked and have not found any alternatives that are better.
- coolsunglasses 12y ago
  Did you rule out Kafka? If so, on what basis?

lgierth 12y ago

Pairs of RabbitMQ brokers, used in a Publish-One-Subscribe-Many fashion for availability. Scaling RabbitMQ at SoundCloud: http://vimeo.com/69960105 http://vimeo.com/69960105 Edit: usually with durable queues on SSDs Edit: text version of the talk: http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-rabbitmq-soundcloud http://blog.pivotal.io/pivotal/case-studies-2/scaling-with-r...

Scuds 12y ago

and here I am, staring down the barrel of Erlang. again

therockhead 12y ago
What type of issues are you having with Erlang?