3 ms·

Apache Kafka seems like a promising alternative, although I haven't fully evaluated it yet.

lobster_johnson 12y ago
I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.
- lindvall 12y ago
  > Not a fan of Java-based software either, to be honest. It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka. It's worth checking out how Zookeeper and Kafka did in aphyr's testing: http://aphyr.com/posts/291-call-me-maybe-zookeeper http://aphyr.com/posts/291-call-me-maybe-zookeeper http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka
  - lobster_johnson 12y ago
    Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
    - krakensden 12y ago
      The worst part is "let's statically preallocate* the heap size like it's 1975". *Note: never more than 31GB
  - lobster_johnson 12y ago
    Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
- slantedview 12y ago
  > Not a fan of Java-based software either, to be honest. Yet so many of the services you use run on it.
  - lobster_johnson 12y ago
    Not sure that's an argument for anything. A lot of businesses build on crappy software.
lobster_johnson 12y ago
Kafka is a lower-level system than RabbitMQ. It doesn't support: * Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication. * Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case. * Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority. * Message TTL. And it's susceptible to network partitions, just like RabbitMQ.

I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.

lindvall 12y ago
> Not a fan of Java-based software either, to be honest. It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka. It's worth checking out how Zookeeper and Kafka did in aphyr's testing: http://aphyr.com/posts/291-call-me-maybe-zookeeper http://aphyr.com/posts/291-call-me-maybe-zookeeper http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka
- lobster_johnson 12y ago
  Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
  - krakensden 12y ago
    The worst part is "let's statically preallocate* the heap size like it's 1975". *Note: never more than 31GB
- lobster_johnson 12y ago
  Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
slantedview 12y ago
> Not a fan of Java-based software either, to be honest. Yet so many of the services you use run on it.
- lobster_johnson 12y ago
  Not sure that's an argument for anything. A lot of businesses build on crappy software.

lobster_johnson 12y ago

Kafka is a lower-level system than RabbitMQ. It doesn't support: * Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication. * Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case. * Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority. * Message TTL. And it's susceptible to network partitions, just like RabbitMQ.