5 ms·

Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and w

Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?

RyanZAG 13y ago
Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.
ecoffey 13y ago
In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream. EDIT: should add that morkbot had a great link too: https://news.ycombinator.com/item?id=6874607 https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...
jbooth 13y ago
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC". Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages. So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
- xorgar831 13y ago
  How does that compare to RabbitMQ's disk backed store?
  - jbooth 13y ago
    I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
    - whisk3rs 13y ago
      It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
- recuter 13y ago
  How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?
  - ddorian43 13y ago
    Since you host it yourself you can't compare it because you on the variable cost of servers and you time to maintain them.
idunno246 13y ago
It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.
- boredandroid 13y ago
  This is not correct: http://kafka.apache.org/documentation.html#design http://kafka.apache.org/documentation.html#design :-)
  - idunno246 13y ago
    Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires." Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc. It's still very good at what it does.
    - mumrah 13y ago
      Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8. What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.
mumrah 13y ago
Use cases (from the project page): http://kafka.apache.org/documentation.html#uses http://kafka.apache.org/documentation.html#uses

Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?

RyanZAG 13y ago

Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.

ecoffey 13y ago

In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream. EDIT: should add that morkbot had a great link too: https://news.ycombinator.com/item?id=6874607 https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

jbooth 13y ago

It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC". Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages. So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).

xorgar831 13y ago
How does that compare to RabbitMQ's disk backed store?
- jbooth 13y ago
  I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
  - whisk3rs 13y ago
    It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
recuter 13y ago
How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?
- ddorian43 13y ago
  Since you host it yourself you can't compare it because you on the variable cost of servers and you time to maintain them.

idunno246 13y ago

It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.

boredandroid 13y ago
This is not correct: http://kafka.apache.org/documentation.html#design http://kafka.apache.org/documentation.html#design :-)
- idunno246 13y ago
  Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires." Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc. It's still very good at what it does.
  - mumrah 13y ago
    Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8. What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.

mumrah 13y ago

Use cases (from the project page): http://kafka.apache.org/documentation.html#uses http://kafka.apache.org/documentation.html#uses