14 ms·

Apache Kafka 0.8.0 released

131 points by mumrah 13y ago

mumrah 13y ago
This is the first release as an Apache top-level project and represents many months of hard work. A few of the major improvements (from https://archive.apache.org/dist/kafka/0.8.0/RELEASE_NOTES.html https://archive.apache.org/dist/kafka/0.8.0/RELEASE_NOTES.ht...): * Intra-cluster replication support * Support multiple data directories * Many new internal metrics * Time based log segment rollout Plus many bug fixes and other improvements.
hans0l074 13y ago
May I ask how Kafka compares with an AMQP solution such as RabbitMQ? Thank you.
- ddorian43 13y ago
  It keeps all messages in a log(sorted) and each client has a position of where they are in the log. So they can restart some parts, or skip some. This makes it very fast. And it has sharding, which no-other messagequeue has (i think).
  - old_sound 13y ago
    RabbitMQ supports sharding via exchanges
- mumrah 13y ago
  Quora discussion: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo... RabbitMQ developer on the kafka-users list: http://mail-archives.apache.org/mod_mbox/kafka-users/201306.mbox/%3CCACOPnvdd0_Xr3oOy8Dc8pu0z-_u813xnCOb80ZWpjVeUudhK9A@mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/kafka-users/201306.... SO discussion on several queuing systems: http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or http://stackoverflow.com/questions/731233/activemq-or-rabbit...
  - morkbot 13y ago
    Quora link where you don't have to register/log-in: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...
nine_k 13y ago
I like the naming trend and expect releases of Apache Ionesco, Apache Lovecraft, and Apache Poe real soon now. </obligatory-joke>
nullymcnull 13y ago
Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?
- RyanZAG 13y ago
  Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.
- ecoffey 13y ago
  In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream. EDIT: should add that morkbot had a great link too: https://news.ycombinator.com/item?id=6874607 https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...
- jbooth 13y ago
  It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC". Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages. So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
xal 13y ago
Kafka is an integral part of Shopify's infrastructure. It's brilliant but under appreciated technology. A full company, scalable event bus like this can totally revolutionize the way you build services.
- nemothekid 13y ago
  And thank you for the amazing go client.
- capkutay 13y ago
  "A full company, scalable event bus like this can totally revolutionize the way you build services." Shameless (and shameful) plug, but if anyone wants to be part of such an enterprise that's already gained traction with big companies, send me a message!
pspeter3 13y ago
Congratulations to the Kafka team! Their work is always extremely impressive.
x3942 13y ago
Finally!
harichinnan 13y ago
Was this developed in Scala?
- boredandroid 13y ago
  Yup.
- mumrah 13y ago
  It is a mixture of Java and Scala, though mostly Scala. GitHub mirror of the source: https://github.com/apache/kafka https://github.com/apache/kafka
shmerl 13y ago
Why is it written in Java? Isn't it a default performance hit? I'd expect such frameworks to be written in high performance languages.
- nine_k 13y ago
  Java is pretty high performance, usually on par with C++, and usually faster than e.g. Go. Writing Java code so that there are no perceivable GC pauses is an art, but it is not impossible to achieve. JVM might require more RAM upfront, but a well-written program is usually reasonably memory-efficient, too, so the consumed memory grows reasonably slowly with the problem size. Writing things in pure C is often just too time-consuming.
  - shmerl 13y ago
    > Java is pretty high performance, usually on par with C++ I'm not convinced. Java I/O is far form perfect, and Kafka is probably very heavy on I/O side. > and usually faster than e.g. Go. That's strange, since Go to some degree was intended as replacement for Java without having Java's downsides. Why would Go be less performant? I'd be interested if someone would write such framework in Rust though. C++ is of course a default expectation, but usage of Java somehow surprises me in this case.
    - kelseyfrancis 13y ago
      > Java I/O is far form perfect The API certainly isn't perfect, but what do you find lacking about the performance of Java I/O? http://docs.oracle.com/javase/7/docs/api/java/nio/channels/package-summary.html http://docs.oracle.com/javase/7/docs/api/java/nio/channels/p...
    - tobz 13y ago
      Why are you so deadset on assuming that Java is slow? That's an incredibly misinformed and narrow view of what's possible on the JVM.
      - 13y ago
turingbook 13y ago
I wrote about this release and Kafka in general in Chinese and gather some information perhaps useful for Chinese guys: http://geek.csdn.net/news/detail/3866 http://geek.csdn.net/news/detail/3866
rmk2 13y ago
This has nothing to do with the software itself, but it bothers me: Why do you call a "high-throughput distributed messaging system" Kafka? Kafka's stories essentially describe the polar-opposite: crippled, ineffective, labyrinthine message systems that are exceedingly hierarchical in nature. They are also rather user-unfriendly, i.e. their users usually die horrible, lonely deaths. Am I just missing the in-joke here, and it's called Kafka because it is exactly the opposite of this, or did somebody overdo the hipster naming scheme?
- mst 13y ago
  It's written in Scala, so perhaps the reference is to the source code rather than the client interface ... ... or maybe you're just overthinking it.
- cleaver 13y ago
  That was my first thought. Based on the novels I've read, I would interpret it in a couple of possible ways: - You receive a message, but the system can't tell you why your received it nor what you should do. (The Trial) - It's not a distributed messaging system with bugs. Actually, you are the bug. (Metamorphosis) As an aside, I went to a tech conference in Prague two months ago and visited Café Slavia, a hangout not just of Kafka, but also author Milan Kundera and president/poet Václav Havel. I had a glass of absinthe in their honour.
- boredandroid 13y ago
  The rationale was that it should be a writer because we were building a distributed log or journal (a service dedicated to writing). The writer needed to be someone that (1) I liked, (2) sounded cool as a name, (3) was dead (because it would be creepy to use the name of someone who was alive).
- mumrah 13y ago
  At LinkedIn, they started code naming projects after authors and literary works. E.g., Kafka, Camus, and a bunch of Harry Potter references. Kafka was donated to Apache by LinkedIn and the name stuck.

131 points by mumrah 13y ago

This is the first release as an Apache top-level project and represents many months of hard work. A few of the major improvements (from https://archive.apache.org/dist/kafka/0.8.0/RELEASE_NOTES.html https://archive.apache.org/dist/kafka/0.8.0/RELEASE_NOTES.ht...): * Intra-cluster replication support * Support multiple data directories * Many new internal metrics * Time based log segment rollout Plus many bug fixes and other improvements.

hans0l074 13y ago

May I ask how Kafka compares with an AMQP solution such as RabbitMQ? Thank you.

ddorian43 13y ago
It keeps all messages in a log(sorted) and each client has a position of where they are in the log. So they can restart some parts, or skip some. This makes it very fast. And it has sharding, which no-other messagequeue has (i think).
- old_sound 13y ago
  RabbitMQ supports sharding via exchanges
mumrah 13y ago
Quora discussion: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo... RabbitMQ developer on the kafka-users list: http://mail-archives.apache.org/mod_mbox/kafka-users/201306.mbox/%3CCACOPnvdd0_Xr3oOy8Dc8pu0z-_u813xnCOb80ZWpjVeUudhK9A@mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/kafka-users/201306.... SO discussion on several queuing systems: http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or http://stackoverflow.com/questions/731233/activemq-or-rabbit...
- morkbot 13y ago
  Quora link where you don't have to register/log-in: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

nine_k 13y ago

I like the naming trend and expect releases of Apache Ionesco, Apache Lovecraft, and Apache Poe real soon now. </obligatory-joke>

nullymcnull 13y ago

Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?

RyanZAG 13y ago
Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.
ecoffey 13y ago
In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream. EDIT: should add that morkbot had a great link too: https://news.ycombinator.com/item?id=6874607 https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features?share=1 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...
jbooth 13y ago
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC". Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages. So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).

xal 13y ago

Kafka is an integral part of Shopify's infrastructure. It's brilliant but under appreciated technology. A full company, scalable event bus like this can totally revolutionize the way you build services.

nemothekid 13y ago
And thank you for the amazing go client.
capkutay 13y ago
"A full company, scalable event bus like this can totally revolutionize the way you build services." Shameless (and shameful) plug, but if anyone wants to be part of such an enterprise that's already gained traction with big companies, send me a message!

pspeter3 13y ago

Congratulations to the Kafka team! Their work is always extremely impressive.

x3942 13y ago

Finally!

harichinnan 13y ago

Was this developed in Scala?

boredandroid 13y ago
Yup.
mumrah 13y ago
It is a mixture of Java and Scala, though mostly Scala. GitHub mirror of the source: https://github.com/apache/kafka https://github.com/apache/kafka

shmerl 13y ago

Why is it written in Java? Isn't it a default performance hit? I'd expect such frameworks to be written in high performance languages.

nine_k 13y ago
Java is pretty high performance, usually on par with C++, and usually faster than e.g. Go. Writing Java code so that there are no perceivable GC pauses is an art, but it is not impossible to achieve. JVM might require more RAM upfront, but a well-written program is usually reasonably memory-efficient, too, so the consumed memory grows reasonably slowly with the problem size. Writing things in pure C is often just too time-consuming.
- shmerl 13y ago
  > Java is pretty high performance, usually on par with C++ I'm not convinced. Java I/O is far form perfect, and Kafka is probably very heavy on I/O side. > and usually faster than e.g. Go. That's strange, since Go to some degree was intended as replacement for Java without having Java's downsides. Why would Go be less performant? I'd be interested if someone would write such framework in Rust though. C++ is of course a default expectation, but usage of Java somehow surprises me in this case.
  - kelseyfrancis 13y ago
    > Java I/O is far form perfect The API certainly isn't perfect, but what do you find lacking about the performance of Java I/O? http://docs.oracle.com/javase/7/docs/api/java/nio/channels/package-summary.html http://docs.oracle.com/javase/7/docs/api/java/nio/channels/p...
  - tobz 13y ago
    Why are you so deadset on assuming that Java is slow? That's an incredibly misinformed and narrow view of what's possible on the JVM.
    - 13y ago

turingbook 13y ago

I wrote about this release and Kafka in general in Chinese and gather some information perhaps useful for Chinese guys: http://geek.csdn.net/news/detail/3866 http://geek.csdn.net/news/detail/3866

rmk2 13y ago

This has nothing to do with the software itself, but it bothers me: Why do you call a "high-throughput distributed messaging system" Kafka? Kafka's stories essentially describe the polar-opposite: crippled, ineffective, labyrinthine message systems that are exceedingly hierarchical in nature. They are also rather user-unfriendly, i.e. their users usually die horrible, lonely deaths. Am I just missing the in-joke here, and it's called Kafka because it is exactly the opposite of this, or did somebody overdo the hipster naming scheme?

mst 13y ago
It's written in Scala, so perhaps the reference is to the source code rather than the client interface ... ... or maybe you're just overthinking it.
cleaver 13y ago
That was my first thought. Based on the novels I've read, I would interpret it in a couple of possible ways: - You receive a message, but the system can't tell you why your received it nor what you should do. (The Trial) - It's not a distributed messaging system with bugs. Actually, you are the bug. (Metamorphosis) As an aside, I went to a tech conference in Prague two months ago and visited Café Slavia, a hangout not just of Kafka, but also author Milan Kundera and president/poet Václav Havel. I had a glass of absinthe in their honour.
boredandroid 13y ago
The rationale was that it should be a writer because we were building a distributed log or journal (a service dedicated to writing). The writer needed to be someone that (1) I liked, (2) sounded cool as a name, (3) was dead (because it would be creepy to use the name of someone who was alive).
mumrah 13y ago
At LinkedIn, they started code naming projects after authors and literary works. E.g., Kafka, Camus, and a bunch of Harry Potter references. Kafka was donated to Apache by LinkedIn and the name stuck.