16 ms·

Kafka without ZooKeeper

393 points by toomanybits 6y ago

haolez 6y ago
Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D
- cmason 6y ago
  What do you use instead?
  - NicoJuicy 6y ago
    Nats
    - unixhero 6y ago
      What kind if projects so you work om? I am genuinely curious about use cases. Feel free to obfuscate.
      - NicoJuicy 6y ago
        E-commerce In the process of splitting up everything in modules. ( Microservices would be Overkill)
  - haolez 6y ago
    Cheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).
    - gwenshap 6y ago
      Confluent Cloud Basic/Standard is a cheap managed Kafka. If the objection is to the deployment and not Kafka clients.
      - KptMarchewa 6y ago
        We might have different definitions of cheap.
    - jganetsk 6y ago
      Pub/Sub Lite is a cheap managed cloud service on GCP: https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs With a Kafka compatibility shim: https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work for GCP.
  - menzella-g 6y ago
    GCP offers Pub/Sub Lite, an inexpensive messaging product with Kafka-compatible client libraries. https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work on this product.
- dmlittle 6y ago
  Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/ https://vectorized.io/
  - deleted 6y ago
    [deleted]
  - timdorr 6y ago
    https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ is a more useful link, since the main domain appears to have some JS errors right now.
    - agallego 6y ago
      oh odd. what setup to repro the js errors. i'll fix.
- math 6y ago
  You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.
  - unixhero 6y ago
    Thank you for disclosing and not disclaiming.
  - alex_anglin 6y ago
    Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
    - d_t_w 6y ago
      Confluent Cloud has some nice point-and-click UI for creating associated Kafka resources like Schema Registries and Connect Clusters. My preference is MSK but I'm very comfortable with vanilla Kafka in AWS at a good price with auto-updates.
      - tamale 6y ago
        One nice thing about confluent cloud vs MSK is the minimum cost of a confluent cloud cluster is far, far cheaper than the minimal cost of an MSK cluster
    - dividedbyzero 6y ago
      Is there a GCP offering that isn't just Confluent Cloud billed via Google?
      - lornajane 6y ago
        You can get managed Kafka on Aiven (disclaimer: I work there) on GCP, either through the marketplace or directly through Aiven.
      - 6y ago
- toomanybits 6y ago
  I think that's one of the main points. Now you can run it as a single process more like a traditional broker (although it's obviously still a log).
- keithnz 6y ago
  yeah, really needs a use case that justifies it, I have a particular IoT backend where I made it pluggable between kafka and rabbitmq, ended up just using rabbitmq as it is simpler to work with / manage, and still not really pushing it in terms of performance with thousands of devices.
- colin_mccabe 6y ago
  Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to manage it yourself, it will be easier running one service than two. Disclosure: I work for Confluent.
  - pdimitar 6y ago
    Your clarification made me wonder: Is the single process deployment only doable via a container? Or will we actually have OS native process as well?
    - colin_mccabe 6y ago
      Yes, you can run a single OS native process in KRaft mode, without using Docker. Docker just avoids the need to install a JVM, but it is not required.
  - jwandborg 6y ago
    A nit regarding the disclosure: I prefer it at the top of the message, and I think that's "best practice", but I don't know for sure.
  - sumtechguy 6y ago
    Oh it most certainly simplifies things. I am looking at half the number of boxes needed to run. Which is not insignificant in my cost structure. What is the migration strategy here? Is it doc'd up yet? I am having flashbacks to migration for follower partitions recently which required a decent amount of pre planning of partition layout. Also as it is pulling in the duties of ZK into kafka what sort of CPU/memory changes are you seeing? Is it 'meh' or all the way to 'you may want to add a couple of CPUs and a few more GB'? Also is it working ok with the stretched cluster? Also if you want to hit an interesting market you may want to look at 'does it run OK on a raspberry PI'.
- mrweasel 6y ago
  For people who just need a queue, Kafka is a bit like using Kubernetes to run a single Docker container. We run a number of Kafka clusters, most are relatively low trafic, and the management overhead is pretty. Earlier version did require a bit more attention, but mostly it’s pretty simple to deal with.
- cfontes 6y ago
  This is huge news. Kafka is awesome, but using it in local envs is a pain in the ass, if this is never becomes PROD ready it is already an immense achievement to be able to run Kafka locally with less complexity and overhead.
airhead969 6y ago
IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.
- cmckn 6y ago
  I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).
  - godisdad 6y ago
    Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind
    - cmckn 6y ago
      Hadoop launched in 2006, the same year as AWS' cloud portfolio. HBase showed up in 2008. Many of the hiccups with running Hadoop and friends in containers or on cloud VM's boils down to how hostnames are resolved and advertised; not any significant design issue.
      - pc86 6y ago
        > Hadoop launched in 2006, the same year as AWS' cloud portfolio. Which makes it all the less reasonable to assume it would be in any way cloud native, when "the cloud" was at best a nascent idea at that point. And how many years did it take AWS to get any serious traction after launch?
        deleted 6y ago
        [deleted]
        cmckn 6y ago
        I haven't assumed anything of the sort, I said I loved the software and wished it was easier to use in what is considered a "modern" environment.
autophagian 6y ago
One immediate downside to this: it is now likely that fewer and fewer people will be exposed to the absolutely excellent Zookeeper logo.
- spicybright 6y ago
  I did a google image search and wish I didn't.
- jaytaylor 6y ago
  See: https://en.wikipedia.org/wiki/Apache_ZooKeeper https://en.wikipedia.org/wiki/Apache_ZooKeeper https://upload.wikimedia.org/wikipedia/en/thumb/8/81/Apache_ZooKeeper_Logo.svg/605px-Apache_ZooKeeper_Logo.svg.png https://upload.wikimedia.org/wikipedia/en/thumb/8/81/Apache_...
  - AdmiralAsshat 6y ago
    Those are some impressively sized hands.
    - temuze 6y ago
      This is the ideal open source logo. You may not like it, but this is what peak performance looks like.
    - ignoramous 6y ago
      https://m.huffpost.com/us/entry/us_55c36ebce4b0923c12bbb2cd https://m.huffpost.com/us/entry/us_55c36ebce4b0923c12bbb2cd
    - koolba 6y ago
      Pretty sure he has gout too.
- timdorr 6y ago
  Is there some backstory on this? It looks like it's modelled after a real person. Perhaps one of its creators?
cmckn 6y ago
Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20 https://docs.nats.io/whats_new_20
- deleted 6y ago
  [deleted]
- vorpalhex 6y ago
  NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.
  - cmckn 6y ago
    Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) NATS also enables some unique patterns like request/reply via a "reply to" message header. Anyway, it's been a joy to use!
    - KptMarchewa 6y ago
      Sounds more like rabbitmq replacement than kafka
    - tamale 6y ago
      you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!
      - jimmyed 6y ago
deveffort 6y ago
Sweet!! Thank you.
CatDogIsGod 6y ago
Awesome!
doliveira 6y ago
Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.
- dbt00 6y ago
  I wasn't there when they made the call, but "we know it works" seems like it was the key element here.
- anonymousDan 6y ago
  Because historically implementing something like Zookeeper yourself from scratch is notoriously difficult?
  - doliveira 6y ago
    I guess what I wonder is why they didn't go with an embedded library or something of sorts. Some NoSQL databases handle it without Zookeeper.
    - nemothekid 6y ago
      >Some NoSQL databases handle it without Zookeeper. Most NoSQL databases, now, use Raft, which didn't exist at the time when Kafka was created. Other NoSQL databases, at the time, were not as stable as Zookeeper or had silent bugs that ate data (see aphyr's Jepsen series[1], which thourghly tested several NoSQL databases and found many to be failing, except for Zookeeper). [1] https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen
      - tammerk 6y ago
        https://github.com/jepsen-io/jepsen/issues/399 https://github.com/jepsen-io/jepsen/issues/399 > Yeah! I mean, I find a lot of linearizability errors in various databases, but this was also my very first time doing this kind of test, and it varies from system to system. Could have easily slipped through the cracks. In summary, aphyr thought Zookeeper is linearizable even though it doesn't provide linearizable ops. Looks like Zookeeper needs to be tested again.
taywrobel 6y ago
If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.
- skyde 6y ago
  I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....
  - ovis 6y ago
    1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?
    - rad_gruchalski 6y ago
      Re 2, have you ever tried using Kafka with a non-JVM client?
      - EdwardDiego 6y ago
        Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
        rad_gruchalski 6y ago
        I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.
whycombin8 6y ago
one step closer to being able to run only a single broker at the edge.
- skyde 6y ago
  what is the point of using Kafka if you are using only one broker (no replication)? Might as well just write your data to /dev/null
knodi 6y ago
Please separate storage from brokers next
- EdwardDiego 6y ago
  Why though? It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka. https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story.html https://blog.twitter.com/engineering/en_us/topics/insights/2... > One catch to this is that for extremely bandwidth-heavy workloads (very high fanout-reads), EventBus theoretically might be more efficient since we can scale out the serving layer independently. However, we’ve found in practice that our fanout is not extreme enough to merit separating the serving layer, especially given the bandwidth available on modern hardware.
  - nick0garvey 6y ago
    Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot. I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.
    - EdwardDiego 6y ago
      Okay, I can see that point, but is it worth the additional latency between broker and Bookie? > I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases. Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.
      - rad_gruchalski 6y ago
        > Okay, I can see that point, but is it worth the additional latency between broker and Bookie? It might depend on what you're ingesting and how much. Being able to independently scale ingest and storage is a good alternative to have. It's not only ingest though. It's also consumption. As it stands, having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory. In that sense, reprocessing historical data is pretty difficult. There's a lot of complexity hidden in additional Druid, HDFS installations or shoe-horned object storage with their own indexing to support access to historical data semi-fast.
didip 6y ago
Reminded me of this project: https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.
- bruth 6y ago
  The author now works at Confluent.
- travisjeffery 6y ago
  Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-go/ https://pragprog.com/titles/tjgo/distributed-services-with-g...
  - chokolad 6y ago
    Heh, I just bought it based on PragProg newsletter :). Literally 5 minutes ago.
    - travisjeffery 6y ago
      Awesome! Hope you enjoy reading the book and thanks for buying it.
  - kif 6y ago
    Sounds just like the kind of book I wanna read. Unfortunately I wasn't able to pay for it due to "merchant configuration". I imagine there are no region restrictions, right? I've already contacted support though.
    - travisjeffery 6y ago
      I'm not sure, I haven't heard of that happening before. I've asked my editor if she knows what's up. You can email me at tj at my HN username dot com if you want to follow up.
dikei 6y ago
I don't understand the hate for Zookeeper. Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.
- z9e 6y ago
  Agreed. I’ve been running Kafka clusters at a few companies now, one that was at a massive Fortune 500 and Zookeeper was the least of my problems. I’m still glad to see it go away, one less operational dependency the better.
- elric 6y ago
  Sure, a basic ensemble isn't that hard. But securing it is a world of pain and misery. The ZK development API is also pretty much awful. Apache Curator (which wraps around the ZK API and implements a bunch of common "recipes") makes it less painful, but it really ought to be part of ZK proper.
  - accuNum 6y ago
    God forbid you hit a bug in ZK too, as there are 5 people in the world who properly understand that system - there is nobody to help.
gigatexal 6y ago
All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/ https://vectorized.io/)
- sgt 6y ago
  What's the downside?
  - gigatexal 6y ago
    Doesn’t seem to be one. They took the Kafka API so it’s a drop in replacement and redid the implementation from the ground up. Your mileage may vary though.
    - sgt 6y ago
      I am just worries that things like 3rd parties integrating with your Kafka broker will stop working or experience issues. Also, I wonder if SASL_SSL is implemented exactly the same way etc.
      - agallego 6y ago
        hi @sgt - yup SASL_SSL due to the client protos have to be compatible, including the hack you are thinking of. from a user perspective, existing SASL + SSL + SCRAM will be released next wednesday - so no code changes.
  - mdaniel 6y ago
    The BSL license makes choosing it "a discussion" versus using OSI licensed software: https://github.com/vectorizedio/redpanda/tree/v21.3.7/licenses#faq https://github.com/vectorizedio/redpanda/tree/v21.3.7/licens... I don't have enough experience with these new vanity licenses to know what the contribution story looks like, either
    - gigatexal 6y ago
tofflos 6y ago
Is there a compiled package for 2.8.0-rc0 available? I only see sources on Github https://github.com/apache/kafka/releases/tag/2.8.0-rc0 https://github.com/apache/kafka/releases/tag/2.8.0-rc0 and couldn't find anything regarding 2.8.0-rc0 on the Kafka downloads page https://kafka.apache.org/downloads https://kafka.apache.org/downloads.
- ewencp 6y ago
  The downloads page is only for official releases, RCs are not distributed the same way and are not permanent unless promoted to an official release. See the mailing list thread for 2.8.0-RC0 for where to find the bits if you want to test https://lists.apache.org/thread.html/r16894a11aec73abac521ff93a8b154b3ebabadc85c0b7a932648be2e%40%3Cdev.kafka.apache.org%3E https://lists.apache.org/thread.html/r16894a11aec73abac521ff... and the project site has some "contact" info for mailing lists where these things are announced and advertised (including releases, Kafka Improvement Proposals, and more) https://kafka.apache.org/contact https://kafka.apache.org/contact
surfsvammel 6y ago
Great news! Props! I have a use case where this would make sense, I’ll dig in directly!
dig1 6y ago
Don't forget that ZooKeeper was the only service that passed all Jepsen [1] tests out of the box. [1] https://jepsen.io/analyses https://jepsen.io/analyses
silasiba 6y ago
Cool feature! I have a concern, if the Kafka broker provide both coordination service and Kafka service, how to achieve the resource isolation? If some of the topic which on the coordination service with very high throughput, this must cause instability of the coordination service, could this further cause the instability of the entire cluster? If some of the broker only provide the coordination service, what is this essential difference? Will this cause more problems for expansion and contraction? Will this bring greater risks when users scale down the brokers? I am very afraid that the coordination service will be shut down due to careless operation.
- mumrah 6y ago
  Part of the new design is that brokers and controllers can be run on separate JVMs.
  - silasiba 6y ago
    Yes, that my second concern. If run on separate JVM, what is the difference between use ZK and use controller? Of course, If the new controller is more stable than ZK, more efficient than ZK, this will bring benefits to users. But this is essentially replacing zk with a better product, I think etcd also can achieve this purpose.
rad_gruchalski 6y ago
Works very well. Had some time to try out the 2.8 build and all existing tooling works just fine. Solid work by everyone involved in this.

393 points by toomanybits 6y ago

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

cmason 6y ago
What do you use instead?
- NicoJuicy 6y ago
  Nats
  - unixhero 6y ago
    What kind if projects so you work om? I am genuinely curious about use cases. Feel free to obfuscate.
    - NicoJuicy 6y ago
      E-commerce In the process of splitting up everything in modules. ( Microservices would be Overkill)
- haolez 6y ago
  Cheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).
  - gwenshap 6y ago
    Confluent Cloud Basic/Standard is a cheap managed Kafka. If the objection is to the deployment and not Kafka clients.
    - KptMarchewa 6y ago
      We might have different definitions of cheap.
  - jganetsk 6y ago
    Pub/Sub Lite is a cheap managed cloud service on GCP: https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs With a Kafka compatibility shim: https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work for GCP.
- menzella-g 6y ago
  GCP offers Pub/Sub Lite, an inexpensive messaging product with Kafka-compatible client libraries. https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work on this product.
dmlittle 6y ago
Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/ https://vectorized.io/
- deleted 6y ago
  [deleted]
- timdorr 6y ago
  https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ is a more useful link, since the main domain appears to have some JS errors right now.
  - agallego 6y ago
    oh odd. what setup to repro the js errors. i'll fix.
math 6y ago
You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.
- unixhero 6y ago
  Thank you for disclosing and not disclaiming.
- alex_anglin 6y ago
  Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
  - d_t_w 6y ago
    Confluent Cloud has some nice point-and-click UI for creating associated Kafka resources like Schema Registries and Connect Clusters. My preference is MSK but I'm very comfortable with vanilla Kafka in AWS at a good price with auto-updates.
    - tamale 6y ago
      One nice thing about confluent cloud vs MSK is the minimum cost of a confluent cloud cluster is far, far cheaper than the minimal cost of an MSK cluster
  - dividedbyzero 6y ago
    Is there a GCP offering that isn't just Confluent Cloud billed via Google?
    - lornajane 6y ago
      You can get managed Kafka on Aiven (disclaimer: I work there) on GCP, either through the marketplace or directly through Aiven.
    - 6y ago
toomanybits 6y ago
I think that's one of the main points. Now you can run it as a single process more like a traditional broker (although it's obviously still a log).
keithnz 6y ago
yeah, really needs a use case that justifies it, I have a particular IoT backend where I made it pluggable between kafka and rabbitmq, ended up just using rabbitmq as it is simpler to work with / manage, and still not really pushing it in terms of performance with thousands of devices.
colin_mccabe 6y ago
Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to manage it yourself, it will be easier running one service than two. Disclosure: I work for Confluent.
- pdimitar 6y ago
  Your clarification made me wonder: Is the single process deployment only doable via a container? Or will we actually have OS native process as well?
  - colin_mccabe 6y ago
    Yes, you can run a single OS native process in KRaft mode, without using Docker. Docker just avoids the need to install a JVM, but it is not required.
- jwandborg 6y ago
  A nit regarding the disclosure: I prefer it at the top of the message, and I think that's "best practice", but I don't know for sure.
- sumtechguy 6y ago
  Oh it most certainly simplifies things. I am looking at half the number of boxes needed to run. Which is not insignificant in my cost structure. What is the migration strategy here? Is it doc'd up yet? I am having flashbacks to migration for follower partitions recently which required a decent amount of pre planning of partition layout. Also as it is pulling in the duties of ZK into kafka what sort of CPU/memory changes are you seeing? Is it 'meh' or all the way to 'you may want to add a couple of CPUs and a few more GB'? Also is it working ok with the stretched cluster? Also if you want to hit an interesting market you may want to look at 'does it run OK on a raspberry PI'.
mrweasel 6y ago
For people who just need a queue, Kafka is a bit like using Kubernetes to run a single Docker container. We run a number of Kafka clusters, most are relatively low trafic, and the management overhead is pretty. Earlier version did require a bit more attention, but mostly it’s pretty simple to deal with.
cfontes 6y ago
This is huge news. Kafka is awesome, but using it in local envs is a pain in the ass, if this is never becomes PROD ready it is already an immense achievement to be able to run Kafka locally with less complexity and overhead.

airhead969 6y ago

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

cmckn 6y ago
I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).
- godisdad 6y ago
  Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind
  - cmckn 6y ago
    Hadoop launched in 2006, the same year as AWS' cloud portfolio. HBase showed up in 2008. Many of the hiccups with running Hadoop and friends in containers or on cloud VM's boils down to how hostnames are resolved and advertised; not any significant design issue.
    - pc86 6y ago
      > Hadoop launched in 2006, the same year as AWS' cloud portfolio. Which makes it all the less reasonable to assume it would be in any way cloud native, when "the cloud" was at best a nascent idea at that point. And how many years did it take AWS to get any serious traction after launch?
      - deleted 6y ago
        [deleted]
      - cmckn 6y ago
        I haven't assumed anything of the sort, I said I loved the software and wished it was easier to use in what is considered a "modern" environment.

autophagian 6y ago

One immediate downside to this: it is now likely that fewer and fewer people will be exposed to the absolutely excellent Zookeeper logo.

spicybright 6y ago
I did a google image search and wish I didn't.
jaytaylor 6y ago
See: https://en.wikipedia.org/wiki/Apache_ZooKeeper https://en.wikipedia.org/wiki/Apache_ZooKeeper https://upload.wikimedia.org/wikipedia/en/thumb/8/81/Apache_ZooKeeper_Logo.svg/605px-Apache_ZooKeeper_Logo.svg.png https://upload.wikimedia.org/wikipedia/en/thumb/8/81/Apache_...
- AdmiralAsshat 6y ago
  Those are some impressively sized hands.
  - temuze 6y ago
    This is the ideal open source logo. You may not like it, but this is what peak performance looks like.
  - ignoramous 6y ago
    https://m.huffpost.com/us/entry/us_55c36ebce4b0923c12bbb2cd https://m.huffpost.com/us/entry/us_55c36ebce4b0923c12bbb2cd
  - koolba 6y ago
    Pretty sure he has gout too.
timdorr 6y ago
Is there some backstory on this? It looks like it's modelled after a real person. Perhaps one of its creators?

cmckn 6y ago

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20 https://docs.nats.io/whats_new_20

deleted 6y ago
[deleted]
vorpalhex 6y ago
NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.
- cmckn 6y ago
  Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) NATS also enables some unique patterns like request/reply via a "reply to" message header. Anyway, it's been a joy to use!
  - KptMarchewa 6y ago
    Sounds more like rabbitmq replacement than kafka
  - tamale 6y ago
    you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!
    - jimmyed 6y ago

deveffort 6y ago

Sweet!! Thank you.

CatDogIsGod 6y ago

Awesome!

doliveira 6y ago

Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.

dbt00 6y ago
I wasn't there when they made the call, but "we know it works" seems like it was the key element here.
anonymousDan 6y ago
Because historically implementing something like Zookeeper yourself from scratch is notoriously difficult?
- doliveira 6y ago
  I guess what I wonder is why they didn't go with an embedded library or something of sorts. Some NoSQL databases handle it without Zookeeper.
  - nemothekid 6y ago
    >Some NoSQL databases handle it without Zookeeper. Most NoSQL databases, now, use Raft, which didn't exist at the time when Kafka was created. Other NoSQL databases, at the time, were not as stable as Zookeeper or had silent bugs that ate data (see aphyr's Jepsen series[1], which thourghly tested several NoSQL databases and found many to be failing, except for Zookeeper). [1] https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen
    - tammerk 6y ago
      https://github.com/jepsen-io/jepsen/issues/399 https://github.com/jepsen-io/jepsen/issues/399 > Yeah! I mean, I find a lot of linearizability errors in various databases, but this was also my very first time doing this kind of test, and it varies from system to system. Could have easily slipped through the cracks. In summary, aphyr thought Zookeeper is linearizable even though it doesn't provide linearizable ops. Looks like Zookeeper needs to be tested again.

taywrobel 6y ago

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

skyde 6y ago
I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....
- ovis 6y ago
  1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?
  - rad_gruchalski 6y ago
    Re 2, have you ever tried using Kafka with a non-JVM client?
    - EdwardDiego 6y ago
      Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
      - rad_gruchalski 6y ago
        I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.

whycombin8 6y ago

one step closer to being able to run only a single broker at the edge.

skyde 6y ago
what is the point of using Kafka if you are using only one broker (no replication)? Might as well just write your data to /dev/null

knodi 6y ago

Please separate storage from brokers next

EdwardDiego 6y ago
Why though? It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka. https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story.html https://blog.twitter.com/engineering/en_us/topics/insights/2... > One catch to this is that for extremely bandwidth-heavy workloads (very high fanout-reads), EventBus theoretically might be more efficient since we can scale out the serving layer independently. However, we’ve found in practice that our fanout is not extreme enough to merit separating the serving layer, especially given the bandwidth available on modern hardware.
- nick0garvey 6y ago
  Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot. I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.
  - EdwardDiego 6y ago
    Okay, I can see that point, but is it worth the additional latency between broker and Bookie? > I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases. Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.
    - rad_gruchalski 6y ago
      > Okay, I can see that point, but is it worth the additional latency between broker and Bookie? It might depend on what you're ingesting and how much. Being able to independently scale ingest and storage is a good alternative to have. It's not only ingest though. It's also consumption. As it stands, having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory. In that sense, reprocessing historical data is pretty difficult. There's a lot of complexity hidden in additional Druid, HDFS installations or shoe-horned object storage with their own indexing to support access to historical data semi-fast.

didip 6y ago

Reminded me of this project: https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.

bruth 6y ago
The author now works at Confluent.
travisjeffery 6y ago
Hey, I'm the author of Jocko. I've been working at Confluent the past four years. I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-go/ https://pragprog.com/titles/tjgo/distributed-services-with-g...
- chokolad 6y ago
  Heh, I just bought it based on PragProg newsletter :). Literally 5 minutes ago.
  - travisjeffery 6y ago
    Awesome! Hope you enjoy reading the book and thanks for buying it.
- kif 6y ago
  Sounds just like the kind of book I wanna read. Unfortunately I wasn't able to pay for it due to "merchant configuration". I imagine there are no region restrictions, right? I've already contacted support though.
  - travisjeffery 6y ago
    I'm not sure, I haven't heard of that happening before. I've asked my editor if she knows what's up. You can email me at tj at my HN username dot com if you want to follow up.

dikei 6y ago

I don't understand the hate for Zookeeper. Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.

z9e 6y ago
Agreed. I’ve been running Kafka clusters at a few companies now, one that was at a massive Fortune 500 and Zookeeper was the least of my problems. I’m still glad to see it go away, one less operational dependency the better.
elric 6y ago
Sure, a basic ensemble isn't that hard. But securing it is a world of pain and misery. The ZK development API is also pretty much awful. Apache Curator (which wraps around the ZK API and implements a bunch of common "recipes") makes it less painful, but it really ought to be part of ZK proper.
- accuNum 6y ago
  God forbid you hit a bug in ZK too, as there are 5 people in the world who properly understand that system - there is nobody to help.

gigatexal 6y ago

All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/ https://vectorized.io/)

sgt 6y ago
What's the downside?
- gigatexal 6y ago
  Doesn’t seem to be one. They took the Kafka API so it’s a drop in replacement and redid the implementation from the ground up. Your mileage may vary though.
  - sgt 6y ago
    I am just worries that things like 3rd parties integrating with your Kafka broker will stop working or experience issues. Also, I wonder if SASL_SSL is implemented exactly the same way etc.
    - agallego 6y ago
      hi @sgt - yup SASL_SSL due to the client protos have to be compatible, including the hack you are thinking of. from a user perspective, existing SASL + SSL + SCRAM will be released next wednesday - so no code changes.
- mdaniel 6y ago
  The BSL license makes choosing it "a discussion" versus using OSI licensed software: https://github.com/vectorizedio/redpanda/tree/v21.3.7/licenses#faq https://github.com/vectorizedio/redpanda/tree/v21.3.7/licens... I don't have enough experience with these new vanity licenses to know what the contribution story looks like, either
  - gigatexal 6y ago

tofflos 6y ago

Is there a compiled package for 2.8.0-rc0 available? I only see sources on Github https://github.com/apache/kafka/releases/tag/2.8.0-rc0 https://github.com/apache/kafka/releases/tag/2.8.0-rc0 and couldn't find anything regarding 2.8.0-rc0 on the Kafka downloads page https://kafka.apache.org/downloads https://kafka.apache.org/downloads.

ewencp 6y ago
The downloads page is only for official releases, RCs are not distributed the same way and are not permanent unless promoted to an official release. See the mailing list thread for 2.8.0-RC0 for where to find the bits if you want to test https://lists.apache.org/thread.html/r16894a11aec73abac521ff93a8b154b3ebabadc85c0b7a932648be2e%40%3Cdev.kafka.apache.org%3E https://lists.apache.org/thread.html/r16894a11aec73abac521ff... and the project site has some "contact" info for mailing lists where these things are announced and advertised (including releases, Kafka Improvement Proposals, and more) https://kafka.apache.org/contact https://kafka.apache.org/contact

surfsvammel 6y ago

Great news! Props! I have a use case where this would make sense, I’ll dig in directly!

dig1 6y ago

Don't forget that ZooKeeper was the only service that passed all Jepsen [1] tests out of the box. [1] https://jepsen.io/analyses https://jepsen.io/analyses

silasiba 6y ago

Cool feature! I have a concern, if the Kafka broker provide both coordination service and Kafka service, how to achieve the resource isolation? If some of the topic which on the coordination service with very high throughput, this must cause instability of the coordination service, could this further cause the instability of the entire cluster? If some of the broker only provide the coordination service, what is this essential difference? Will this cause more problems for expansion and contraction? Will this bring greater risks when users scale down the brokers? I am very afraid that the coordination service will be shut down due to careless operation.

mumrah 6y ago
Part of the new design is that brokers and controllers can be run on separate JVMs.
- silasiba 6y ago
  Yes, that my second concern. If run on separate JVM, what is the difference between use ZK and use controller? Of course, If the new controller is more stable than ZK, more efficient than ZK, this will bring benefits to users. But this is essentially replacing zk with a better product, I think etcd also can achieve this purpose.

rad_gruchalski 6y ago

Works very well. Had some time to try out the 2.8 build and all existing tooling works just fine. Solid work by everyone involved in this.