3 ms·

thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published

• film42 2y ago
  First time I've heard of KIP-932 and it looks very good. The two biggest issues IMO are finding a good Kafka client in the language you need (even for ruby this is a challenge) and easy at-least-once workers. You can over partition and make at-least-once workers happen (if you have a good Kafka client), or you use an http gateway and give up safe at-least-once. Hopefully this will make it easier to build an at-least-once style gateway that's easier to work with across a variety of languages. I know many have tried in the past but not dropping messages is hard to do right.
  • akshayshah 2y ago
    Couldn’t agree more - the most exciting thing about KIP-932 is how much easier it’ll become to build a good HTTP push gateway. Uber wrote a Kafka push gateway years ago, when it was considerably harder to do well: https://www.uber.com/blog/kafka-async-queuing-with-consumer-proxy/ https://www.uber.com/blog/kafka-async-queuing-with-consumer-...
  • mensfeld 2y ago
    Do you mind explaining what you mean by not being able to find a "good Kafka client" for Ruby? There are pretty good bindings to librdkafka and frameworks like Karafka (https://github.com/karafka/karafka/ https://github.com/karafka/karafka/) that provide many functionalities, including a Web UI.
    • film42 2y ago
      Late reply... I don't have my notes anymore on kafka client evals in ruby. When evaluating it was for a former employer. Karafka is very impressive though. Well done on the library.
• PhilippGille 2y ago
  TFA mentions it in the third paragraph: > Note: when Queues for Kafka (KIP-932) becomes a thing, a lot of these concerns go away. I look forward to it!
  • enether 2y ago
    yep, I was mostly clarifying the timeline