4 ms·

Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compa

by apgwoz 9mo ago

Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!

nchmy 9mo ago
I wonder how it compares to Redpanda
- anticodon 9mo ago
  I've used Redpanda for local development and testing stands. It is super easy to setup in docker, starts really fast and consumes less resources than Java version. Haven't really compared it to anything, but I remember using Java version of Kafka before and it was a resource hog. It is important when you develop on laptop with constrained resources.
  - nchmy 9mo ago
    What I meant was how Tensu compares to Redpanda
  - enether 9mo ago
    to be fair, Kafka now has a GraalVM docker image[0][1] which was made for local dev/testing, and it has caught up fairly well to these alternatives re: memory and startup time [0] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A... [1] - https://hub.docker.com/r/apache/kafka-native https://hub.docker.com/r/apache/kafka-native
kitd 9mo ago
My thoughts too. > kafka compatible Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience). So I'd be interested how (backward-)compatible they are.
- shortishly 9mo ago
  I agree that it isn't straight forward! Tansu uses the JSON protocol descriptors from Apache Kafka, generating ~60k LoC of Rust to represent the structures. It then uses a custom Serde encoder/decoder to implement the protocol: original, flexible and tag buffers formats for every API version (e.g., the 18 just in FETCH). It is based off spending the past ~10 years using Kafka, and writing/maintaining an Erlang client (there are no "good" Kafka clients for Erlang!). It also uses a bunch of collected protocol examples, to encode/decode during the tests. Tansu is also a Kafka proxy, which is also used to feed some of those tests. Some of the detail: https://blog.tansu.io/articles/serde-kafka-protocol https://blog.tansu.io/articles/serde-kafka-protocol However, there are definitely cases I am sure where Tansu isn't compatible. For example, Kafka UI (kafbat) reports a strange error when doing a fetch (despite actually showing the fetched data), which I've yet to get to the bottom of. If you find any compatibility issues, then please raise an issue, and I can take a look.

by apgwoz 9mo ago

nchmy 9mo ago

I wonder how it compares to Redpanda

anticodon 9mo ago
I've used Redpanda for local development and testing stands. It is super easy to setup in docker, starts really fast and consumes less resources than Java version. Haven't really compared it to anything, but I remember using Java version of Kafka before and it was a resource hog. It is important when you develop on laptop with constrained resources.
- nchmy 9mo ago
  What I meant was how Tensu compares to Redpanda
- enether 9mo ago
  to be fair, Kafka now has a GraalVM docker image[0][1] which was made for local dev/testing, and it has caught up fairly well to these alternatives re: memory and startup time [0] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A... [1] - https://hub.docker.com/r/apache/kafka-native https://hub.docker.com/r/apache/kafka-native

kitd 9mo ago

My thoughts too. > kafka compatible Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience). So I'd be interested how (backward-)compatible they are.

shortishly 9mo ago
I agree that it isn't straight forward! Tansu uses the JSON protocol descriptors from Apache Kafka, generating ~60k LoC of Rust to represent the structures. It then uses a custom Serde encoder/decoder to implement the protocol: original, flexible and tag buffers formats for every API version (e.g., the 18 just in FETCH). It is based off spending the past ~10 years using Kafka, and writing/maintaining an Erlang client (there are no "good" Kafka clients for Erlang!). It also uses a bunch of collected protocol examples, to encode/decode during the tests. Tansu is also a Kafka proxy, which is also used to feed some of those tests. Some of the detail: https://blog.tansu.io/articles/serde-kafka-protocol https://blog.tansu.io/articles/serde-kafka-protocol However, there are definitely cases I am sure where Tansu isn't compatible. For example, Kafka UI (kafbat) reports a strange error when doing a fetch (despite actually showing the fetched data), which I've yet to get to the bottom of. If you find any compatibility issues, then please raise an issue, and I can take a look.