3 ms·

Kafka is (IME) pleasant to use for people writing application code. It can handle high message volumes. It has good documentation. Its core abstraction is easy

Kafka is (IME) pleasant to use for people writing application code. It can handle high message volumes. It has good documentation. Its core abstraction is easy to understand and use. OTOH, the author is correct that it's more work to set up and maintain than non-distributed software. Don't introduce it unless you have carefully considered alternatives. I became de facto responsible for running a small Kafka cluster at a small company that could have gotten away with something less shiny. That was not fun. (Especially since the company started with Kafka 0.8 and encountered many changes along the way to modern releases). Eventually the company ponied up for Confluent to take over running Kafka for us as a service. Now I'm back to enjoying the nice Kafka APIs without having to deal with the annoying Kafka servers. I like the Kafka API/capabilities but dislike running actual Kafka servers. If I were to start a one-man venture that needed event streaming I'd probably try Redpanda [1]. Its developer hangs out here, which is how I first learned of it. It's supposed to implement Kafka service but with no Zookeeper or JVM knowledge needed to run it. I wouldn't even consider running the original Apache Kafka for a solo/small team. [1] https://vectorized.io/ https://vectorized.io/

rjgonza 6y ago
100% agree on redpanda
- deleted 6y ago
  [deleted]
agallego 6y ago
that's correct - we actually don't manually write the kafka api, but we codegenerate it: https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafka/protocol/schemata/generator.py#L576-L631 https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafk... the tl;dr is that we wrote a pretty fast fragmented buffer parser (due to our thread-per-core design), we consume some json files and codegen it.... which basically means, if you connect an existing app that uses the kafka driver and it doesn't work it's a bug, and it's a matter of teaching the codegen to fix it most of the time. Transactions should land in the next month or so, which is the only thing really missing today. we have a ton of solo devs typically that come from the Go, Rust, JS & Python seem to like it quite a bit because of what you said.
mvolfik 6y ago
this. I'm working on a project which will need to be deployed in a local network without internet access. deploying kafka is a nightmare, more so in docker or k8s. I ended up switching to other tech rather than undergo this

rjgonza 6y ago

100% agree on redpanda

deleted 6y ago
[deleted]

agallego 6y ago

that's correct - we actually don't manually write the kafka api, but we codegenerate it: https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafka/protocol/schemata/generator.py#L576-L631 https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafk... the tl;dr is that we wrote a pretty fast fragmented buffer parser (due to our thread-per-core design), we consume some json files and codegen it.... which basically means, if you connect an existing app that uses the kafka driver and it doesn't work it's a bug, and it's a matter of teaching the codegen to fix it most of the time. Transactions should land in the next month or so, which is the only thing really missing today. we have a ton of solo devs typically that come from the Go, Rust, JS & Python seem to like it quite a bit because of what you said.

mvolfik 6y ago

this. I'm working on a project which will need to be deployed in a local network without internet access. deploying kafka is a nightmare, more so in docker or k8s. I ended up switching to other tech rather than undergo this