5 ms·
Could be just me. I see Apache kafka more nightmarish than bureaucracy in 'The Castle' or Police/judiciary in 'The Trial' from author Franz Kafka. Mr Franz btw
by geodel 5y ago
Could be just me. I see Apache kafka more nightmarish than bureaucracy in 'The Castle' or Police/judiciary in 'The Trial' from author Franz Kafka. Mr Franz btw stole his last name from this very famous Apache Kafka project.
- rjzzleep 5y agoIt's quite beautiful, but put so simply it does look nightmarish. "Then, the otters would decide which part of the river to put the message in" I'd imagine a child looking at this going like "why are they doing this to the river". Why are they are throwing things into the river. :D
- lostcolony 5y agoNo, put so simply it's cute. In production it's nightmarish.
- gilbetron 5y agoWhat do you find nightmarish about it?
- manishsharan 5y agoKafka is very configurable .. and teams struggle to optimize the production deployment configuration and tend to second guess their decisions. Getting a managed deployment from Confluent etc. mitigates these issues as you can count on proven SMEs to deliver a well optimized solution. However, if you were to DIY , you might not be so blessed. Disclosure: I never did a DIY Kafka prod deploy -- my employer was already a Confluent customer.
- monkeyfacebag 5y agoMaybe it's just you, maybe not. It would be beneficial for you to provide some substance to your claim. Then we would have something to discuss.
- geodel 5y agoUnfortunately we do not have something to discuss. All these consumer/producer/broker failing randomly, system behaving erratically is me doing it wrong. Those white papers from Confluent and testimonial from customers are proof that Kafka works fine.
- dkarl 5y agoIndividual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, but people tend to keep going until they experience pain, at which point they have a system that is right at the limit of what they can understand. Then further feature work pushes them over the limit into darkness. I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.
- cbsmith 5y agoKSQL seems to address this gap.
- morelisp 5y agohttps://www.asyncapi.com/ https://www.asyncapi.com/ is in its infancy but is attempting what you describe. (I view OpenAPI as a near-failure, but, good luck to everyone trying.)
- dkarl 5y agoThanks for the tip; that's a really cool project. It looks like the documentation generator for it works on a single file at a time, and a single file defines a single service. I was thinking of something that works on the level of a system of interacting services. I want a tool that reads the API specifications for a group of services and documents the interactions between them. I'd like to look at service X and see that it emits message Y on queue Z, and then see which services read message Y on queue Z, jump to their documentation, etc. I think the AsyncAPI format is perfect to build on, though. I'll start there if I decide to take a hack at it.
- ridiculous_lol 5y agoI am not sure if I completely understand your use-case but it seems that the config can be distributed across multiple files. https://www.asyncapi.com/docs/specifications/v2.0.0#referenceObject https://www.asyncapi.com/docs/specifications/v2.0.0#referenc...
- Hamuko 5y agoIs this comment intentionally Kafkaesque?