4 ms·
Individual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, b
by dkarl 5y ago
Individual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, but people tend to keep going until they experience pain, at which point they have a system that is right at the limit of what they can understand. Then further feature work pushes them over the limit into darkness.
I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.
- cbsmith 5y agoKSQL seems to address this gap.
- morelisp 5y agohttps://www.asyncapi.com/ https://www.asyncapi.com/ is in its infancy but is attempting what you describe. (I view OpenAPI as a near-failure, but, good luck to everyone trying.)
- dkarl 5y agoThanks for the tip; that's a really cool project. It looks like the documentation generator for it works on a single file at a time, and a single file defines a single service. I was thinking of something that works on the level of a system of interacting services. I want a tool that reads the API specifications for a group of services and documents the interactions between them. I'd like to look at service X and see that it emits message Y on queue Z, and then see which services read message Y on queue Z, jump to their documentation, etc. I think the AsyncAPI format is perfect to build on, though. I'll start there if I decide to take a hack at it.
- ridiculous_lol 5y agoI am not sure if I completely understand your use-case but it seems that the config can be distributed across multiple files. https://www.asyncapi.com/docs/specifications/v2.0.0#referenceObject https://www.asyncapi.com/docs/specifications/v2.0.0#referenc...
- dkarl 5y agoI'm taking a closer look today, and it looks to me like one AsyncAPI document, which can be defined using multiple files, defines one application. If you look at the "fixed fields" section under the root object, the second field is "id"[0]: id Identifier Identifier of the application the AsyncAPI document is defining. This format is fine, but the tool I'm looking for is something that will read the definitions of multiple AsyncAPI documents (multiple applications) and show how their inputs and outputs connect, so I can answer a question like, "When application X publishes message Y on channel Z, which applications consume that message?" AsyncAPI gets me 90% there by defining the service spec and providing code to parse it. It's possible somebody has already written the rest; I'll have to see. [0] https://www.asyncapi.com/docs/specifications/v2.0.0#fixed-fields https://www.asyncapi.com/docs/specifications/v2.0.0#fixed-fi...