8 ms·

Show HN: Kaskade – A text user interface for Kafka

164 points by sauljp 2y ago

deleted 2y ago
[deleted]
sauljp 2y ago
Install it: brew install kaskade pipx install kaskade
1bent 2y ago
In case anyone else didn't know: https://en.wikipedia.org/wiki/Apache_Kafka https://en.wikipedia.org/wiki/Apache_Kafka
- Huntsecker 2y ago
  genuinely surprised that kafka is something unknown, thought it had gained ubiquitous status similar to k8s but maybe that's me walking into a Baader–Meinhof effect
  - vitaliyf 2y ago
    Franz Kafka is certainly not unknown, ever since long before people who worked on K8s were even born. See previous discussion here: https://news.ycombinator.com/item?id=29296969 https://news.ycombinator.com/item?id=29296969
cyberpunk 2y ago
Can it rebalance partitions? And delete crap from topics? Every time we have to call up our ops guys for this; it’s like “deep breath, first utter some indescribable magic aws iam nonsense and somehow get an ephemeral shell on some rando bitnami Kafka image’s ./kafka-topic shell scripts to work over the next few hours” and ultimately succeed but with deep regrets
- rockwotj 2y ago
  > delete crap from topics You have to rewrite the whole topic to do this right? (or do some hacks with compaction if you have unique keys)
  - bink 2y ago
    Or just advance the offsets for that partition beyond the problematic data, or adjust retention, or both.
    - nick0garvey 2y ago
      There is also the deleteRecords API specifically for this. It's easier than the retention shrink -> increase dance, as it is a single API call and retention does not kick in immediately. The log segment must roll for retention to apply, either due to size or time. https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/admin/AdminClient.html#deleteRecords-java.util.Map- https://kafka.apache.org/11/javadoc/org/apache/kafka/clients...
- sauljp 2y ago
  Hi! kaskade is more like AKHQ than Cruise control. We use strimzi by default, so we deploy cruise control with kafka, and it takes care of rebalancing the data across the nodes. Also you can deploy it without strimzi. Delete crap is more complicated, usually with kafka-delete-records (this is king of new I think). The problem is the offsets. By general rule you should not delete data from topics
- kvakerok 2y ago
  If you have to manually delete data from topics, you are doing Kafka wrong. The whole point of it is high speed data throughput, so that something automated does it for you downstream.
  - sauljp 2y ago
    Totally agree
terminaltrove 2y ago
We featured kaskade on Terminal Trove as tool of the week and looks really cool and useful. https://terminaltrove.com/kaskade/ https://terminaltrove.com/kaskade/ For those interested, kaskade is made with the Textual TUI framework. https://www.textualize.io/ https://www.textualize.io/ Thanks for making this sauljp.
- sauljp 2y ago
  thanks! that is awesome
- potamic 2y ago
  Slightly tangential, but curious why you have a browse by language section? The language a tool is written in shouldn't matter much for using it.
  - blowski 2y ago
    All other things being equal, Id prefer to use a tool written in a language with which I’m already comfortable. It’s nice to find a tool that meets my requirements _and_ I can easily read the source code.
    - sauljp 2y ago
      100%
  - zellyn 2y ago
    It often has large implications on how easy/difficult or self-contained/messy it is to install.
- deleted 2y ago
  [deleted]
ddoolin 2y ago
Does it support Protobuf deserialization via Schema Registry? This is basically where every other tool falls apart. Kafka UI just added support very recently but kcat falls apart.
- sauljp 2y ago
  Hi! it validates if the message was (or not) generated for schema registry, it checks if the message has the schema registry magic bytes (this bytes has the schema id). so, it deserializes messages with or without schema registry. This is the drawback: you have to generate the descriptor first. 1) download the schema from schema registry: http :8081/schemas/ids/<my-id>/schema 2) generate the descriptor with the schema: protoc --include_imports --descriptor_set_out=my-descriptor.desc --proto_path=. my-schema.proto 3) use kaskade: kaskade consumer -b my-kafka:9092 -x auto.offset.reset=earliest -k string -v protobuf -t my-protobuf-topic -p descriptor=my-descriptor.desc -p value=mypackage.MyMessage
  - sauljp 2y ago
    about the first step: if you already have the .proto file you do not need to download it from schema registry
  - ddoolin 2y ago
    Perfect! I had tried this last night before your comment (I saw these instructions in your README) but I was trying to include `-s url=` and `-s basic.auth...` which appears to force you into using `-k avro` or `-v avro`. But, once I removed the Schema Registry config options, I finally got the TUI showing up and was able to read messages! Thank you very much. Nearly all of my topics are Pb-encoded so I'll have to throw something together to get all the descriptors and perhaps abstract a bit. I love seeing the different uses of Textual coming out and this is no different; this is a big improvement from kcat IMO.
    - sauljp 2y ago
      Hey ddoolin, It's so nice to read this, yes! kaskade ignores the schema registry metadata for json and protobuf messages generated for schema registry, you do not need to provide url (only in case of using avro). The only catch (in the protobuf case) is to generate the descriptors with protoc, but the good thing is generally we all have them, so maybe is not a big problem.
kvakerok 2y ago
I'm not even going to ask why. Just start following the repo...
- sauljp 2y ago
  Thanks!
westurner 2y ago
Celery Flower TUI doesn't yet support Kafka in Python, though Kombu (the protocol wrapper for celery) does. It looks like Kaskade has non-kombu support for consumers and ? https://github.com/celery/celery/issues/7674#issuecomment-1204053595 https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365 https://news.ycombinator.com/item?id=40842365 (Edit) Kaskade models.py, consumer.py https://github.com/sauljabin/kaskade/blob/main/kaskade/models.py https://github.com/sauljabin/kaskade/blob/main/kaskade/model... kombu/transport/confluentkafka.py: https://github.com/celery/kombu/blob/main/kombu/transport/confluentkafka.py https://github.com/celery/kombu/blob/main/kombu/transport/co... confluent-kafka-python wraps librdkafka with binary wheels: https://github.com/confluentinc/confluent-kafka-python https://github.com/confluentinc/confluent-kafka-python librdkafka: https://github.com/edenhill/librdkafka https://github.com/edenhill/librdkafka
oulipo 2y ago
In Textual, I wanted to write a logging app with a big data table and filtering, and I'd hoped that it would be a bit more straightforward to write an optimized one I guess it would be a nice addition to have some kind of FilterableDataTable with history, filtering, caching, and fast rendering I guess you probably developed something like that for this tool, perhaps you could share it in Textual, or some kind of "textual widgets extension lib"?
- sauljp 2y ago
  Hi! I did not implemented something like that, but I can say this year textual has a great set of features and a nice community willing to help and share. I totally recommend textual.
lordgrenville 2y ago
I always thought TUI stood for "terminal user interface", though I guess "text" makes sense. First sentence of the Wikipedia article has both. https://en.wikipedia.org/wiki/Text-based_user_interface https://en.wikipedia.org/wiki/Text-based_user_interface
- sauljp 2y ago
  I guess both work :)

164 points by sauljp 2y ago

deleted 2y ago

[deleted]

sauljp 2y ago

Install it: brew install kaskade pipx install kaskade

1bent 2y ago

In case anyone else didn't know: https://en.wikipedia.org/wiki/Apache_Kafka https://en.wikipedia.org/wiki/Apache_Kafka

Huntsecker 2y ago
genuinely surprised that kafka is something unknown, thought it had gained ubiquitous status similar to k8s but maybe that's me walking into a Baader–Meinhof effect
- vitaliyf 2y ago
  Franz Kafka is certainly not unknown, ever since long before people who worked on K8s were even born. See previous discussion here: https://news.ycombinator.com/item?id=29296969 https://news.ycombinator.com/item?id=29296969

cyberpunk 2y ago

Can it rebalance partitions? And delete crap from topics? Every time we have to call up our ops guys for this; it’s like “deep breath, first utter some indescribable magic aws iam nonsense and somehow get an ephemeral shell on some rando bitnami Kafka image’s ./kafka-topic shell scripts to work over the next few hours” and ultimately succeed but with deep regrets

rockwotj 2y ago
> delete crap from topics You have to rewrite the whole topic to do this right? (or do some hacks with compaction if you have unique keys)
- bink 2y ago
  Or just advance the offsets for that partition beyond the problematic data, or adjust retention, or both.
  - nick0garvey 2y ago
    There is also the deleteRecords API specifically for this. It's easier than the retention shrink -> increase dance, as it is a single API call and retention does not kick in immediately. The log segment must roll for retention to apply, either due to size or time. https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/admin/AdminClient.html#deleteRecords-java.util.Map- https://kafka.apache.org/11/javadoc/org/apache/kafka/clients...
sauljp 2y ago
Hi! kaskade is more like AKHQ than Cruise control. We use strimzi by default, so we deploy cruise control with kafka, and it takes care of rebalancing the data across the nodes. Also you can deploy it without strimzi. Delete crap is more complicated, usually with kafka-delete-records (this is king of new I think). The problem is the offsets. By general rule you should not delete data from topics
kvakerok 2y ago
If you have to manually delete data from topics, you are doing Kafka wrong. The whole point of it is high speed data throughput, so that something automated does it for you downstream.
- sauljp 2y ago
  Totally agree

terminaltrove 2y ago

We featured kaskade on Terminal Trove as tool of the week and looks really cool and useful. https://terminaltrove.com/kaskade/ https://terminaltrove.com/kaskade/ For those interested, kaskade is made with the Textual TUI framework. https://www.textualize.io/ https://www.textualize.io/ Thanks for making this sauljp.

sauljp 2y ago
thanks! that is awesome
potamic 2y ago
Slightly tangential, but curious why you have a browse by language section? The language a tool is written in shouldn't matter much for using it.
- blowski 2y ago
  All other things being equal, Id prefer to use a tool written in a language with which I’m already comfortable. It’s nice to find a tool that meets my requirements _and_ I can easily read the source code.
  - sauljp 2y ago
    100%
- zellyn 2y ago
  It often has large implications on how easy/difficult or self-contained/messy it is to install.
deleted 2y ago
[deleted]

ddoolin 2y ago

Does it support Protobuf deserialization via Schema Registry? This is basically where every other tool falls apart. Kafka UI just added support very recently but kcat falls apart.

sauljp 2y ago
Hi! it validates if the message was (or not) generated for schema registry, it checks if the message has the schema registry magic bytes (this bytes has the schema id). so, it deserializes messages with or without schema registry. This is the drawback: you have to generate the descriptor first. 1) download the schema from schema registry: http :8081/schemas/ids/<my-id>/schema 2) generate the descriptor with the schema: protoc --include_imports --descriptor_set_out=my-descriptor.desc --proto_path=. my-schema.proto 3) use kaskade: kaskade consumer -b my-kafka:9092 -x auto.offset.reset=earliest -k string -v protobuf -t my-protobuf-topic -p descriptor=my-descriptor.desc -p value=mypackage.MyMessage
- sauljp 2y ago
  about the first step: if you already have the .proto file you do not need to download it from schema registry
- ddoolin 2y ago
  Perfect! I had tried this last night before your comment (I saw these instructions in your README) but I was trying to include `-s url=` and `-s basic.auth...` which appears to force you into using `-k avro` or `-v avro`. But, once I removed the Schema Registry config options, I finally got the TUI showing up and was able to read messages! Thank you very much. Nearly all of my topics are Pb-encoded so I'll have to throw something together to get all the descriptors and perhaps abstract a bit. I love seeing the different uses of Textual coming out and this is no different; this is a big improvement from kcat IMO.
  - sauljp 2y ago
    Hey ddoolin, It's so nice to read this, yes! kaskade ignores the schema registry metadata for json and protobuf messages generated for schema registry, you do not need to provide url (only in case of using avro). The only catch (in the protobuf case) is to generate the descriptors with protoc, but the good thing is generally we all have them, so maybe is not a big problem.

kvakerok 2y ago

I'm not even going to ask why. Just start following the repo...

sauljp 2y ago
Thanks!

westurner 2y ago

Celery Flower TUI doesn't yet support Kafka in Python, though Kombu (the protocol wrapper for celery) does. It looks like Kaskade has non-kombu support for consumers and ? https://github.com/celery/celery/issues/7674#issuecomment-1204053595 https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365 https://news.ycombinator.com/item?id=40842365 (Edit) Kaskade models.py, consumer.py https://github.com/sauljabin/kaskade/blob/main/kaskade/models.py https://github.com/sauljabin/kaskade/blob/main/kaskade/model... kombu/transport/confluentkafka.py: https://github.com/celery/kombu/blob/main/kombu/transport/confluentkafka.py https://github.com/celery/kombu/blob/main/kombu/transport/co... confluent-kafka-python wraps librdkafka with binary wheels: https://github.com/confluentinc/confluent-kafka-python https://github.com/confluentinc/confluent-kafka-python librdkafka: https://github.com/edenhill/librdkafka https://github.com/edenhill/librdkafka

oulipo 2y ago

In Textual, I wanted to write a logging app with a big data table and filtering, and I'd hoped that it would be a bit more straightforward to write an optimized one I guess it would be a nice addition to have some kind of FilterableDataTable with history, filtering, caching, and fast rendering I guess you probably developed something like that for this tool, perhaps you could share it in Textual, or some kind of "textual widgets extension lib"?

sauljp 2y ago
Hi! I did not implemented something like that, but I can say this year textual has a great set of features and a nice community willing to help and share. I totally recommend textual.

lordgrenville 2y ago

I always thought TUI stood for "terminal user interface", though I guess "text" makes sense. First sentence of the Wikipedia article has both. https://en.wikipedia.org/wiki/Text-based_user_interface https://en.wikipedia.org/wiki/Text-based_user_interface

sauljp 2y ago
I guess both work :)