5 ms·
Topicctl – an easier way to manage Kafka topics
- rbranson 6y agoSegment employee here. Our Kafka runbook looks 100X cleaner after this tool was rolled out by Ben and folks. One of my favorite parts is how it makes dangerous things harder :)
- sumtechguy 6y agonice! Just skimmed the docs. Does it handle ACL or stretched cluster yet? Planned?
- bhyolken 6y agoNot yet! But would be happy to look into those and/or get help adding them.
- dblooman 6y agoDid you look at using terraform for managing kafka topics?
- pryz 6y agoSegment employee here. Yes, we looked at Terraform to manage our topics. We are big Terraform user to manage our infrastructure. We mostly didn't go that way because we didn't want to have to manage the Terraform state. We also wanted the tool to be extensible and writing custom TF providers can sometime be tricky.
- xref 6y agoIs there a reason you chose not to use ACLs? We did enable them, but it feels like it is causing more headaches than any security benefit it might provide...
- pryz 6y agoI guess we just haven't got the need to use ACL yet. This may be something we will look later on. We have been started to implement what's needed to leverage things like SASL in our instrumentation but we are not there yet (see https://github.com/segmentio/kafka-go https://github.com/segmentio/kafka-go).
- tekkertje 6y agoThanks for this! Managing Kafka and topics can be messy at times, so a really welcome addition to apply more robust Kafka changes.
- retzkek 6y ago> Replica placement strategy This is huge, currently you have to manually specify placement for all replicas when you increase the replication factor for a topic. Do this for many topics and it quickly becomes very tedious and error-prone. Thanks for this, I'm very excited to try it out!
- abledon 6y ago"""Many of these were shaped by our experiences making AWS changes with Terraform and Kubernetes changes with kubectl. We wanted an equivalent to these for Kafka!""" so use this topicctl for advanced topic changes.... and the more primitive kafkactl[1] for basic stuff? What about the 10 or so other GUI managers like this one? https://github.com/cloudhut/kowl https://github.com/cloudhut/kowl [2] [1]https://github.com/deviceinsight/kafkactl https://github.com/deviceinsight/kafkactl [2] https://news.ycombinator.com/item?id=24099037 https://news.ycombinator.com/item?id=24099037
- bhyolken 6y agoAuthor here, thanks for the comment. topicctl is really motivated by a desire to support rigorous, git-based topic management. The read-only views (tailing, repl, etc.) are secondary to that and are useful for our command-line-based workflows inside Segment, but are definitely not intended to replace all of the other good tooling out there, including the ones you've referenced.
- EdwardDiego 6y agoI think the key part is that your topic config is stored in version control.
- infogulch 6y agokafka-gitops [1] is another declarative topic/acl management tool that enables a planfile/apply strategy. [1]: https://github.com/devshawn/kafka-gitops https://github.com/devshawn/kafka-gitops
- caiobegotti 6y agoThat looks very interesting! About managing YAMLs like we'd do with kubectl as explained in the article, how does topicctl differ both in concept and operation from Strimzi's Kafka Operator for Kubernetes? I know, they are a different company etc but that Kafka operator is gitops-friendly and everything so I'm honestly trying to compare them as managing Kafka is a REALLY HARD problem, so kudos for trying to make it easier!
- pryz 6y agoAt Segment we manage clusters which are both on ECS and EKS. We didn't want to tie our Kafka management to the container orchestration. If I'm right the Strimzi operator is also managing the brokers. Here we only focus on the topics.