3 ms·

it’s a little funny to think that if you’re running ZooKeeper in Kubernetes that you’re using etcd to manage the state of the servers of your state management s

by klysm 7y ago

it’s a little funny to think that if you’re running ZooKeeper in Kubernetes that you’re using etcd to manage the state of the servers of your state management servers

FBISurveillance 7y ago
In context of Kafka, hopefully KIP-500 [1] will get implemented sometime soon. If you're feeling lucky, you can also use zetcd [2] to connect ZK apps to etcd. I've been able to actually run Kafka with it as a toy project a little while ago. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... [2] https://github.com/etcd-io/zetcd https://github.com/etcd-io/zetcd
- klysm 7y ago
  Given the close similarity in guarantees, this kind of thing seemed possible but I didn’t know it actually existed! Would it be sketchy to use the same etcd instance as k8s though? It seems desirable to keep that isolated so you don’t fuck up your whole cluster
- takeda 7y ago
  What is the problem with ZK though? The only thing I can think of is that requires JVM, but other than that it is quite solid as far as the locking service is concerned, and since you are using Kafka you already are using JVM.
  - FridgeSeal 7y ago
    Because if I already have a K8’s cluster, and I want to run a application that requires ZK, I now have to dedicate resources to running a ZK cluster, within my K8’s cluster just to run the one application I cared about. I also personally find configuring and running Java applications confusing AF. Why are the configs seemingly split into different places and environment variables? Does it get clearer after dealing with Java things for a while?
- jpgvm 7y ago
  I would caution against using zetcd specifically with Kafka however as we ran into issues with this in the past.
hinkley 7y ago
Circular dependencies at the bottom of your tech stack are trouble. I'm having a slow motion argument with a coworker about a piece of code I maintain. It provides bootstrapping data that nearly all of our code uses in some way. Everything from stats and logging up to user-visible functionality. Every time I run into a hiccup, he's there asking why I'm not using our telemetry or networking code for the internals. It's true that if you are very, very careful, you can manage circular startup dependencies, but anyone sneezes and your app won't start or worse, drops into an infinite loop. Either you build on simpler building blocks with similar functionality, or you find a different way to organize the code.

by klysm 7y ago

it’s a little funny to think that if you’re running ZooKeeper in Kubernetes that you’re using etcd to manage the state of the servers of your state management servers

FBISurveillance 7y ago

In context of Kafka, hopefully KIP-500 [1] will get implemented sometime soon. If you're feeling lucky, you can also use zetcd [2] to connect ZK apps to etcd. I've been able to actually run Kafka with it as a toy project a little while ago. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... [2] https://github.com/etcd-io/zetcd https://github.com/etcd-io/zetcd

klysm 7y ago
Given the close similarity in guarantees, this kind of thing seemed possible but I didn’t know it actually existed! Would it be sketchy to use the same etcd instance as k8s though? It seems desirable to keep that isolated so you don’t fuck up your whole cluster
takeda 7y ago
What is the problem with ZK though? The only thing I can think of is that requires JVM, but other than that it is quite solid as far as the locking service is concerned, and since you are using Kafka you already are using JVM.
- FridgeSeal 7y ago
  Because if I already have a K8’s cluster, and I want to run a application that requires ZK, I now have to dedicate resources to running a ZK cluster, within my K8’s cluster just to run the one application I cared about. I also personally find configuring and running Java applications confusing AF. Why are the configs seemingly split into different places and environment variables? Does it get clearer after dealing with Java things for a while?
jpgvm 7y ago
I would caution against using zetcd specifically with Kafka however as we ran into issues with this in the past.

hinkley 7y ago

Circular dependencies at the bottom of your tech stack are trouble. I'm having a slow motion argument with a coworker about a piece of code I maintain. It provides bootstrapping data that nearly all of our code uses in some way. Everything from stats and logging up to user-visible functionality. Every time I run into a hiccup, he's there asking why I'm not using our telemetry or networking code for the internals. It's true that if you are very, very careful, you can manage circular startup dependencies, but anyone sneezes and your app won't start or worse, drops into an infinite loop. Either you build on simpler building blocks with similar functionality, or you find a different way to organize the code.