5 ms·

Introducing zetcd

264 points by rtp 9y ago

tyingq 9y ago
Interesting. Maybe hashicorp should release an etcd compatible layer for Consul. Or CockroachDB even. Both have one thing etcd does not...inbuilt WAN support.
- philips 9y ago
  etcd can cross WAN links with tuning for the expected latencies. Tuning latencies is required to ensure the leader election algorithms know when to trigger a failure[1]. What is your use case? [1] https://coreos.com/etcd/docs/latest/tuning.html https://coreos.com/etcd/docs/latest/tuning.html
  - tyingq 9y ago
    Yes, you can sort of detune the whole cluster. That's not quite the same as Consul's and CockroachDB's specific WAN awareness. Those two take different approaches, but do specifically understand and compensate. >What is your use case? Contract work, so use case varies. I agree that etcd is often the right answer.
    - philips 9y ago
      Right, etcd's entire focus is on being a consensus database for distributed systems needing coordination, locking, etc. So, eventually consistent WAN replication hasn't really been a focus. I do think this sort of cross-cluster key replication is useful and we offer it as a userspace external tool called make-mirror[1]. [1] https://github.com/coreos/etcd/blob/master/etcdctl/README.md#utility-commands https://github.com/coreos/etcd/blob/master/etcdctl/README.md...
      - tyingq 9y ago
        Worth noting that CockroachDB isn't using an eventually consistent model. Yes, it's not the same thing as etcd, but I can see some potential use case overlap.
        ideal0227 9y ago
        There is some overlapping. But we should choose solution wisely :P. Here is a doc [https://github.com/coreos/etcd/blob/master/Documentation/learning/why.md https://github.com/coreos/etcd/blob/master/Documentation/lea...] comparing etcd with other systems, including CockroachDB. I work on etcd.
- jacksonnic 9y ago
  We might have just been talking about that
  - tyingq 9y ago
    "We" as in Hashicorp?
    - jacksonnic 9y ago
      Yeah, few people internally would like to tackle this
callumjones 9y ago
Great idea, we only run Zookeeper because of Kafka but everything else is in Consul so the idea of this existing is pretty neat.
- wink 9y ago
  Same here, Storm and Kafka - everything else is in consul - and 2 single sources of truth.. well :)
batbomb 9y ago
A few years ago I tried to reimplement the server connections directly in Zookeeper to be HTTP connections and a REST-like API and try to use long-polling and some other things. I got some code written, but no prototype up and I had to put it aside. This was initially before etcd was really going (I think it was at version 0.2 when I started the work). I had always hoped somebody else would try the same thing, but no such luck.
philips 9y ago
A big use case of this that we are thinking about is enabling people to use the etcd Operator[1], which makes it simple to run etcd clusters on Kubernetes, to back their ZooKeeper applications. The neat thing about the etcd Operator is you can define a cluster and the etcd Operator takes care of normal operations by using the Kubernetes API. apiVersion: "etcd.coreos.com/v1beta1" kind: "Cluster" metadata: name: "example-etcd-cluster" spec: size: 5 version: "3.1.8" Pretty neat! Anyways, the zetcd project is still super young but would love more folks to try it out. As the post says folks have already tried using Kafka, Mesos, and others. [1] https://coreos.com/blog/introducing-the-etcd-operator.html https://coreos.com/blog/introducing-the-etcd-operator.html
- philips 9y ago
  Oh, and here is a video explanation of the etcd Operator https://youtu.be/Uf7PiHXqmnw?t=11 https://youtu.be/Uf7PiHXqmnw?t=11
- manojlds 9y ago
  So I can run a Kubernetes cluster on Mesos and have the Zookeeper for Mesos deployed on the Kubernetes cluster using the etcd operator and zetcd Joking, of course.
  - koolba 9y ago
    The funny thing about the situation you describe is that there are real world examples of similar circular dependencies. I recall GitHub having an issue like that where their build pipeline used Bower which is hosted on GitHub. When shit hit the fan and a build broke the site, they could build the "fix" as Bower didn't work.
    - jzelinskie 9y ago
      My own experience working at CoreOS is that many of our projects exploit self-referentiality as it's a particularly useful property. Off the top of my head: - Quay.io, our registry service, is built and deployed by itself - Clair, our static analysis tool for detecting security vulnerabilities, analyzes itself - Tectonic, our enterprise Kubernetes distro, is "self-driving" and manages itself - discovery.etcd.io, a service we run to make it easier to bootstrap new etcd quorums, is just a quorum of etcd nodes
ninjakeyboard 9y ago
So how does zetcd handle the zookeeper session? It's in zetcd and then the "ephemeral nodes" are removed from etcd if the zk client session expires? This is handled by the proxy? This seems to be the major design compatibility item between the two is that the zookeeper protocol has a session state and so supports ephemeral nodes. If a gc pauses the jvm for a long time the client session will expire. In etcd there is ttl instead (zookeeper always lacked ttl as it wasn't needed)
- philips 9y ago
  A Go routine is spawned which runs the etcd client's KeepAlive method to tie a session to a lease in the proxy. The code is here: https://github.com/coreos/zetcd/blob/d33e3b836a2a2de8a8ec0771f5d249aa66d4bf8a/session.go#L48 https://github.com/coreos/zetcd/blob/d33e3b836a2a2de8a8ec077... Disclaimer: The person who wrote this is AFK at the moment so I might be completely wrong ;)
- Randgalt 9y ago
  > zookeeper always lacked ttl ZooKeeper 3.5.3 supports TTLs
Randgalt 9y ago
I did some quick testing with Apache Curator (note: I'm the main author of Curator) and it looks like zetcd isn't implementing the create2 opcode and several others. If the goal is to really be ZK compatible it has a long way to go. I'm not sure who the target of this is. But, I'll keep following and try to get the Curator tests to run when it's further along.
- philips 9y ago
  Issue filed: https://github.com/coreos/zetcd/issues/49 https://github.com/coreos/zetcd/issues/49
moderation 9y ago
Another reason to use zetcd over Zookeeper is security. I don't believe it is possible yet to use TLS for Zookeeper-to-Zookeeper communication (clients can connnect to Zookeeper using TLS). The Jira covering this feature [0]. This will increasingly be a problem for those running Zookeeper for Mesos, Kafka etc. from a security and risk point of view. [0] https://issues.apache.org/jira/browse/ZOOKEEPER-236 https://issues.apache.org/jira/browse/ZOOKEEPER-236

264 points by rtp 9y ago

Interesting. Maybe hashicorp should release an etcd compatible layer for Consul. Or CockroachDB even. Both have one thing etcd does not...inbuilt WAN support.

philips 9y ago
etcd can cross WAN links with tuning for the expected latencies. Tuning latencies is required to ensure the leader election algorithms know when to trigger a failure[1]. What is your use case? [1] https://coreos.com/etcd/docs/latest/tuning.html https://coreos.com/etcd/docs/latest/tuning.html
- tyingq 9y ago
  Yes, you can sort of detune the whole cluster. That's not quite the same as Consul's and CockroachDB's specific WAN awareness. Those two take different approaches, but do specifically understand and compensate. >What is your use case? Contract work, so use case varies. I agree that etcd is often the right answer.
  - philips 9y ago
    Right, etcd's entire focus is on being a consensus database for distributed systems needing coordination, locking, etc. So, eventually consistent WAN replication hasn't really been a focus. I do think this sort of cross-cluster key replication is useful and we offer it as a userspace external tool called make-mirror[1]. [1] https://github.com/coreos/etcd/blob/master/etcdctl/README.md#utility-commands https://github.com/coreos/etcd/blob/master/etcdctl/README.md...
    - tyingq 9y ago
      Worth noting that CockroachDB isn't using an eventually consistent model. Yes, it's not the same thing as etcd, but I can see some potential use case overlap.
      - ideal0227 9y ago
        There is some overlapping. But we should choose solution wisely :P. Here is a doc [https://github.com/coreos/etcd/blob/master/Documentation/learning/why.md https://github.com/coreos/etcd/blob/master/Documentation/lea...] comparing etcd with other systems, including CockroachDB. I work on etcd.
jacksonnic 9y ago
We might have just been talking about that
- tyingq 9y ago
  "We" as in Hashicorp?
  - jacksonnic 9y ago
    Yeah, few people internally would like to tackle this

callumjones 9y ago

Great idea, we only run Zookeeper because of Kafka but everything else is in Consul so the idea of this existing is pretty neat.

wink 9y ago
Same here, Storm and Kafka - everything else is in consul - and 2 single sources of truth.. well :)

batbomb 9y ago

A few years ago I tried to reimplement the server connections directly in Zookeeper to be HTTP connections and a REST-like API and try to use long-polling and some other things. I got some code written, but no prototype up and I had to put it aside. This was initially before etcd was really going (I think it was at version 0.2 when I started the work). I had always hoped somebody else would try the same thing, but no such luck.

philips 9y ago

A big use case of this that we are thinking about is enabling people to use the etcd Operator[1], which makes it simple to run etcd clusters on Kubernetes, to back their ZooKeeper applications. The neat thing about the etcd Operator is you can define a cluster and the etcd Operator takes care of normal operations by using the Kubernetes API. apiVersion: "etcd.coreos.com/v1beta1" kind: "Cluster" metadata: name: "example-etcd-cluster" spec: size: 5 version: "3.1.8" Pretty neat! Anyways, the zetcd project is still super young but would love more folks to try it out. As the post says folks have already tried using Kafka, Mesos, and others. [1] https://coreos.com/blog/introducing-the-etcd-operator.html https://coreos.com/blog/introducing-the-etcd-operator.html

philips 9y ago
Oh, and here is a video explanation of the etcd Operator https://youtu.be/Uf7PiHXqmnw?t=11 https://youtu.be/Uf7PiHXqmnw?t=11
manojlds 9y ago
So I can run a Kubernetes cluster on Mesos and have the Zookeeper for Mesos deployed on the Kubernetes cluster using the etcd operator and zetcd Joking, of course.
- koolba 9y ago
  The funny thing about the situation you describe is that there are real world examples of similar circular dependencies. I recall GitHub having an issue like that where their build pipeline used Bower which is hosted on GitHub. When shit hit the fan and a build broke the site, they could build the "fix" as Bower didn't work.
  - jzelinskie 9y ago
    My own experience working at CoreOS is that many of our projects exploit self-referentiality as it's a particularly useful property. Off the top of my head: - Quay.io, our registry service, is built and deployed by itself - Clair, our static analysis tool for detecting security vulnerabilities, analyzes itself - Tectonic, our enterprise Kubernetes distro, is "self-driving" and manages itself - discovery.etcd.io, a service we run to make it easier to bootstrap new etcd quorums, is just a quorum of etcd nodes

ninjakeyboard 9y ago

So how does zetcd handle the zookeeper session? It's in zetcd and then the "ephemeral nodes" are removed from etcd if the zk client session expires? This is handled by the proxy? This seems to be the major design compatibility item between the two is that the zookeeper protocol has a session state and so supports ephemeral nodes. If a gc pauses the jvm for a long time the client session will expire. In etcd there is ttl instead (zookeeper always lacked ttl as it wasn't needed)

philips 9y ago
A Go routine is spawned which runs the etcd client's KeepAlive method to tie a session to a lease in the proxy. The code is here: https://github.com/coreos/zetcd/blob/d33e3b836a2a2de8a8ec0771f5d249aa66d4bf8a/session.go#L48 https://github.com/coreos/zetcd/blob/d33e3b836a2a2de8a8ec077... Disclaimer: The person who wrote this is AFK at the moment so I might be completely wrong ;)
Randgalt 9y ago
> zookeeper always lacked ttl ZooKeeper 3.5.3 supports TTLs

Randgalt 9y ago

I did some quick testing with Apache Curator (note: I'm the main author of Curator) and it looks like zetcd isn't implementing the create2 opcode and several others. If the goal is to really be ZK compatible it has a long way to go. I'm not sure who the target of this is. But, I'll keep following and try to get the Curator tests to run when it's further along.

philips 9y ago
Issue filed: https://github.com/coreos/zetcd/issues/49 https://github.com/coreos/zetcd/issues/49

moderation 9y ago

Another reason to use zetcd over Zookeeper is security. I don't believe it is possible yet to use TLS for Zookeeper-to-Zookeeper communication (clients can connnect to Zookeeper using TLS). The Jira covering this feature [0]. This will increasingly be a problem for those running Zookeeper for Mesos, Kafka etc. from a security and risk point of view. [0] https://issues.apache.org/jira/browse/ZOOKEEPER-236 https://issues.apache.org/jira/browse/ZOOKEEPER-236