4 ms·

One thing I've found extremely difficult to handle is the Zookeeper cluster model of containers. Where when a thing dies, a thing has to come back and be able t

One thing I've found extremely difficult to handle is the Zookeeper cluster model of containers. Where when a thing dies, a thing has to come back and be able to referred to as "zookeeper-1" forever. The way to do this currently is to use a service in front of a replication controller with one pod. This feels wrong all over. Supposedly they have a thing called Pet Sets [1] coming to solve this, but it's been in the works for an eternity. Also we've started to outgrow the load balancing simplicity that the k8 load balancer gives you, and I have not seen a nice migration path to something like HAProxy in Kubernetes. All that said, we like kubernetes a lot. [1] To distinguish from cattle. If you have a red blue and green goldfish, and your red goldfish dies, you can replace with another red fish and not really notice, but if it's purple, the others won't play with it.

jat850 10y ago
Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or your load balancer of choice. Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?
- _asummers 10y ago
  So it would basically be daemon set sitting on top of all the other services, with their own load balancing turned off? That should work. I like that
  - olafmol 10y ago
    That's how we do it with Vamp. Interested in your thoughts: http://vamp.io/documentation/installation/kubernetes/ http://vamp.io/documentation/installation/kubernetes/
  - jat850 10y ago
    Email's in profile. I'm happy to discuss specifics offline, as I have some experience with this. Can show some example configs of how we've employed it.
  - ex3ndr 10y ago
    While we haven't hit performance limits, but we are doing something same and it works great.
crb 10y ago
PetSet will indeed solve your exact use case, and the docs are "Alpha" as is the feature. But here is a Zookeeper example: https://github.com/kubernetes/contrib/tree/master/pets/zookeeper https://github.com/kubernetes/contrib/tree/master/pets/zooke...
philips 10y ago
For a lot of clustered software it makes sense to build small controllers and resources that bridge Kubernetes's knowledge of the system and the distributed systems knowledge. See this demo of a WIP etcd controller from the community hangout: https://youtu.be/Pc9NlFEjEOc?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&t=46 https://youtu.be/Pc9NlFEjEOc?list=PL69nYSiGNLP1pkHsbPjzAewvM...
lobster_johnson 10y ago
Re HAProxy, take a look at Traefik. It's a thin, full-featured load balancer that supports Kubernetes out of the box. https://traefik.io/ https://traefik.io/ Another option is running Nginx via the Nginx Ingress Controller, which is also great.
lobster_johnson 10y ago
Petsets can be used in 1.3, but there are some outstanding bugs that still need to be squashed. I believe it will reach beta status in 1.4. Beta features are usually pretty production-ready (deployments, for example, are flagged as beta in 1.3, and have been rock solid for me).
- smarterclayton 10y ago
  Disclaimer: PetSet co-author PetSets will probably be beta in 1.5. The problem with rushing PetSets is that the worst thing that can happen would be that we say PetSets are beta and then it eats someone's DB for lunch. The goal is to get the core function stable and predictable (put enough stress testing and partition testing that we are comfortable with it) and put it into a longish beta while we add out the features that are necessary to make it able to predictably and stably keep running your clustered services. We want beta to mean "stable enough to trust".
  - lobster_johnson 10y ago
    That's a bit disappointing. People have been waiting for petsets for quite a while now. But thanks for the info. FWIW, some other less-stable features (the new volume attaching logic in KCM, for example) should have been marked as beta. You guys are not quite consistent about what you feel is stable. :-)
    - smarterclayton 10y ago
      Yeah, there's a fair amount of stuff going in and we aren't always consistent about what we turn on / off. In OpenShift we tend to be much more conservative (i.e. 1.3 OpenShift defaults controller volume attach to off, but the Ansible upgrader will turn it on by default). We promise to do a better job of communicating alpha status - it's been a topic of a lot of discussion recently. To be fair - we've only had two (almost three!) update releases so we're still trying to be consistent in our messaging. 1.4 will try to be a lot better.
      - 10y ago

jat850 10y ago

Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or your load balancer of choice. Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?

_asummers 10y ago
So it would basically be daemon set sitting on top of all the other services, with their own load balancing turned off? That should work. I like that
- olafmol 10y ago
  That's how we do it with Vamp. Interested in your thoughts: http://vamp.io/documentation/installation/kubernetes/ http://vamp.io/documentation/installation/kubernetes/
- jat850 10y ago
  Email's in profile. I'm happy to discuss specifics offline, as I have some experience with this. Can show some example configs of how we've employed it.
- ex3ndr 10y ago
  While we haven't hit performance limits, but we are doing something same and it works great.

crb 10y ago

PetSet will indeed solve your exact use case, and the docs are "Alpha" as is the feature. But here is a Zookeeper example: https://github.com/kubernetes/contrib/tree/master/pets/zookeeper https://github.com/kubernetes/contrib/tree/master/pets/zooke...

philips 10y ago

For a lot of clustered software it makes sense to build small controllers and resources that bridge Kubernetes's knowledge of the system and the distributed systems knowledge. See this demo of a WIP etcd controller from the community hangout: https://youtu.be/Pc9NlFEjEOc?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&t=46 https://youtu.be/Pc9NlFEjEOc?list=PL69nYSiGNLP1pkHsbPjzAewvM...

lobster_johnson 10y ago

Re HAProxy, take a look at Traefik. It's a thin, full-featured load balancer that supports Kubernetes out of the box. https://traefik.io/ https://traefik.io/ Another option is running Nginx via the Nginx Ingress Controller, which is also great.

lobster_johnson 10y ago

Petsets can be used in 1.3, but there are some outstanding bugs that still need to be squashed. I believe it will reach beta status in 1.4. Beta features are usually pretty production-ready (deployments, for example, are flagged as beta in 1.3, and have been rock solid for me).

smarterclayton 10y ago
Disclaimer: PetSet co-author PetSets will probably be beta in 1.5. The problem with rushing PetSets is that the worst thing that can happen would be that we say PetSets are beta and then it eats someone's DB for lunch. The goal is to get the core function stable and predictable (put enough stress testing and partition testing that we are comfortable with it) and put it into a longish beta while we add out the features that are necessary to make it able to predictably and stably keep running your clustered services. We want beta to mean "stable enough to trust".
- lobster_johnson 10y ago
  That's a bit disappointing. People have been waiting for petsets for quite a while now. But thanks for the info. FWIW, some other less-stable features (the new volume attaching logic in KCM, for example) should have been marked as beta. You guys are not quite consistent about what you feel is stable. :-)
  - smarterclayton 10y ago
    Yeah, there's a fair amount of stuff going in and we aren't always consistent about what we turn on / off. In OpenShift we tend to be much more conservative (i.e. 1.3 OpenShift defaults controller volume attach to off, but the Ansible upgrader will turn it on by default). We promise to do a better job of communicating alpha status - it's been a topic of a lot of discussion recently. To be fair - we've only had two (almost three!) update releases so we're still trying to be consistent in our messaging. 1.4 will try to be a lot better.
    - 10y ago