4 ms·

Kubernetes is very much still under development. Most of the work up to 1.0 has gone into refinement of the API, rather than stability. For instance, HA is on

Kubernetes is very much still under development. Most of the work up to 1.0 has gone into refinement of the API, rather than stability. For instance, HA is only now being seriously worked on - and if your API server goes down and you need to start it on another host, if you don't have an external reconvergence mechanism you have to restart every kubelet so they can point to the new one. Mesos is much more mature, but is more low-level, and schedulers like Aurora or Marathon do not give you the same service abstraction, but they have many features not currently present in Kubernetes. But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes. Personally I see Kubernetes-on-Mesos as being a really bright future choice, but like pretty much anything other than Marathon/Aurora on Mesos, it's pretty nascent.

kalmar 11y ago
> But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes. Can you give an example? I know next to nothing about Mesos, Aurora, or Marathon.
- krenoten 11y ago
  It's like many things that are declarative - you can only do what has been explicitly enabled in the declaration handling logic. This is good and bad. It's good because it restricts what is possible (protecting you slightly more from bad users) and can make learning and using the system easier. But you can't go beyond this. With a non-declarative scheduler you can implement far more interesting (for better or worse) management systems. One area where we've been seeing some interesting work happening is in stateful services running on top of schedulers. With something like postgres, there is interesting logic that must happen for responsible management beyond just throwing up 3 of them. You need to manage replication carefully. When a node goes down somewhere, you need to react somewhere else in a controlled way. With Mesos, you can write a scheduler that encodes your runbook. With kubernetes, you need a lot more out-of-band stuff.
  - TheIronYuppie 11y ago
    Disclaimer: I work at Google on Kubernetes. You're correct that the scheduler in the box is declarative. However, it is easy (well, as easy as writing a framework) to swap out a scheduler that might be a better fit for what you're looking for. See this stackoverflow answer by one of the core Kubernetes team for more info: http://stackoverflow.com/questions/28857993/how-does-kubernetes-scheduler-work http://stackoverflow.com/questions/28857993/how-does-kuberne...
mkulke 11y ago
Yeah, the HA information was buried/non-existent in the docs until recently, I guess. We also ran into issues when replacing the master, b/c the kubelets where fixed to an ip and the nodes' configuration was basically immutable (provisioned by cloud-config). What worked for us: Accessing the the master by DNS and putting etcd on a persistent volume. This way we're able to replace the master within a DNS records ttl. As the api server is not a hard requirement for the workload, this is HA enough for us.

kalmar 11y ago

> But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes. Can you give an example? I know next to nothing about Mesos, Aurora, or Marathon.

krenoten 11y ago
It's like many things that are declarative - you can only do what has been explicitly enabled in the declaration handling logic. This is good and bad. It's good because it restricts what is possible (protecting you slightly more from bad users) and can make learning and using the system easier. But you can't go beyond this. With a non-declarative scheduler you can implement far more interesting (for better or worse) management systems. One area where we've been seeing some interesting work happening is in stateful services running on top of schedulers. With something like postgres, there is interesting logic that must happen for responsible management beyond just throwing up 3 of them. You need to manage replication carefully. When a node goes down somewhere, you need to react somewhere else in a controlled way. With Mesos, you can write a scheduler that encodes your runbook. With kubernetes, you need a lot more out-of-band stuff.
- TheIronYuppie 11y ago
  Disclaimer: I work at Google on Kubernetes. You're correct that the scheduler in the box is declarative. However, it is easy (well, as easy as writing a framework) to swap out a scheduler that might be a better fit for what you're looking for. See this stackoverflow answer by one of the core Kubernetes team for more info: http://stackoverflow.com/questions/28857993/how-does-kubernetes-scheduler-work http://stackoverflow.com/questions/28857993/how-does-kuberne...

mkulke 11y ago

Yeah, the HA information was buried/non-existent in the docs until recently, I guess. We also ran into issues when replacing the master, b/c the kubelets where fixed to an ip and the nodes' configuration was basically immutable (provisioned by cloud-config). What worked for us: Accessing the the master by DNS and putting etcd on a persistent volume. This way we're able to replace the master within a DNS records ttl. As the api server is not a hard requirement for the workload, this is HA enough for us.