5 ms·

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2

> “Sometimes when we do an etcd failover, the API server starts timing out requests until we restart it.” This is likely related a set of Kubernetes bugs [1][2] (and grpc[3]) that CoreOS is working diligently to get fixed. The first set of these, the endpoint reconciler[4], has landed in 1.9. More work is pending on the etcd client in Kubernetes. The good news is that the client is used everywhere, so one fix and all components will benefit. [1]: https://github.com/kubernetes/community/pull/939 https://github.com/kubernetes/community/pull/939 [2]: https://github.com/kubernetes/kubernetes/issues/22609 https://github.com/kubernetes/kubernetes/issues/22609 [3]: https://github.com/kubernetes/kubernetes/issues/47131 https://github.com/kubernetes/kubernetes/issues/47131 [4]: https://github.com/kubernetes/kubernetes/pull/51698 https://github.com/kubernetes/kubernetes/pull/51698

pishpash 9y ago
I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?
- ecnahc515 9y ago
  The concepts are based on similar ideas in Borg, but the things it shares with Borg ends there. The implementation is effectively entirely from scratch, so bugs will exist.
- rsanders 9y ago
  Kubernetes didn't come with all of the other Google infrastructure.
- alpb 9y ago
  Kubernetes takes some concepts from Borg. A system like Borg would be very closely coupled to Google‘s infrastructure that there’s probably very little to open source from there without open sourcing the entire machinery. Also, any large scale system like Borg developed at a large company like Facebook or Google will have completely opinionated one-way-of-doing-things for a lot of aspects. This doesn’t work for the world outside where lots of developers from different backgrounds, lots of projects with different requirements exist.
  - obeattie 9y ago
    I think this bit from "Borg, Omega, and Kubernetes"[1] (which is an excellent read) sheds light on this: > The Borgmaster is a monolithic component that knows the semantics of every API operation. It contains the cluster management logic such as the state machines for jobs, tasks, and machines; and it runs the Paxos-based replicated storage system used to record the master’s state. So it sounds as though Borg includes its own storage system. As I understand, Google has a set of (very complex) libraries written in C++ that implement Paxos/Multi-Paxos[2], which they have not open sourced. [1] https://research.google.com/pubs/pub44843.html https://research.google.com/pubs/pub44843.html [2] https://research.google.com/archive/paxos_made_live.html https://research.google.com/archive/paxos_made_live.html
- ShakataGaNai 9y ago
  IIRC from one of their talks.... K8s was supposed to be Borg 2.0 in many respects. They decided early on in development that it was a good tool and had lots of potential, but "fixing" Borg would be less work than replacing it. So k8s takes the Borg 2.0 concepts without being any of Borg code.

pishpash 9y ago

I don't get this. Didn't Kubernetes come out of Google Borg that had been in use forever? The second write should be more elegant and impressive -- why so many basic bugs?

ecnahc515 9y ago
The concepts are based on similar ideas in Borg, but the things it shares with Borg ends there. The implementation is effectively entirely from scratch, so bugs will exist.
rsanders 9y ago
Kubernetes didn't come with all of the other Google infrastructure.
alpb 9y ago
Kubernetes takes some concepts from Borg. A system like Borg would be very closely coupled to Google‘s infrastructure that there’s probably very little to open source from there without open sourcing the entire machinery. Also, any large scale system like Borg developed at a large company like Facebook or Google will have completely opinionated one-way-of-doing-things for a lot of aspects. This doesn’t work for the world outside where lots of developers from different backgrounds, lots of projects with different requirements exist.
- obeattie 9y ago
  I think this bit from "Borg, Omega, and Kubernetes"[1] (which is an excellent read) sheds light on this: > The Borgmaster is a monolithic component that knows the semantics of every API operation. It contains the cluster management logic such as the state machines for jobs, tasks, and machines; and it runs the Paxos-based replicated storage system used to record the master’s state. So it sounds as though Borg includes its own storage system. As I understand, Google has a set of (very complex) libraries written in C++ that implement Paxos/Multi-Paxos[2], which they have not open sourced. [1] https://research.google.com/pubs/pub44843.html https://research.google.com/pubs/pub44843.html [2] https://research.google.com/archive/paxos_made_live.html https://research.google.com/archive/paxos_made_live.html
ShakataGaNai 9y ago
IIRC from one of their talks.... K8s was supposed to be Borg 2.0 in many respects. They decided early on in development that it was a good tool and had lots of potential, but "fixing" Borg would be less work than replacing it. So k8s takes the Borg 2.0 concepts without being any of Borg code.