4 ms·

I’ve looked extensively at the documentation for gke autopilot (a system I use extensively) and haven’t found any documentation on how they orchestrate those cl

I’ve looked extensively at the documentation for gke autopilot (a system I use extensively) and haven’t found any documentation on how they orchestrate those clusters under the covers. I’ve always assumed it was a plugin to borg, not a different fleet orchestrator. Not to be overtly needy, but do you have a link to docs that contradicts that?

dilyevsky 3y ago
It is definitely not a plugin for borg. Borg is totally different api (source - i was borg sre). Afaik it’s actually vanilla k8s apiserver with some shimmed bespoke storage but it’s not really documented anywhere. You can test that fact using kubectl proxy though
verdverm 3y ago
- https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture https://cloud.google.com/kubernetes-engine/docs/concepts/clu... - https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview https://cloud.google.com/kubernetes-engine/docs/concepts/kub... Each of these has more peer pages that are pretty good too
- kasey_junk 3y ago
  You’ll note those documents are extremely careful in describing the control plane architecture to not promise you are running a stock k8s install. Which is why I’ve always assumed otherwise. But I’ll trust the sibling comment which suggests the only bespoke component in gke is storage well enough to leave it alone.
  - verdverm 3y ago
    Maybe you are reading it differently than me, but when they refer to the same binaries, as if I managed it myself, as being the pieces they use, it definitely seems like the open source project being used. > The control plane is the unified endpoint for your cluster. You interact with the control plane through Kubernetes API calls. The control plane runs the Kubernetes API server process (kube-apiserver) to handle API requests. > A node runs the services necessary to support the containers that make up your cluster's workloads. These include the runtime and the Kubernetes node agent (kubelet)
    - kasey_junk 3y ago
      We are well into the reeds of what doesn’t matter, only in that fly has given us a very under the covers look at their implementation that is hard to find with other alternatives. But as someone who has run K8s in other contexts I find the following to be pretty circumspect (not in a way that causes me concern, I’m a happy gke user) > GKE Autopilot manages the entire underlying infrastructure of clusters, including the control plane, nodes, and all system components. If you use GKE Standard mode, GKE manages the control plane and system components, and you manage the nodes. There is a mile of implementation detail in that. Which I’m happy for them to keep on their side of the street.
      - verdverm 3y ago
        > I’m a happy gke user && Which I’m happy for them to keep on their side of the street. 100% agreement iirc, there was a time where I thought they were doing some consolidation things with how they run the control plane, then at some point my cluster updates had a warning related to control plane unavailability during an update, this was on a single node cluster I get what you are saying though, there's probably some magic going on somewhere, but after many years on GKE, I don't really think about it.

dilyevsky 3y ago

It is definitely not a plugin for borg. Borg is totally different api (source - i was borg sre). Afaik it’s actually vanilla k8s apiserver with some shimmed bespoke storage but it’s not really documented anywhere. You can test that fact using kubectl proxy though

verdverm 3y ago

- https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture https://cloud.google.com/kubernetes-engine/docs/concepts/clu... - https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview https://cloud.google.com/kubernetes-engine/docs/concepts/kub... Each of these has more peer pages that are pretty good too

kasey_junk 3y ago
You’ll note those documents are extremely careful in describing the control plane architecture to not promise you are running a stock k8s install. Which is why I’ve always assumed otherwise. But I’ll trust the sibling comment which suggests the only bespoke component in gke is storage well enough to leave it alone.
- verdverm 3y ago
  Maybe you are reading it differently than me, but when they refer to the same binaries, as if I managed it myself, as being the pieces they use, it definitely seems like the open source project being used. > The control plane is the unified endpoint for your cluster. You interact with the control plane through Kubernetes API calls. The control plane runs the Kubernetes API server process (kube-apiserver) to handle API requests. > A node runs the services necessary to support the containers that make up your cluster's workloads. These include the runtime and the Kubernetes node agent (kubelet)
  - kasey_junk 3y ago
    We are well into the reeds of what doesn’t matter, only in that fly has given us a very under the covers look at their implementation that is hard to find with other alternatives. But as someone who has run K8s in other contexts I find the following to be pretty circumspect (not in a way that causes me concern, I’m a happy gke user) > GKE Autopilot manages the entire underlying infrastructure of clusters, including the control plane, nodes, and all system components. If you use GKE Standard mode, GKE manages the control plane and system components, and you manage the nodes. There is a mile of implementation detail in that. Which I’m happy for them to keep on their side of the street.
    - verdverm 3y ago
      > I’m a happy gke user && Which I’m happy for them to keep on their side of the street. 100% agreement iirc, there was a time where I thought they were doing some consolidation things with how they run the control plane, then at some point my cluster updates had a warning related to control plane unavailability during an update, this was on a single node cluster I get what you are saying though, there's probably some magic going on somewhere, but after many years on GKE, I don't really think about it.