7 ms·

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

by superbcarrot 6y ago

bo0tzz 6y ago
https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api
- seneca 6y ago
  CAPI is for creating clusters only. It doesn't offer any orchestration.
  - moondev 6y ago
    I'm curious what you mean here, since CAPI allows you to manage and scale clusters like any other Kubernetes object. You can literally "kubectl scale --replicas" a cluster control-plane and worker nodes.
bcrosby95 6y ago
A Kubernetes for Kubernetes! But what about when we want to upgrade that?
- dabeeeenster 6y ago
  I mean thats simply Kubernetes for Kubernetes for Kubernetes. Lets go!
  - testplzignore 6y ago
    I need a Kubernetes Kubernetes Kubernetes Kubernetes Kubernetes emulator so I can run Civilization 14 on my iPhone 35XSXS.
    - deleted 6y ago
      [deleted]
  - higeorge13 6y ago
    Inception!
- deleted 6y ago
  [deleted]
- verst 6y ago
  In a way this reminds me of Kubernetes Virtual Clusters. Each virtual cluster has its own tenant control plane, namespaces. Multiple virtual clusters exist in a super cluster. https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster https://github.com/kubernetes-sigs/multi-tenancy/tree/master...
- tutfbhuf 6y ago
  I think at a certain point up in the stack you don't want to have another orchestrator for the orchestrator. You would just tell each team to keep their cluster up-to-date without creating a (dangerous) master button to update them all at once.
- durnygbur 6y ago
  To ugrade it you summon Scrum of Scrums, the Master Scrum Master has all the answers you might need.
- rwoerz 6y ago
  Wake me up when I can finally run a bare-metal Kubernetes cluster in Kubernetes to get rid of all my hardware.
thirdlamp 6y ago
it's kubernetes all the way down
codeulike 6y ago
Kubermeta
mullingitover 6y ago
Rancher does this nicely.
YawningAngel 6y ago
You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region
- WJW 6y ago
  Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.
  - tutfbhuf 6y ago
    While it is certainly possible to have mixed region clusters, from my experience it is better to have one cluster per region. Do your synchronous communication within the cluster / within the datacenter with sub millisecond latencies and do asynchronous communication across cluster boundaries with up to triple digit millisecond latencies.
verdverm 6y ago
KubeFed - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
theptip 6y ago
It used to be called Ubernetes, shame that name didn't stick.
jknoepfler 6y ago
I operate one at work, actually. It's not so bad.
yongjik 6y ago
> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare
- ex_amazon_sde 6y ago
  Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)
  - paulryanrogers 6y ago
    You mean the infra behind non-AWS services like retail Amazon.com? Or the infra underlying AWS itself?
    - ex_amazon_sde 6y ago
      The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.
      - paulryanrogers 6y ago
        Fascinating because I would have thought the whole idea is AWS is dog-fooded by retail. Yet in truth AWS is itself too expensive and / or complex for Amazon itself! (Or at least the original part of it.)
        ex_amazon_sde 6y ago
        The majority of retail is dog-fooding AWS, but a lot of code was written before AWS existed or before it provided many of the required services.
  - 6y ago
Aeolun 6y ago
I think we’re already there? Maybe we’re better off looking for a cluster orchestrator orchestrator.
Turbots 6y ago
What, like the [Cluster API](https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/), [RedHat Advanced Cluster Mgmt](https://www.redhat.com/en/resources/advanced-cluster-management-FAQ https://www.redhat.com/en/resources/advanced-cluster-managem...), [Rancher](https://rancher.com/ https://rancher.com/), [VMware Tanzu Mission Control](https://tanzu.vmware.com/mission-control https://tanzu.vmware.com/mission-control)? Most large enterprises that have been using Kubernetes for a while, have many dozens or hundreds of Kubernetes clusters running. Clusters for different Business Groups, environments or clouds. Something needs to manage User Access, Network policies, storage policies/quotas, Security Policies across those clusters. Something needs to provision, scale, patch and upgrade those clusters in an automated way. There are 4 major releases of Kubernetes a year, probably 20-30 minor releases and patch versions. Enterprises need to keep those up to date. Terraform, Ansible or Chef/Puppet won't solve that problem without spending hundreds of mandays per year on it.
deleted 6y ago
[deleted]
seneca 6y ago
That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.
- p_l 6y ago
  Federation v2 is nice, but not many places use it. Unfortunately, all my clusters are either confined locally anyway, or are deliberately supposed to be separated in all ways :(
6nf 6y ago
Agreed, this is absolute insanity and people can't see it from the weeds.
jachee 6y ago
Isn't that basically what helmfile[0] does? It lets you declaratively define helm deployments, which in turn orchestrate k8s infrastructure. 0 - https://github.com/roboll/helmfile https://github.com/roboll/helmfile
nyellin 6y ago
Kubernetes solves a problem. It isn't appropriate for all use cases, but the real problem isn't kubernetes but rather the knowledge gap around new technology.

Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?

bo0tzz 6y ago

https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api

seneca 6y ago
CAPI is for creating clusters only. It doesn't offer any orchestration.
- moondev 6y ago
  I'm curious what you mean here, since CAPI allows you to manage and scale clusters like any other Kubernetes object. You can literally "kubectl scale --replicas" a cluster control-plane and worker nodes.

bcrosby95 6y ago

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

dabeeeenster 6y ago
I mean thats simply Kubernetes for Kubernetes for Kubernetes. Lets go!
- testplzignore 6y ago
  I need a Kubernetes Kubernetes Kubernetes Kubernetes Kubernetes emulator so I can run Civilization 14 on my iPhone 35XSXS.
  - deleted 6y ago
    [deleted]
- higeorge13 6y ago
  Inception!
deleted 6y ago
[deleted]
verst 6y ago
In a way this reminds me of Kubernetes Virtual Clusters. Each virtual cluster has its own tenant control plane, namespaces. Multiple virtual clusters exist in a super cluster. https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster https://github.com/kubernetes-sigs/multi-tenancy/tree/master...
tutfbhuf 6y ago
I think at a certain point up in the stack you don't want to have another orchestrator for the orchestrator. You would just tell each team to keep their cluster up-to-date without creating a (dangerous) master button to update them all at once.
durnygbur 6y ago
To ugrade it you summon Scrum of Scrums, the Master Scrum Master has all the answers you might need.
rwoerz 6y ago
Wake me up when I can finally run a bare-metal Kubernetes cluster in Kubernetes to get rid of all my hardware.

thirdlamp 6y ago

it's kubernetes all the way down

codeulike 6y ago

Kubermeta

mullingitover 6y ago

Rancher does this nicely.

YawningAngel 6y ago

You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region

WJW 6y ago
Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.
- tutfbhuf 6y ago
  While it is certainly possible to have mixed region clusters, from my experience it is better to have one cluster per region. Do your synchronous communication within the cluster / within the datacenter with sub millisecond latencies and do asynchronous communication across cluster boundaries with up to triple digit millisecond latencies.

verdverm 6y ago

KubeFed - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed

theptip 6y ago

It used to be called Ubernetes, shame that name didn't stick.

jknoepfler 6y ago

I operate one at work, actually. It's not so bad.

yongjik 6y ago

> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare

ex_amazon_sde 6y ago
Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)
- paulryanrogers 6y ago
  You mean the infra behind non-AWS services like retail Amazon.com? Or the infra underlying AWS itself?
  - ex_amazon_sde 6y ago
    The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.
    - paulryanrogers 6y ago
      Fascinating because I would have thought the whole idea is AWS is dog-fooded by retail. Yet in truth AWS is itself too expensive and / or complex for Amazon itself! (Or at least the original part of it.)
      - ex_amazon_sde 6y ago
        The majority of retail is dog-fooding AWS, but a lot of code was written before AWS existed or before it provided many of the required services.
- 6y ago

Aeolun 6y ago

I think we’re already there? Maybe we’re better off looking for a cluster orchestrator orchestrator.

Turbots 6y ago

What, like the [Cluster API](https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/), [RedHat Advanced Cluster Mgmt](https://www.redhat.com/en/resources/advanced-cluster-management-FAQ https://www.redhat.com/en/resources/advanced-cluster-managem...), [Rancher](https://rancher.com/ https://rancher.com/), [VMware Tanzu Mission Control](https://tanzu.vmware.com/mission-control https://tanzu.vmware.com/mission-control)? Most large enterprises that have been using Kubernetes for a while, have many dozens or hundreds of Kubernetes clusters running. Clusters for different Business Groups, environments or clouds. Something needs to manage User Access, Network policies, storage policies/quotas, Security Policies across those clusters. Something needs to provision, scale, patch and upgrade those clusters in an automated way. There are 4 major releases of Kubernetes a year, probably 20-30 minor releases and patch versions. Enterprises need to keep those up to date. Terraform, Ansible or Chef/Puppet won't solve that problem without spending hundreds of mandays per year on it.

deleted 6y ago

[deleted]

seneca 6y ago

That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.

p_l 6y ago
Federation v2 is nice, but not many places use it. Unfortunately, all my clusters are either confined locally anyway, or are deliberately supposed to be separated in all ways :(

6nf 6y ago

Agreed, this is absolute insanity and people can't see it from the weeds.

jachee 6y ago

Isn't that basically what helmfile[0] does? It lets you declaratively define helm deployments, which in turn orchestrate k8s infrastructure. 0 - https://github.com/roboll/helmfile https://github.com/roboll/helmfile

nyellin 6y ago

Kubernetes solves a problem. It isn't appropriate for all use cases, but the real problem isn't kubernetes but rather the knowledge gap around new technology.