5 ms·

Gardener: Manage Kubernetes clusters across multiple cloud providers

87 points by rusht 8y ago

number6 8y ago
There is also rancher.com
- flurdy 8y ago
  And https://containership.io https://containership.io and Im sure many others
- jacques_chester 8y ago
  And BOSH, via Cloud Foundry Container Runtime[0]. BOSH is also used to deploy Cloud Foundry and a bunch of data services (RabbitMQ, Redis etc). It has the advantage of having been under continuous development for 7 years, with backends (CPIs) for most IaaSes being provided by the IaaSes themselves. I work for Pivotal, we sponsor most of the work done on BOSH. As it happens, SAP released an experimental BOSH CPI for Kubernetes[1] and there is work underway to make that experience smoother[2]. [0] https://docs-cfcr.cfapps.io/ https://docs-cfcr.cfapps.io/ [1] https://github.com/SAP/bosh-kubernetes-cpi-release https://github.com/SAP/bosh-kubernetes-cpi-release [2] https://www.dropbox.com/s/6jv9su650a76qmq/BOSH%20Kube%20CPI-20180401.pdf?dl=0 https://www.dropbox.com/s/6jv9su650a76qmq/BOSH%20Kube%20CPI-...
  - erikb 8y ago
    the first time I heard BOSH in a CF related youtube video. There I thought it would mean the company Bosch and was quite surprised how advanced this company was in container technology.
    - cmsd2 8y ago
      Possibly sightly obscure, but there's also an xmpp extension called BOSH[1] which is iirc long-polling applied to xmpp. [1] https://xmpp.org/extensions/xep-0124.html https://xmpp.org/extensions/xep-0124.html
  - vasu1124 8y ago
    That's all nice. The issue though is that BOSH has been completely leapfrogged by Kubernetes with its extensible API. Nowhere will BOSH ever get to the community reach and acceptance of the level of Kubernetes. That Boat has sailed. And with implementations of the machine specs [0,1,2,3] you get rid of the "media break" that you deem BOSH is filling (it is not). Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo? As for the bunch of data services, I guess it's only a matter of time until you see a cambrian explosion of productive operators. I mean this non-comprehensive list [4] is already impressive. SAP works in many projects where obligations with long term commitments have to be kept. And that is ok. BOSH CPI is one experiment to get CF on K8s. Have a look at the one which seems more attainable [6]. But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? [5] I work for SAP in the inter-junction of SaaS, PaaS & IaaS and K8s. [0] https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api [1] https://github.com/kube-node/nodeset https://github.com/kube-node/nodeset [2] https://github.com/gardener/machine-controller-manager https://github.com/gardener/machine-controller-manager [3] https://github.com/kubeup/archon https://github.com/kubeup/archon [4] https://github.com/operator-framework/awesome-operators https://github.com/operator-framework/awesome-operators [5] https://www.youtube.com/watch?v=4ow7IumxkOM https://www.youtube.com/watch?v=4ow7IumxkOM [6] https://docs.google.com/document/d/1qs6UQQDWMkfOpY19XqS3CfvI00jCns876TjplJ6E95s/edit https://docs.google.com/document/d/1qs6UQQDWMkfOpY19XqS3CfvI...
    - jacques_chester 8y ago
      It may surprise you to learn that I disagree about your core thesis. My qualifications are working for Pivotal at the inter-junction of PaaS, CaaS and FaaS. IaaS is just a hobby. > Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo? This is being investigated too. The main difficulty (as Brendan Burns has noted for virtual kubelet) has been that Kubernetes, ostensibly providing a smooth abstraction away from machines, actually has layer-breaking assumptions about the existence machines after all. Cloud Foundry always had BOSH to insulate it from that concern. But BOSH-on-K8s was not super pretty in the early days, because they had overlapping concerns (mostly disks, I believe). Kubernetes-on-BOSH is a natural fit. Standing up large distributed systems on IaaSes is BOSH's bread and butter. More to the point, that is its sole focus. Its mission is not spread amongst a cambrian explosion of alternatives (almost all of whom, you may recall, went extinct). But in any case, it's doable. Which has the nice property that as the cluster API matures, BOSH will happily take workloads that run on VMs and run them on pods. The same experience we have today -- run an upgrade, everything is upgraded, nobody bats an eye -- will be exactly the same. > But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? If you look at the community activity, the answer is pretty clearly yes: Diego can be placed behind an OPI (Project Eirini) and CF itself can run control-plane components in containers instead of VMs. Personally I am all for it. But as you pointed out, enterprise vendors need to keep their word. Adopting Kubernetes isn't a button-press operation. We need to prove that CF-on-K8s is at least as safe and performant as CF-on-Diego has been. Your customers, and Pivotal's customers, and IBM's customers, and SUSE's customers, expect all of us to provide the roadmap and prove that it is something they can bet a company on.
  - vlerenc 8y ago
    As a 2y+ Bosh user and short to 2y Kubernetes user, I can only second here, that Kubernetes has outpaced Bosh. Don't get me wrong, while I would always prefer Bosh over Chef or Ansible, Kubernetes is simply the better deployment underlay. I mean, either I have to deploy Bosh or Kubernetes (to deploy more Kubernetes clusters, because that's what we are speaking of here). Kubernetes is now commodity. The idea is to place Kubernetes control planes into Kubernetes clusters (like it became good practice to deploy the OpenStack control planes) and benefit from all the advantages Kubernetes brings along. No need to go for another tool, but there are more points: - First off, Bosh is no thing in the Kubernetes community – it is primarily a tool used in the context of Cloud Foundry. - By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. - Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. - VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. - Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA. - Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. - Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). - Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. - Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS). - Bosh packages all software into proprietary Bosh Releases and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers. The list goes on and on and it is unlikely that Bosh can catch up again (the Pivotal IPO statement seems to indicate the same as major risk [1]). Kubernetes is the new virtualization layer and is made to run software reliably with minimal TCO. I have seen this in practice - Kubernetes is absolutely amazing - also/especially for these types of workloads. [1] https://techcrunch.com/2018/03/23/pivotal-software-files-for-ipo/ https://techcrunch.com/2018/03/23/pivotal-software-files-for...
- HerrMonnezza 8y ago
  And http://elasticluster.readthedocs.io/en/latest/playbooks.html#kubernetes http://elasticluster.readthedocs.io/en/latest/playbooks.html... (disclaimer: I work on it)
bauerd 8y ago
Letter spacing makes this painful to read …
rusht 8y ago
Here’s the blog post: https://kubernetes.io/blog/2018/05/17/gardener/ https://kubernetes.io/blog/2018/05/17/gardener/
msohn 8y ago
video recording of the Gardener demo at the Kubernetes community meeting 2018-05-17: https://www.youtube.com/watch?v=DpFTcTnBxbM&t=27m0s https://www.youtube.com/watch?v=DpFTcTnBxbM&t=27m0s
pietromenna 8y ago
I used Gardner to create k8s clusters for testing, I found it really easy to use and just have the configurations needed.
madmulita 8y ago
I don't mean to hijack the thread, sincere concern. I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this kind of infrastructure. It's not clear if we are allowed to use the public cloud yet. (We are a bank, yeah, I know) I've tried, at least: - kubeadm - rancher - canonical kubernetes - canonical kubernetes core - some random internet recipes And for some IaaS: - cloudfoundry - openstack - cloudstack - opennebula - ganeti Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc. I've been able to hammer some of this solutions until the cluster started and had some pods or VMs running, but it feels like this are not ready for production or not for 'secure' on-premise deployment.
- cfontes 8y ago
  https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic It just works...
  - madmulita 8y ago
    I have one Firefox tab opened with that repo, haven't tried it yet in part because I'm kind of loosing hope. Starting to believe we'll have to build something ourselves.
    - bryanlarsen 8y ago
      Yes, talking to the Apprenda guys I got the impression that's their specialty: air-gapped and otherwise isolated on-premises clusters. I'm sure they'd love to get you on a support contract.
      - ikester 8y ago
        I work for Apprenda. Sure, support contracts are an option but the Kismatic Enterprise Toolkit (KET) is completely Open Source and free. And you can get community support via the Slack channel: kismatic.slack.com
donttrack 8y ago
I did this last year by using wireguard VPN between all nodes. It worked quite good I would say. It was for a proof of concept project, so I don’t know how it would perform in production but it was promising.
deleted 8y ago
[deleted]

87 points by rusht 8y ago

number6 8y ago

There is also rancher.com

flurdy 8y ago
And https://containership.io https://containership.io and Im sure many others
jacques_chester 8y ago
And BOSH, via Cloud Foundry Container Runtime[0]. BOSH is also used to deploy Cloud Foundry and a bunch of data services (RabbitMQ, Redis etc). It has the advantage of having been under continuous development for 7 years, with backends (CPIs) for most IaaSes being provided by the IaaSes themselves. I work for Pivotal, we sponsor most of the work done on BOSH. As it happens, SAP released an experimental BOSH CPI for Kubernetes[1] and there is work underway to make that experience smoother[2]. [0] https://docs-cfcr.cfapps.io/ https://docs-cfcr.cfapps.io/ [1] https://github.com/SAP/bosh-kubernetes-cpi-release https://github.com/SAP/bosh-kubernetes-cpi-release [2] https://www.dropbox.com/s/6jv9su650a76qmq/BOSH%20Kube%20CPI-20180401.pdf?dl=0 https://www.dropbox.com/s/6jv9su650a76qmq/BOSH%20Kube%20CPI-...
- erikb 8y ago
  the first time I heard BOSH in a CF related youtube video. There I thought it would mean the company Bosch and was quite surprised how advanced this company was in container technology.
  - cmsd2 8y ago
    Possibly sightly obscure, but there's also an xmpp extension called BOSH[1] which is iirc long-polling applied to xmpp. [1] https://xmpp.org/extensions/xep-0124.html https://xmpp.org/extensions/xep-0124.html
- vasu1124 8y ago
  That's all nice. The issue though is that BOSH has been completely leapfrogged by Kubernetes with its extensible API. Nowhere will BOSH ever get to the community reach and acceptance of the level of Kubernetes. That Boat has sailed. And with implementations of the machine specs [0,1,2,3] you get rid of the "media break" that you deem BOSH is filling (it is not). Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo? As for the bunch of data services, I guess it's only a matter of time until you see a cambrian explosion of productive operators. I mean this non-comprehensive list [4] is already impressive. SAP works in many projects where obligations with long term commitments have to be kept. And that is ok. BOSH CPI is one experiment to get CF on K8s. Have a look at the one which seems more attainable [6]. But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? [5] I work for SAP in the inter-junction of SaaS, PaaS & IaaS and K8s. [0] https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api [1] https://github.com/kube-node/nodeset https://github.com/kube-node/nodeset [2] https://github.com/gardener/machine-controller-manager https://github.com/gardener/machine-controller-manager [3] https://github.com/kubeup/archon https://github.com/kubeup/archon [4] https://github.com/operator-framework/awesome-operators https://github.com/operator-framework/awesome-operators [5] https://www.youtube.com/watch?v=4ow7IumxkOM https://www.youtube.com/watch?v=4ow7IumxkOM [6] https://docs.google.com/document/d/1qs6UQQDWMkfOpY19XqS3CfvI00jCns876TjplJ6E95s/edit https://docs.google.com/document/d/1qs6UQQDWMkfOpY19XqS3CfvI...
  - jacques_chester 8y ago
    It may surprise you to learn that I disagree about your core thesis. My qualifications are working for Pivotal at the inter-junction of PaaS, CaaS and FaaS. IaaS is just a hobby. > Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo? This is being investigated too. The main difficulty (as Brendan Burns has noted for virtual kubelet) has been that Kubernetes, ostensibly providing a smooth abstraction away from machines, actually has layer-breaking assumptions about the existence machines after all. Cloud Foundry always had BOSH to insulate it from that concern. But BOSH-on-K8s was not super pretty in the early days, because they had overlapping concerns (mostly disks, I believe). Kubernetes-on-BOSH is a natural fit. Standing up large distributed systems on IaaSes is BOSH's bread and butter. More to the point, that is its sole focus. Its mission is not spread amongst a cambrian explosion of alternatives (almost all of whom, you may recall, went extinct). But in any case, it's doable. Which has the nice property that as the cluster API matures, BOSH will happily take workloads that run on VMs and run them on pods. The same experience we have today -- run an upgrade, everything is upgraded, nobody bats an eye -- will be exactly the same. > But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? If you look at the community activity, the answer is pretty clearly yes: Diego can be placed behind an OPI (Project Eirini) and CF itself can run control-plane components in containers instead of VMs. Personally I am all for it. But as you pointed out, enterprise vendors need to keep their word. Adopting Kubernetes isn't a button-press operation. We need to prove that CF-on-K8s is at least as safe and performant as CF-on-Diego has been. Your customers, and Pivotal's customers, and IBM's customers, and SUSE's customers, expect all of us to provide the roadmap and prove that it is something they can bet a company on.
- vlerenc 8y ago
  As a 2y+ Bosh user and short to 2y Kubernetes user, I can only second here, that Kubernetes has outpaced Bosh. Don't get me wrong, while I would always prefer Bosh over Chef or Ansible, Kubernetes is simply the better deployment underlay. I mean, either I have to deploy Bosh or Kubernetes (to deploy more Kubernetes clusters, because that's what we are speaking of here). Kubernetes is now commodity. The idea is to place Kubernetes control planes into Kubernetes clusters (like it became good practice to deploy the OpenStack control planes) and benefit from all the advantages Kubernetes brings along. No need to go for another tool, but there are more points: - First off, Bosh is no thing in the Kubernetes community – it is primarily a tool used in the context of Cloud Foundry. - By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. - Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. - VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. - Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA. - Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. - Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). - Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. - Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS). - Bosh packages all software into proprietary Bosh Releases and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers. The list goes on and on and it is unlikely that Bosh can catch up again (the Pivotal IPO statement seems to indicate the same as major risk [1]). Kubernetes is the new virtualization layer and is made to run software reliably with minimal TCO. I have seen this in practice - Kubernetes is absolutely amazing - also/especially for these types of workloads. [1] https://techcrunch.com/2018/03/23/pivotal-software-files-for-ipo/ https://techcrunch.com/2018/03/23/pivotal-software-files-for...
HerrMonnezza 8y ago
And http://elasticluster.readthedocs.io/en/latest/playbooks.html#kubernetes http://elasticluster.readthedocs.io/en/latest/playbooks.html... (disclaimer: I work on it)

bauerd 8y ago

Letter spacing makes this painful to read …

rusht 8y ago

Here’s the blog post: https://kubernetes.io/blog/2018/05/17/gardener/ https://kubernetes.io/blog/2018/05/17/gardener/

msohn 8y ago

video recording of the Gardener demo at the Kubernetes community meeting 2018-05-17: https://www.youtube.com/watch?v=DpFTcTnBxbM&t=27m0s https://www.youtube.com/watch?v=DpFTcTnBxbM&t=27m0s

pietromenna 8y ago

I used Gardner to create k8s clusters for testing, I found it really easy to use and just have the configurations needed.

madmulita 8y ago

I don't mean to hijack the thread, sincere concern. I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this kind of infrastructure. It's not clear if we are allowed to use the public cloud yet. (We are a bank, yeah, I know) I've tried, at least: - kubeadm - rancher - canonical kubernetes - canonical kubernetes core - some random internet recipes And for some IaaS: - cloudfoundry - openstack - cloudstack - opennebula - ganeti Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc. I've been able to hammer some of this solutions until the cluster started and had some pods or VMs running, but it feels like this are not ready for production or not for 'secure' on-premise deployment.

cfontes 8y ago
https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic It just works...
- madmulita 8y ago
  I have one Firefox tab opened with that repo, haven't tried it yet in part because I'm kind of loosing hope. Starting to believe we'll have to build something ourselves.
  - bryanlarsen 8y ago
    Yes, talking to the Apprenda guys I got the impression that's their specialty: air-gapped and otherwise isolated on-premises clusters. I'm sure they'd love to get you on a support contract.
    - ikester 8y ago
      I work for Apprenda. Sure, support contracts are an option but the Kismatic Enterprise Toolkit (KET) is completely Open Source and free. And you can get community support via the Slack channel: kismatic.slack.com

donttrack 8y ago

I did this last year by using wireguard VPN between all nodes. It worked quite good I would say. It was for a proof of concept project, so I don’t know how it would perform in production but it was promising.

deleted 8y ago

[deleted]