5 ms·

As a 2y+ Bosh user and short to 2y Kubernetes user, I can only second here, that Kubernetes has outpaced Bosh. Don't get me wrong, while I would always prefer B

As a 2y+ Bosh user and short to 2y Kubernetes user, I can only second here, that Kubernetes has outpaced Bosh. Don't get me wrong, while I would always prefer Bosh over Chef or Ansible, Kubernetes is simply the better deployment underlay. I mean, either I have to deploy Bosh or Kubernetes (to deploy more Kubernetes clusters, because that's what we are speaking of here). Kubernetes is now commodity. The idea is to place Kubernetes control planes into Kubernetes clusters (like it became good practice to deploy the OpenStack control planes) and benefit from all the advantages Kubernetes brings along. No need to go for another tool, but there are more points: - First off, Bosh is no thing in the Kubernetes community – it is primarily a tool used in the context of Cloud Foundry. - By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. - Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. - VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. - Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA. - Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. - Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). - Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. - Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS). - Bosh packages all software into proprietary Bosh Releases and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers. The list goes on and on and it is unlikely that Bosh can catch up again (the Pivotal IPO statement seems to indicate the same as major risk [1]). Kubernetes is the new virtualization layer and is made to run software reliably with minimal TCO. I have seen this in practice - Kubernetes is absolutely amazing - also/especially for these types of workloads. [1] https://techcrunch.com/2018/03/23/pivotal-software-files-for-ipo/ https://techcrunch.com/2018/03/23/pivotal-software-files-for...

jacques_chester 8y ago
As a 4 year BOSH user and 1 year Kubernetes user, I have to pick some nits. > By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. If you're running Kubernetes on-prem, you still need to solve how to manage the Kubernetes control plane. Kubernetes-on-Kubernetes doesn't make that vanish. > Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. BOSH provisions anything for which there is a CPI. Virtual machines, hardware, Kubernetes containers. Some folks will go directly to virtual kubelets. A lot of folks won't. For on-prem they will need to solve the bootstrapping problem to get to the first install of Kubernetes. Even GKE, pretty much the gold standard public service, is still creating VMs for you. > VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. And you've never over or under provisioned a container? > Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA BOSH is a purely operator tool. Kubernetes is not very clear about where it wants to draw the boundaries. At CF Summit a consistent theme was "we ought to use Kubernetes under the hood". At Kubecon a consistent theme was "hmmm ... maybe we should have a hood". Scaling apps is up to whomever cares about them. On Kubernetes that's HPA or VPA or another autoscaler (I happen to work on an autoscaler that is for Kubernetes). On Cloud Foundry it's one of the several app autoscalers you can install. > Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. Nobody wanted it, to be frank. Now they do and folks from Pivotal, IBM and I think SAP are working on it (as well as the cluster API). > Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). Kubernetes cannot scale its components. It needs an outside tool to manage its own components. A tool like BOSH, for example. Or, in GKE, the cluster autoscaler. Which, again, spins up VMs, not containers. > Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. BOSH monitors both processes and VMs. Monit is being aggressively retired in favour of BOSH Process Manager. > Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS) This goes back to my point that Kubernetes has to run on something, and that it would be nice if that something is easy to manage. One way to do that is to totally relieve the operator of unnecessary toil. The only way to ensure you have a consistent machine image is to ... have a consistent machine. Red Hat agree, which is why OpenShift has always run on RHEL. So if "lockin" means "totally opensource Linux distribution with profitable vendors who rapidly backport fixes for years in very short timeframes", you're locked in by everyone. Our experience is that operators like the stemcell system. It's been a competitive advantage for us. Red Hat's purchase of CoreOS -- which is much more BOSH-like than Ansible -- is not a coincidence. > Bosh packages all software into proprietary Bosh Releases This is flatly wrong. Releases are opensource. By default they're published to bosh.io and rely on having a public git repository. > and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers. Or you tell your deployment to spread them across machines. On IaaSes that means different VMs. On Kubernetes it will mean different containers. > The list goes on and on and it is unlikely that Bosh can catch up again BOSH isn't trying to be Kubernetes, so there's no "catching up". It's a different tool that serves a different audience with laser focus. I might as well claim that Kubernetes "can't catch up" with BOSH. It's because we listen to our customers that Pivotal has PKS. It's also because we listen to our customers that PKS uses BOSH for the compute, network and storage substrate. They are different tools and they solve different problems. As I see the trends, Kubernetes will take over most of the workloads which BOSH is used for now. It's much easier to package a Helm chart than to construct a BOSH release, so it's only sensible to target the Helm chart. An increasing amount of control-plane components will be able to hoist themselves into containers, which reduces the things BOSH will supervise. At some point virtual kubelets will become the de facto compute API, which means that CPIs are no longer the primary abstraction layer. Disclosure: I work for Pivotal.

jacques_chester 8y ago

As a 4 year BOSH user and 1 year Kubernetes user, I have to pick some nits. > By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. If you're running Kubernetes on-prem, you still need to solve how to manage the Kubernetes control plane. Kubernetes-on-Kubernetes doesn't make that vanish. > Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. BOSH provisions anything for which there is a CPI. Virtual machines, hardware, Kubernetes containers. Some folks will go directly to virtual kubelets. A lot of folks won't. For on-prem they will need to solve the bootstrapping problem to get to the first install of Kubernetes. Even GKE, pretty much the gold standard public service, is still creating VMs for you. > VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. And you've never over or under provisioned a container? > Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA BOSH is a purely operator tool. Kubernetes is not very clear about where it wants to draw the boundaries. At CF Summit a consistent theme was "we ought to use Kubernetes under the hood". At Kubecon a consistent theme was "hmmm ... maybe we should have a hood". Scaling apps is up to whomever cares about them. On Kubernetes that's HPA or VPA or another autoscaler (I happen to work on an autoscaler that is for Kubernetes). On Cloud Foundry it's one of the several app autoscalers you can install. > Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. Nobody wanted it, to be frank. Now they do and folks from Pivotal, IBM and I think SAP are working on it (as well as the cluster API). > Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). Kubernetes cannot scale its components. It needs an outside tool to manage its own components. A tool like BOSH, for example. Or, in GKE, the cluster autoscaler. Which, again, spins up VMs, not containers. > Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. BOSH monitors both processes and VMs. Monit is being aggressively retired in favour of BOSH Process Manager. > Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS) This goes back to my point that Kubernetes has to run on something, and that it would be nice if that something is easy to manage. One way to do that is to totally relieve the operator of unnecessary toil. The only way to ensure you have a consistent machine image is to ... have a consistent machine. Red Hat agree, which is why OpenShift has always run on RHEL. So if "lockin" means "totally opensource Linux distribution with profitable vendors who rapidly backport fixes for years in very short timeframes", you're locked in by everyone. Our experience is that operators like the stemcell system. It's been a competitive advantage for us. Red Hat's purchase of CoreOS -- which is much more BOSH-like than Ansible -- is not a coincidence. > Bosh packages all software into proprietary Bosh Releases This is flatly wrong. Releases are opensource. By default they're published to bosh.io and rely on having a public git repository. > and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers. Or you tell your deployment to spread them across machines. On IaaSes that means different VMs. On Kubernetes it will mean different containers. > The list goes on and on and it is unlikely that Bosh can catch up again BOSH isn't trying to be Kubernetes, so there's no "catching up". It's a different tool that serves a different audience with laser focus. I might as well claim that Kubernetes "can't catch up" with BOSH. It's because we listen to our customers that Pivotal has PKS. It's also because we listen to our customers that PKS uses BOSH for the compute, network and storage substrate. They are different tools and they solve different problems. As I see the trends, Kubernetes will take over most of the workloads which BOSH is used for now. It's much easier to package a Helm chart than to construct a BOSH release, so it's only sensible to target the Helm chart. An increasing amount of control-plane components will be able to hoist themselves into containers, which reduces the things BOSH will supervise. At some point virtual kubelets will become the de facto compute API, which means that CPIs are no longer the primary abstraction layer. Disclosure: I work for Pivotal.