5 ms·

Has anyone already successfully created a HA-cluster on a bare metal infrastructure? It seems overly complicated and not that well documented from what I can se

by jstoja 10y ago

Has anyone already successfully created a HA-cluster on a bare metal infrastructure? It seems overly complicated and not that well documented from what I can see.

hnarayanan 10y ago
I have attempted this using https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare... . One can get quite far, but I am not sure it is worth the effort. I understand this is a vague analogy, but it feels like you're trying to setup email infrastructure when what you really want is to send email. It is a lot easier on the public cloud, or easier still on a managed service.
- iagooar 10y ago
  What about people who cannot go to the cloud? What about people who need more performance? Lower costs? Kubernetes seems to be such a great abstraction for the underlying hardware, why not use it where it is needed the most?
  - jdoliner 10y ago
    People do run Kubernetes on bare metal... but there's no denying it takes effort. That effort should be coming down over time, but unfortunately there's still a ways to go.
    - AlexB138 10y ago
      That's not entirely true. I attend meet-ups with engineers from a fairly large company whose name you would recognize who run K8s on bare metal. I know that's vague, but I don't know how public their infrastructure is, so I'd rather not speak out of turn. You're absolutely right that it takes some effort.
      - SEJeff 10y ago
        Bloomberg runs Kubernetes on bare metal and one of my friends is an engineer on the project (but did not speak at KubeCon about it). They've got a relatively sizable install as well.
  - nixgeek 10y ago
    citation required, since there are IaaS offerings that do not involve a hypervisor (e.g. Rackspace OnMetal) and therefore its possible to move "to the cloud" without performance penalties. on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
    - manigandham 10y ago
      Citation needed for what? its common knowledge that the major clouds VMs with lower performance and higher cost in exchange for flexibility.
    - vidarh 10y ago
      > and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own. You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same. And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo. For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
    - iagooar 10y ago
      citation required for what? For stating, that it is complicated to install on bare-metal and that it's been a priority for Kubernetes to be deploying on cloud servers?
- jabzd 10y ago
  I felt very similar. Especially thinking about it from a disaster recovery/re-bootstrapping perspective. That scared me into using docker swarm kit which is stupidly easy to bootstrap. Keep in mind that's not necessarily a like for like replacement but it was enough for my use case. I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
- philips 10y ago
  What would your ideal setup look like on baremetal? The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months. [1] https://github.com/kubernetes-incubator/bootkube https://github.com/kubernetes-incubator/bootkube [2] http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/
vlad00 10y ago
I've deployed kubernetes on aws as if it were bare metal. Try to run through https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way translating what they do there to your particular setup(configure firewall, add routes to router, etc)
- amatix 10y ago
  I'd suggest looking into kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) for Kubernetes on AWS, whether you let it take the actions or you tweak/use the Terraform config it can generate
appelza 10y ago
Yeah, quite easily. Look at Openshift, ansible playbook installer for Fedora 24. You can configure 2 masters/ha/etc in the playbook.
olavgg 10y ago
Could you use Canonical MAAS (https://www.ubuntu.com/server/maas https://www.ubuntu.com/server/maas) for this?
- jcastro 10y ago
  Indeed, we fully support Kubernetes on MAAS on Ubuntu. Currently for 1.4 but we'll be updating support for 1.5 over the next week or so. Disclaimer: I'm on the team that works on this.
ownagefool 10y ago
Yep. We PXE boot coreos and include a cloud-config that bootstraps a kubelet and pulls master manifests. It's actually not very complicated at all.
sandGorgon 10y ago
+1 there has been a discussion of making a sig-on-premise/sig-bare-metal (with a fair bit of opposition from others) [1]. Would help to have your comments there as well. The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/17 https://github.com/kubernetes/ingress/issues/17 The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip. a lot of this is taken care for the cloud...but NOT for bare metal. [1] https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/ztVnvbrpTK8/-R4PlfHVBwAJ https://groups.google.com/forum/m/?utm_medium=email&utm_sour...
- jcastro 10y ago
  sig-onprem is coming, it was decided at the last community meeting, expect an announcement soonish.
- elcct 10y ago
  It also has problems with using certificates when reporting etcd health: https://github.com/kubernetes/kubernetes/issues/27343 https://github.com/kubernetes/kubernetes/issues/27343 I was writing a blog about installing it on bare metal, but this issue got me blocked.
  - sandGorgon 10y ago
    could you ping on slack on sig-cluster-ops? thats the defacto meeting ground for bare metal issues. Though honestly i think it is a fundamental issue and not specifically related to BM.
tyingq 10y ago
It won't help you understand how to install it, but if you just want to get it done, see http://stackpoint.io http://stackpoint.io They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean. Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.
- irontoby 10y ago
  Stackpoint user here; running a very small (3 node) cluster on DigitalOcean currently using their service. It's been relatively painless so far, but I'm pretty new to k8s so my needs are very basic.
hanikesn 10y ago
We're using Puppet to deploy our kubernetes cluster on bare-metal as well as on AWS. Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin. We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB. Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.
fen4o 10y ago
PXE CoreOS and cloud-config to run kubelet and pull manifests. [0] [0] https://youtu.be/4gyeixJLabo?t=814 https://youtu.be/4gyeixJLabo?t=814
old-gregg 10y ago
Regarding HA-on-bare metal, we do this for our customers here: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
jwelch55 10y ago
I'd suggest checking out KET (Kismatic Enterprise Toolkit) - https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic There will be a 1.5 compatible release coming soon. Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
TheIronYuppie 10y ago
Tons of great suggestions in the thread, but let me also point you to the kubeadm work - in which in Kubernetes 1.6, folks are trying to make it easier to install HA setups. Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc... Disclosure: I work at Google on Kubernetes.
robhirschfeld 10y ago
Yes. We've been doing this w/ Digital Rebar. There are additional steps (like creating load balancer automation) that we are planning to add. FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison. Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.

by jstoja 10y ago

Has anyone already successfully created a HA-cluster on a bare metal infrastructure? It seems overly complicated and not that well documented from what I can see.

hnarayanan 10y ago

I have attempted this using https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare... . One can get quite far, but I am not sure it is worth the effort. I understand this is a vague analogy, but it feels like you're trying to setup email infrastructure when what you really want is to send email. It is a lot easier on the public cloud, or easier still on a managed service.

iagooar 10y ago
What about people who cannot go to the cloud? What about people who need more performance? Lower costs? Kubernetes seems to be such a great abstraction for the underlying hardware, why not use it where it is needed the most?
- jdoliner 10y ago
  People do run Kubernetes on bare metal... but there's no denying it takes effort. That effort should be coming down over time, but unfortunately there's still a ways to go.
  - AlexB138 10y ago
    That's not entirely true. I attend meet-ups with engineers from a fairly large company whose name you would recognize who run K8s on bare metal. I know that's vague, but I don't know how public their infrastructure is, so I'd rather not speak out of turn. You're absolutely right that it takes some effort.
    - SEJeff 10y ago
      Bloomberg runs Kubernetes on bare metal and one of my friends is an engineer on the project (but did not speak at KubeCon about it). They've got a relatively sizable install as well.
- nixgeek 10y ago
  citation required, since there are IaaS offerings that do not involve a hypervisor (e.g. Rackspace OnMetal) and therefore its possible to move "to the cloud" without performance penalties. on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
  - manigandham 10y ago
    Citation needed for what? its common knowledge that the major clouds VMs with lower performance and higher cost in exchange for flexibility.
  - vidarh 10y ago
    > and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own. You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same. And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo. For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
  - iagooar 10y ago
    citation required for what? For stating, that it is complicated to install on bare-metal and that it's been a priority for Kubernetes to be deploying on cloud servers?
jabzd 10y ago
I felt very similar. Especially thinking about it from a disaster recovery/re-bootstrapping perspective. That scared me into using docker swarm kit which is stupidly easy to bootstrap. Keep in mind that's not necessarily a like for like replacement but it was enough for my use case. I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
philips 10y ago
What would your ideal setup look like on baremetal? The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months. [1] https://github.com/kubernetes-incubator/bootkube https://github.com/kubernetes-incubator/bootkube [2] http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/

vlad00 10y ago

I've deployed kubernetes on aws as if it were bare metal. Try to run through https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way translating what they do there to your particular setup(configure firewall, add routes to router, etc)

amatix 10y ago
I'd suggest looking into kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) for Kubernetes on AWS, whether you let it take the actions or you tweak/use the Terraform config it can generate

appelza 10y ago

Yeah, quite easily. Look at Openshift, ansible playbook installer for Fedora 24. You can configure 2 masters/ha/etc in the playbook.

olavgg 10y ago

Could you use Canonical MAAS (https://www.ubuntu.com/server/maas https://www.ubuntu.com/server/maas) for this?

jcastro 10y ago
Indeed, we fully support Kubernetes on MAAS on Ubuntu. Currently for 1.4 but we'll be updating support for 1.5 over the next week or so. Disclaimer: I'm on the team that works on this.

ownagefool 10y ago

Yep. We PXE boot coreos and include a cloud-config that bootstraps a kubelet and pulls master manifests. It's actually not very complicated at all.

sandGorgon 10y ago

+1 there has been a discussion of making a sig-on-premise/sig-bare-metal (with a fair bit of opposition from others) [1]. Would help to have your comments there as well. The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/17 https://github.com/kubernetes/ingress/issues/17 The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip. a lot of this is taken care for the cloud...but NOT for bare metal. [1] https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/ztVnvbrpTK8/-R4PlfHVBwAJ https://groups.google.com/forum/m/?utm_medium=email&utm_sour...

jcastro 10y ago
sig-onprem is coming, it was decided at the last community meeting, expect an announcement soonish.
elcct 10y ago
It also has problems with using certificates when reporting etcd health: https://github.com/kubernetes/kubernetes/issues/27343 https://github.com/kubernetes/kubernetes/issues/27343 I was writing a blog about installing it on bare metal, but this issue got me blocked.
- sandGorgon 10y ago
  could you ping on slack on sig-cluster-ops? thats the defacto meeting ground for bare metal issues. Though honestly i think it is a fundamental issue and not specifically related to BM.

tyingq 10y ago

It won't help you understand how to install it, but if you just want to get it done, see http://stackpoint.io http://stackpoint.io They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean. Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.

irontoby 10y ago
Stackpoint user here; running a very small (3 node) cluster on DigitalOcean currently using their service. It's been relatively painless so far, but I'm pretty new to k8s so my needs are very basic.

hanikesn 10y ago

We're using Puppet to deploy our kubernetes cluster on bare-metal as well as on AWS. Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin. We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB. Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.

fen4o 10y ago

PXE CoreOS and cloud-config to run kubelet and pull manifests. [0] [0] https://youtu.be/4gyeixJLabo?t=814 https://youtu.be/4gyeixJLabo?t=814

old-gregg 10y ago

Regarding HA-on-bare metal, we do this for our customers here: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.

jwelch55 10y ago

I'd suggest checking out KET (Kismatic Enterprise Toolkit) - https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic There will be a 1.5 compatible release coming soon. Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.

TheIronYuppie 10y ago

Tons of great suggestions in the thread, but let me also point you to the kubeadm work - in which in Kubernetes 1.6, folks are trying to make it easier to install HA setups. Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc... Disclosure: I work at Google on Kubernetes.

robhirschfeld 10y ago

Yes. We've been doing this w/ Digital Rebar. There are additional steps (like creating load balancer automation) that we are planning to add. FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison. Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.