6 ms·

How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally pro

by pojzon 7y ago

How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.

jrockway 7y ago
The big problem with running your own cluster is the extra machines you need for a high-availability control plane, which is expensive. That is why Amazon and now Google feel like they can charge for this; you can't really do it any better yourself.
takeda 7y ago
From what I've seen looks like managing k8s on your own often ends up requiring a dedicated team to keep with their insane release cycle.
- freedomben 7y ago
  Can confirm. Depending on your cluster size you will need at least 2 dedicated people on the "Kubernetes" team. You'll probably also end up rolling-your-own deployment tools because K8s API is a little overwhelming for most devs.
  - mleonhard 7y ago
    I started learning Kubernetes and was overwhelmed. The biggest problem was the missing docs. I filed a Github issue asking for missing Kubernetes YAML docs: https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139 Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.
    - jrockway 7y ago
      "kubectl explain deployment"
  - apple4ever 7y ago
    Confirmed here too. I don’t think management realized the amount of toil that Kube requests to stay up to date.
  - pojzon 7y ago
    To be honest we are heavly using EKS and AKS in other teams and each of those teams has a dedicated devops subteam to help them not only with k8s but also other infrastructure because bare k8s is pretty useless for business. So either way you end up in a situation where you require dedicated devops team pr dedicated teammembers to keep up with changing requirements.
freedomben 7y ago
Disclaimer: I work for Red Hat and am very biased, but this is my own honest opinion. If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allows you to scale nodes quickly and easily. Works on bare metal or in the cloud (or both, but make sure you have super low latency between data centers if you are going to do that). It's also pretty valuable to be able to call Red Hat support if something goes wrong. (I still shake my head over the number of days I spent debugging arcane networking issues on EKS before moving to OpenShift, which would have paid for a year or more of support just by itself).
- stas2k 7y ago
  Don't want to sound snarky, but how about an upgrade path from 3.11 to 4.x? I am a heavy Openshift user and it seems that RH just dumped whatever architecture they had with pre-4 clusters and switched to a Tectonic-like 4.x installations without any way to upgrade other than a new installation. This makes it hard to migrate with physical nodes.
  - bgracely 7y ago
    There is a free tool to help with migrations from OpenShift 3.x to 4.x - https://access.redhat.com/documentation/en-us/openshift_container_platform/4.2/html/migration/index?extIdCarryOver=true&sc_cid=701f2000000u72fAAA https://access.redhat.com/documentation/en-us/openshift_cont... It's not a rolling upgrade, but it allows you to move the apps, PVs, policies in as granular a manner as you wish.
  - freedomben 7y ago
    Not snarky at all, you are more correct than you may realize. The upgrade is a challenge because we move from RHEL 7 to Red Hat CoreOS 8 as the host OS, as well move all the OpenShift code to operators instead of in the binary. OpenShift itself can now also manage the nodes (thanks to RHCOS), as well as fully self-upgrade. For container runtime we move from Docker to CRI-O (for a number of reasons, high on the list is security). It's a massive overhaul which is really more akin to a brand new product than a major version. I generally can't stand major overhauls because most of the time they don't give you much new and often bring regressions. However, this really was a tremendous improvement, the fruits of which are not even fully realized. Because of that major change, the cluster upgrade path is a little more involved than usual. It's a complete reinstall of the OS and rebuild of the cluster. There are tools to help tho, and as the path is tread things will get easier and better supported. If you go through the same wave as me, you'll be annoyed at first but then once it is done and you have a 4.x cluster you'll be really happy with it (especially when you can manage everything as an operator). Luckily from an app perspective very little will change since Kubernetes is the API. A sibling comment linked to some helpful documentation. I can't be specific right now but I can tell you that your need is known, and some very smart people are working on it. If you want to email me (check my HN bio page) or jump in the Keybase group called "openshift_okd," I'm happy to chat more about it (not in an official Red Hat support capacity, just as friend to friend :-) ). I haven't done a migration myself yet but I know people who have, and I plan to get into it personally soon as well.
bcheung 7y ago
I'm running my own on bare metal dedicated servers. You will need to install a few extra things (MetalLB for LoadBalancer, CertManager for SSL, an ingress controller (nginx, Ambassador, Gloo), and one of the CSI plugins for your preferred storage method). It is extra work but as a personal cluster for hobby work, I'm paying $65/mo total for the cluster. Same specs would probably be $1000/mo at a public cloud provider.
- hitpointdrew 7y ago
  Traefik can also automatically get letsencrypt certs, if you don't want to use CertManager. Traefik gives the added benefit of also doing Ingress.
- theptip 7y ago
  MetalLB looks fun, hadn't seen that one. If you want something production-grade (i.e. doesn't say "beta" on the tin) then I think Calico should solve most of the same problems too (it does BGP peering to your ToR switch): https://docs.projectcalico.org/networking/determine-best-networking https://docs.projectcalico.org/networking/determine-best-net... Does MetalLB do something extra I'm missing?
  - shaklee3 7y ago
    Metallb had more features than calico before calico had the external service advertisement feature. Now that they do, services can use ecmp load balancers just as metallb does.

by pojzon 7y ago

jrockway 7y ago

The big problem with running your own cluster is the extra machines you need for a high-availability control plane, which is expensive. That is why Amazon and now Google feel like they can charge for this; you can't really do it any better yourself.

takeda 7y ago

From what I've seen looks like managing k8s on your own often ends up requiring a dedicated team to keep with their insane release cycle.

freedomben 7y ago
Can confirm. Depending on your cluster size you will need at least 2 dedicated people on the "Kubernetes" team. You'll probably also end up rolling-your-own deployment tools because K8s API is a little overwhelming for most devs.
- mleonhard 7y ago
  I started learning Kubernetes and was overwhelmed. The biggest problem was the missing docs. I filed a Github issue asking for missing Kubernetes YAML docs: https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139 Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.
  - jrockway 7y ago
    "kubectl explain deployment"
- apple4ever 7y ago
  Confirmed here too. I don’t think management realized the amount of toil that Kube requests to stay up to date.
- pojzon 7y ago
  To be honest we are heavly using EKS and AKS in other teams and each of those teams has a dedicated devops subteam to help them not only with k8s but also other infrastructure because bare k8s is pretty useless for business. So either way you end up in a situation where you require dedicated devops team pr dedicated teammembers to keep up with changing requirements.

freedomben 7y ago

Disclaimer: I work for Red Hat and am very biased, but this is my own honest opinion. If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allows you to scale nodes quickly and easily. Works on bare metal or in the cloud (or both, but make sure you have super low latency between data centers if you are going to do that). It's also pretty valuable to be able to call Red Hat support if something goes wrong. (I still shake my head over the number of days I spent debugging arcane networking issues on EKS before moving to OpenShift, which would have paid for a year or more of support just by itself).

stas2k 7y ago
Don't want to sound snarky, but how about an upgrade path from 3.11 to 4.x? I am a heavy Openshift user and it seems that RH just dumped whatever architecture they had with pre-4 clusters and switched to a Tectonic-like 4.x installations without any way to upgrade other than a new installation. This makes it hard to migrate with physical nodes.
- bgracely 7y ago
  There is a free tool to help with migrations from OpenShift 3.x to 4.x - https://access.redhat.com/documentation/en-us/openshift_container_platform/4.2/html/migration/index?extIdCarryOver=true&sc_cid=701f2000000u72fAAA https://access.redhat.com/documentation/en-us/openshift_cont... It's not a rolling upgrade, but it allows you to move the apps, PVs, policies in as granular a manner as you wish.
- freedomben 7y ago
  Not snarky at all, you are more correct than you may realize. The upgrade is a challenge because we move from RHEL 7 to Red Hat CoreOS 8 as the host OS, as well move all the OpenShift code to operators instead of in the binary. OpenShift itself can now also manage the nodes (thanks to RHCOS), as well as fully self-upgrade. For container runtime we move from Docker to CRI-O (for a number of reasons, high on the list is security). It's a massive overhaul which is really more akin to a brand new product than a major version. I generally can't stand major overhauls because most of the time they don't give you much new and often bring regressions. However, this really was a tremendous improvement, the fruits of which are not even fully realized. Because of that major change, the cluster upgrade path is a little more involved than usual. It's a complete reinstall of the OS and rebuild of the cluster. There are tools to help tho, and as the path is tread things will get easier and better supported. If you go through the same wave as me, you'll be annoyed at first but then once it is done and you have a 4.x cluster you'll be really happy with it (especially when you can manage everything as an operator). Luckily from an app perspective very little will change since Kubernetes is the API. A sibling comment linked to some helpful documentation. I can't be specific right now but I can tell you that your need is known, and some very smart people are working on it. If you want to email me (check my HN bio page) or jump in the Keybase group called "openshift_okd," I'm happy to chat more about it (not in an official Red Hat support capacity, just as friend to friend :-) ). I haven't done a migration myself yet but I know people who have, and I plan to get into it personally soon as well.

bcheung 7y ago

I'm running my own on bare metal dedicated servers. You will need to install a few extra things (MetalLB for LoadBalancer, CertManager for SSL, an ingress controller (nginx, Ambassador, Gloo), and one of the CSI plugins for your preferred storage method). It is extra work but as a personal cluster for hobby work, I'm paying $65/mo total for the cluster. Same specs would probably be $1000/mo at a public cloud provider.

hitpointdrew 7y ago
Traefik can also automatically get letsencrypt certs, if you don't want to use CertManager. Traefik gives the added benefit of also doing Ingress.
theptip 7y ago
MetalLB looks fun, hadn't seen that one. If you want something production-grade (i.e. doesn't say "beta" on the tin) then I think Calico should solve most of the same problems too (it does BGP peering to your ToR switch): https://docs.projectcalico.org/networking/determine-best-networking https://docs.projectcalico.org/networking/determine-best-net... Does MetalLB do something extra I'm missing?
- shaklee3 7y ago
  Metallb had more features than calico before calico had the external service advertisement feature. Now that they do, services can use ecmp load balancers just as metallb does.