11 ms·
Bare-Metal Kubernetes with K3s
- mwcampbell 6y agoWould it be feasible to adapt this configuration to use just 3 machines total, with the control plane and workers running on the same machines?
- ncrmro 6y agoI actually just got a NUC with 64gb ram. Alpine installed to encrypted NVMe zfs zpool. This runs KVM One of these guests is alpine with k3s (tried k3os, very limiting in a good way) that allows me to pass a host directory directly into the vm using p9 (tried nfs but lil heavy) So any storage needs get the benifit of regular snapshots, compression and sync to nas. Really wish it would be more know that k8 single node is all most people need to get started! Actual kubernetes deployments and services etc deployed with ansible managed helm
- alexellisuk 6y agoMaybe you could set up multiple K3s hosts and form a cluster with the VMs? Have you taken a look at inlets-operator that I mention in the post for hosting? My favourite party trick is exposing the IngressController's port 80 and 443. https://docs.inlets.dev/#/get-started/quickstart-ingresscontroller-cert-manager?id=expose-your-ingresscontroller-and-get-tls-from-letsencrypt https://docs.inlets.dev/#/get-started/quickstart-ingresscont... There was a user who was paying 7 USD / mo per site to host 20 side-project sides.. expensive. They switched out to a computer under their desk and saved a lot of money that way.
- ncrmro 6y agoThat is the plan eventually so that I can do node upgrades with lil downtime
- ClumsyPilot 6y agoI think this is accurate, but thrre is great educational value when you can see several physical computers working in a cluster, scaling pod on CPU, etc. I used to have 3 low-power Nuc-style computers, and after playing around for a few months I did the same thing as you have - replaced them with a single beefier machine, and its a lot more practical.
- ncrmro 6y agoTotes agree, I think I got the vms set up once a long time ago before k3s and it was some real work
- mitjam 6y agoCan you explain what p9 is?
- CodesInChaos 6y agoA network filesystem developed for the Plan 9 operating system https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol)
- ncrmro 6y agoWhen you create or a edit a vm you can add a file system passthrough. You mount this in your guest vms fstab and specify 9p in the options
- kosikond 6y agothanks Alex for all your work in k8s ecosystem! i have 3 cheapish VPS with public IPs and no VPC . Am but struggling to find a way to get k3s HA working with MetalLB, I dont have EIP /NodeBalancer resource, and dont want to resort to mere 1master+2agents cluster. Any tips or links are appreciated
- alexellisuk 6y agoAre you able to use BGP or ARP as suggested in the post? Does your VPS provide have a managed LoadBalancer option? I would probably suggest you go for 3x servers in HA, it uses marginally more memory but can tolerate a failure. You can always go and use DigitalOcean managed K8s, or self-host at home using inlets that I mention in the post.
- tasqa 6y agoWhat I'd like to see for change is actually doing the bare metal part itself. I've seen so many k8s showcase posts of this or that, but never actually someone who's running it on actual servers they own and without using any big four cloud API's (I consider Equinix to be part of those too soon...) to handle the LB/Ingress/Network virtualization stuff they provide and still say it is easy to use..
- alexellisuk 6y agoYou may enjoy the parts on ARP at the end of the post. I am planning a post on HA K3s with etcd on netbooted RPis. The netbooted RPi part is already available for free to my GitHub Sponsors. Here's a gist I put together whilst figuring out how it should look: https://gist.github.com/alexellis/09b708a8ddeeb1aa07ec276cd8ab228a https://gist.github.com/alexellis/09b708a8ddeeb1aa07ec276cd8... Not sure what you mean re: "network virtualisation" though?
- sandGorgon 6y agowe are running k3s on metal in production. works great actually. we use haproxy as ingress and lb.
- polskibus 6y agoHow stable do you see it ? What's your cluster size? How long has it been running? Any tips for how to approach starting such setup?
- sandGorgon 6y agowe scale up to about 100 machines. We use spot instances EXTENSIVELY. And that configuration was tricky actually. Its been a couple of months now. Works pretty ok. k3s is actually pretty simple to use now. the tricky part was to integrate with https://github.com/kubernetes/cloud-provider-aws https://github.com/kubernetes/cloud-provider-aws and https://github.com/DirectXMan12/k8s-prometheus-adapter https://github.com/DirectXMan12/k8s-prometheus-adapter The hardest part is to get it to work with spot instances. we use https://github.com/AutoSpotting/AutoSpotting https://github.com/AutoSpotting/AutoSpotting to integrate with it.
- rabbut 6y agoI saw Terraform and was out, because I don’t have the money to pour into recurring costs. Can we have nice things without it being a hook to buy something?
- meddlepal 6y agoTerraform is free and opensource? Or are you making a metacomment about Terraform being painful at times?
- alexellisuk 6y agoHow is using terraform something to do with spending money? If you want to go cheap, you can host at home on old servers or RPis.. there's even links included to help you with that. https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
- certifiedloud 6y agoTerraform is free and open source. Their paid product is a hosting solution.
- miked85 6y agoWhat recurring costs are you referring to?
- im_down_w_otp 6y agoI don't entirely understand what's happening here. The title and the post talk a lot about "Bare-Metal", but it also seems to indicate that everything is hosted on servers running Ubuntu. Which, if accurate, seems to be Kubernetes running on Linux, not bare-metal?
- scaladev 6y agoI guess "bare-metal" is Newspeak for "running directly on hardware without a hypervisor".
- oblio 6y agoIt's not "Newspeak". Kubernetes is an application, it runs on an OS. You can call it "Newspeak" if you consider anything after VMs became super popular "Newspeak" (around 2010, I think?). Bare-metal = no VMs or other virtualization involved.
- tyingq 6y agoIt is somewhat "newspeak", as bare metal has meant "without an OS" in the embedded space for a very long time. This is just a case of two different spaces using the same term for different ideas.
- throwaway894345 6y agoSometimes terms are overloaded for different domains. This is pretty common; no need to argue about which is The One True Definition.
- tyingq 6y agoHence, "just a case".
- dividedbyzero 6y ago
- xaduha 6y agoThat's not what bare-metal means.
- lifty 6y agoWhat does it mean, in your opinion?
- xaduha 6y agohttps://wiki.c2.com/?CloseToTheMetal https://wiki.c2.com/?CloseToTheMetal Think various boards bootloaders, not even BIOS to rely on, that's running on bare metal. Calling something running on a normal x64 OS launched with UEFI as such is silly and inaccurate.
- theptip 6y agoIt's common parlance to use "running on bare metal" as the opposite of "running in a virtual machine", i.e. a synonym of "running your OS without a hypervisor". This usage has been commonplace for a long time, probably two decades. https://en.wikipedia.org/wiki/Bare-metal_server https://en.wikipedia.org/wiki/Bare-metal_server
- lifty 6y agoI am a professional in the field for many years and your version of the term is not widely accepted. There is a wikipedia page which describes the accepted definition: https://en.wikipedia.org/wiki/Bare-metal_server https://en.wikipedia.org/wiki/Bare-metal_server
- xaduha 6y agoMan you guys are laughable, brandishing one wiki page like it's one true source. But nevermind that, several older ones were presented to you and crickets. https://en.wikipedia.org/wiki/Close_to_Metal https://en.wikipedia.org/wiki/Close_to_Metal https://en.wikipedia.org/wiki/Bare_machine https://en.wikipedia.org/wiki/Bare_machine https://wiki.c2.com/?CloseToTheMetal https://wiki.c2.com/?CloseToTheMetal This term goes back to early 2000 at least.
- Damogran6 6y agoIs it just me, or does this need a QA pass? Using the 2017 instructions went about like you'd expect in this space, with everything moving as fast as it does. Using the instructions here has it complaining "Terraform initialized in an empty directory!" mucking about and thinking the main.tfvars file was a typo and it wanted main.tf got things a little further along then generated another error because it WAS meant to be main.tfvars... And at this point I'm frustrated, confused, and no closer to understanding the concepts...
- ACAVJW4H 6y agohttps://metallb.universe.tf/ https://metallb.universe.tf/ is also worth checking out
- bogomipz 6y agoThe post mentions kube-vip. I would be curious to hear peoples experiences with kube-vip. The documentation for kube-vip seems a bit scant unfortunately. Does anyone know how this compares/differs to the MetalLB project?
- thebsdbox 6y agoHowdy! https://kube-vip.io https://kube-vip.io does two things: - Control Plane HA w/BGP or ARP - Service Type: LoadBalancer w/BGP or ARP It is similar to metallb but has a number of differences under the covers in how it works inside Kubernetes, it is cloud controller agnostic so as long as something attaches an IP to spec.IngressIP then kube-vip will advertise it. For edge deployments loadBalancers addresses can use the local DHCP for addresses etc..
- bogomipz 6y ago>"It is similar to metallb but has a number of differences under the covers in how it works inside Kubernetes" This was atually the part I was curious about. Could you elaborate? Or is there a design doc somewhere?
- thebsdbox 6y agohttps://kube-vip.io/hybrid/services/ https://kube-vip.io/hybrid/services/ covers a bit of it, the idea is to be decoupled so any on-premises environment can throw together their own CCM that speaks to their local network or IPAM.
- thebsdbox 6y agoFor clarity the Control plane High-Availability and Service type LoadBalancer is all provided by kube-vip.io out of the box. The goal is to provide as "close to cloud provider" environment as possible :-)
- mrweasel 6y agoGetting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. The tricky part is long term maintenance and storage. I'm not really sure what people expect gain from these kinds of article, they're great as notes, but it's not something I'd use as a starting point for installing a production Kubernetes cluster. The initial setup of a Kubernetes cluster is something most HN readers could do in half a day or so. Learning to manage a cluster, that's tricky. Even if you resort to tools like Rancher or similar, you're still in deep waters. Also why would people assume that there's any difference in installing Kubernetes on an operating system running on physical hardware vs. on virtual machines?
- Spivak 6y agoWhen people talk about “bare metal” k8s they mean on-prem without the support of integrated cloud infrastructure. All the external stuff like load balancers, ingress controllers, routing, bgp, public ip pools, shared storage, vm creation for scaling, are all things the cluster can’t do for itself and have to be implemented and integrated.
- Already__Taken 6y agoShared storage being a particular head-f to get solved. Everything else is a reasonably approachable problem you can tackle as it comes.
- tene 6y agohttps://rook.io/ https://rook.io/ is a really great way to get shared storage up quickly and simply.
- Damogran6 6y agoI'm still looking for a good resource as someone who's taken their eye off the ball for a decade or so. Things have (and continue to) move at an astounding pace...and I'm needing something that goes from 'Pets and autoconf' to where we are now.