11 ms·

MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

359 points by punnerud 5y ago

m1ghtym0 5y ago
How does it compare against minikube? Seems to be based on Snaps vs minikube's VM-based approach. Any other major pros/cons?
- GrantZvolsky 5y ago
  From my experience of using microk8s for mostly offline development and trying minikube once: Microk8s pros: - bind-mount support => it is possible to mount a project including its node_modules and work on it from the host while it hot-reloads in the pod. - The addons for dns, registry, istio, and metallb just work. - Feels more snappy than minikube. Microk8s cons: - Distributed exclusively via snap => can't be easily installed on nix/nixos.
  - m1ghtym0 5y ago
    Thanks!
  - panlinux 5y ago
    microk8s also makes it super easy to have a multi-node cluster, even with HA.
- Fiahil 5y ago
  I thought minikibe would not cluster multiple machine ?
  - daitangio 5y ago
    Yes, it seems not possibile. You can EMULATE a multi mode cluster but inside a big host machine. Minikube is very effective on my huble opinion to STUDY K8s becauese it is always strong aligned with K8s releases. I do not think minikube is a good choice for production but hey, I could be wrong... someone want to share any experience?
    - m1ghtym0 5y ago
      Yes, definitely not production. Using them (MicroK8s, Minikube, K3s) for local testing mostly.
- hodgesrm 5y ago
  I used Minikube constantly for development. The best way to run it is directly on Linux, in which case it does not require a VM and is very stable. VMs on the other hand seem to cause a lot of problems--it's not especially stable on Mac OS and does not seem to survive reboots very well. (YMMV.)
polskibus 5y ago
I thought Kubernetes is not great for environments with poor network connectivity, which is quite common when dealing with Edge and IoT scenarios. Has that changed?
- alexellisuk 5y ago
  Yes it's changed massively, but it's recent - only since 2019. I am the creator of the k3sup (k3s installer) tool that was mentioned and have a fair amount of experience with K3s on Raspberry Pi too. You might also like my video "Exploring K3s with K3sup" - https://www.youtube.com/watch?v=_1kEF-Jd9pw https://www.youtube.com/watch?v=_1kEF-Jd9pw https://k3sup.dev/ https://k3sup.dev/
  - ngrilly 5y ago
    What has been changed to improve k8s for applications with poor connectivity between workers on the edge and the control plane in the cloud?
    - smarterclayton 5y ago
      Other than bugs around bad connections causing hangs (the kubelet is less vulnerable to pathological failures in networking causing it to stall), nothing significant. Kube is designed for nodes to have continuous connectivity to the control plane. If connectivity is disrupted and the machine restarts, none of the workloads will be restarted until connectivity is restored. I.e. if you can have up to 10m of network disruption then at worst a restart / reboot will take 10m to restore the apps on that node. Many other components (networking, storage, per node workers) will likely also have issues if they aren’t tested in those scenarios (i’ve seen some networking plugins hang or otherwise fail). That said, there are lots of people successfully running clusters like this as long as worst case network disruption is bounded, and it’s a solvable problem for many of them. I doubt we’ll see a significant investment in local resilience in Kubelet from the core project (because it’s a lot of work), but I do think eventually it might get addressed in the community (lots of OpenShift customers have asked for that behavior). The easier way today is run edge single node clusters, but then you have to invent new distribution and rollout models on top (ie gitops / custom controllers) instead of being able to reuse daemonsets. We are experimenting in various ecosystem projects with patterns would let you map a daemonset on one cluster to smaller / distributed daemonsets on other clusters (which gets you local resilience).
Kostic 5y ago
From all of the low-ops K8s distributions, k3s[0] is the best from perspective of inital setup, maintenance and usage on less powerful hardware. There are even now higher-level tools such as k3os and k3sup to further reduce the initial deployment pains. MicroK8s prides with 'No APIs added or removed'. That's not that positive in my book. K3s on the other hand actively removes the alpha APIs to reduce the binary size and memory usage. Works great if you only use stable Kubernetes primitives. [0] https://k3s.io/ https://k3s.io/
- alexellisuk 5y ago
  Thanks for mentioning K3sup [0] I used Microk8s on a client project late last year and it was really painful, but I am sure it serves a particular set of users who are very much into the Snap/Canonical ecosystem. In contrast, K3s is very light-weight and can be run in a container via the K3d project. If folks want to work with K8s upstream or development patches against Kubernetes, they will probably find that KinD is much quicker and easier. Minikube has also got a lot of love recently, and can run without having a dependency on Virtual Box too. [0] https://k3sup.dev/ https://k3sup.dev/ [1] https://kind.sigs.k8s.io/docs/user/quick-start/ https://kind.sigs.k8s.io/docs/user/quick-start/
- stavros 5y ago
  This looks interesting, what control plane do the nodes usually connect to? I'm trying to see the use case for me, where I have a main NAS in my house and a few disparate Raspberry Pis, but I'm not sure if I would run the control plane on the NAS or if I would use a hosted one somewhere else.
- rcarmo 5y ago
  I've had a number of issues with k3s on very low spec hardware (typically ARM), where it would take up to 25-50% of CPU just sitting idle with no pods. Stopped using it for those scenarios a year ago, wonder if that's fixed.
  - Fiahil 5y ago
    I had the same issue, it wasn't fixed on my last upgrade. I just let it eat some cpu : my pi is somewhat busy anyway
hardwaresofton 5y ago
Deploying k8s has gotten a lot easier these days -- some alternatives in this space: - https://docs.k0sproject.io https://docs.k0sproject.io (https://github.com/k0sproject/k0s https://github.com/k0sproject/k0s) - https://k3s.io https://k3s.io (https://github.com/k3s-io/k3s/ https://github.com/k3s-io/k3s/) k0s is my personal favorite and what I run, the decisions they have made align very well with how I want to run my clusters versus k3s which is similar but slightly different. Of course you also can't go wrong with kubeadm[0][1] -- it was good enough to use minimally (as in you could imagine sprinkling a tiny bit of ansible and maintaining a cluster easily) years ago, and has only gotten better. [0]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/ https://kubernetes.io/docs/reference/setup-tools/kubeadm/ [1]: https://github.com/kubernetes/kubeadm https://github.com/kubernetes/kubeadm
- ofrzeta 5y ago
  I had a good time with Kubespray. Essentially you just need to edit the Ansible inventory and assign the appropriate roles.
  - cyberpunk 5y ago
    Sure, if it works, upgrades are somewhat fraught though (I mean, upgrading a 20 node cluster is an hour long ansible run, or it was when we were using it) We switched to rke, it’s much better.
    - ofrzeta 5y ago
      An hour to upgrade a 20 node cluster doesn't seem unreasonable for me - when you are doing a graceful upgrade that includes moving workloads between nodes. I don't know anything about rke. Might be interesting but it seems different enough from upstream Kubernetes that you have to learn new things. Seems to me a bit similar to Openshift 4 where the cluster manages the underlying machines.
      - cyberpunk 5y ago
        RKE is minutes on the same cluster, and a one click rollback too. It’s just K8s packaged as containers really.
kissgyorgy 5y ago
If anybody seriously believe that Kubernetes is good on Edge and IoT, our industry is in deep trouble. I usually like Canonical, but this is next level bullshit.
- qbasic_forever 5y ago
  Why? In its smallest form k8s is a few go processes handling a proxy and some container scheduling/runtime logic. Sure if you run an entire control plane on the edge you're adding more complexity... but you don't have to do that, and control planes are complex beasts by their nature.
  - kissgyorgy 5y ago
    IoT devices doesn't even have the memory required for running a k8s worker process. They should run a low-cost, very small embedded system. Most of them doesn't even have a Linux kernel.
- Jonnax 5y ago
  Care to explain why? And what industry you're talking about? Kubernetes has made efforts for nodes to work with poor network connectivity. The node requirements aren't that big. A lot of use cases are relatively easy to containerise. And edge / IoT devices are getting more powerful as well. They aren't talking about consumer IoT here as well. It has become a meme that Kubernetes is complicated. But it solves a lot of orchestration problems that would need to be implemented in other ways.
  - stonemetal12 5y ago
    IDK about OP, but when I think IoT, I think small single purpose devices: a ring door bell, a "smart" thermostat or fire alarm, a security camera. There at most a couple of processes running, what is there to orchestrate on an IoT doorbell? On the backend where you have services being fed, processing, and presenting all that data sure Kubernetes that part up, but that doesn't seem to need a special distro of Kubernetes.
holografix 5y ago
Another vote for K3s. Easy to set up on a Pi cluster. Pity Istio doesn’t offer an ARM build
- crb 5y ago
  There are unofficial builds at https://github.com/querycap/istio https://github.com/querycap/istio, and it seems like it's got to a point (with Envoy now supporting arm64) that it could be upstreamed. I'll keep track of it for you! - istio SC member
- Mumps 5y ago
  their tutorial [1] is a bit unclear on which Pi's can be used - I have a stack of pi zeroes not doing anything at the moment. Do you happen to know if they could handle it (just for a learning project at home). The docs say recommended 4gb memory, but could I run a huge swap partition for that? [1] https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-raspberry-pi#1-overview https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-ra...
kalev 5y ago
Having a RaspberryPi laying idle in a drawer somewhere, I wonder if others have installed microk8s or k3s on it. What kind of workloads are you running on it? Whoever has OpenFaas installed on your rpi, what type of functions are you running?
- rcarmo 5y ago
  I mostly gave up on containers on Pis due to the overhead (disk space, deployment hassles with private registries, etc.). Built my own mini-Heroku and run all my services/functions there: https://github.com/piku https://github.com/piku
  - agucova 5y ago
    This is honestly just awesome. I love how it just uses git, no web interface or complex configuration needed.
    - rcarmo 5y ago
      Thanks! I got fed up with massive manifests and was enamoured with Heroku's approach, so I looked at uwsgi, realised it was a damn good process supervisor, and just did it.
  - pixelmonkey 5y ago
    +1 to piku. I use it on a homelab server (not a Raspberry Pi) and just love how simple it is. It sets up a Python deploy environment for web apps that is exactly how I'd personally hand-configure the same in production (nginx, uwsgi, git). The best part of the project is that the entire thing is 1,000 lines of open source and readable Python code, so there's truly no magic whatsoever.
    - rcarmo 5y ago
      My blog used to run on it until I turned it into a static site, but the builder “function” now happily lives alongside a Node-RED install and a bunch of other things on a tinier VM.
puppet-master 5y ago
Looking forward to this being decoupled from snapd eventually. Until then, not the fientist hope I'd touch this when any alternative exists where snapd can be avoided
- alias_neo 5y ago
  I feel like snapd must be hindering the uptake of microK8s, surely? Canonical is pushing it hard, but snapd is a hard no. Every time I see it mentioned, I check to see if they ditched snapd yet, alas, today is not the that day.
gnfargbl 5y ago
Plain k8s has a fearsome reputation as being complex to deploy, which I don't think is quite deserved. It isn't totally straightforward, but the documentation does tend to make it sound a bit worse than it actually is. I run a couple of small clusters and my Ansible script for installing them is pretty much: * Set up the base system. Set up firewall. Add k8s repo. Keep back kubelet & kubeadm. * Install and configure docker. * On one node, run kubeadm init. Capture the output. * Install flannel networking. * On the other nodes, run the join command that is printed out by kubeadm init. Running in a multi-master setup requires an extra argument to kubeadm init. There are a couple of other bits of faffing about to get metrics working, but the documentation covers that pretty clearly. I'm definitely not knocking k3s/microk8s, they're a great and quick way to experiment with Kubernetes (and so is GKE).
- professor_v 5y ago
  I remember about 5 years ago I tried to deploy it on CoreOS using the available documentation and literally couldn't get it working. I haven't done a manual deployment since. I hope it got significantly better and I may be an idiot but the reputation isn't fully undeserved. The problem back then was also that this was usually the first thing you had to do to try it out. Doing a complicated deployment without knowing much about it doesn't make it any easier.
  - Uberphallus 5y ago
    Same here. I just wanted to play with it for my toy projects and personal services, so I didn't really push a whole lot, but it just felt like there were too many moving parts to figure out. I didn't need autoscaling or most of the advanced features of k8s, so I just went back to my libvirt-based set of scripts.
  - ClumsyPilot 5y ago
    I run kubernetes on a home server, but it took me a couple weeks of testing and trial and error to arrive at a setup I was happy with, and I already had experience of K8S in the cloud. At the time I was stuck without a work laptop, so had time to self-educate, but normally I wouldnt have that kind of time to sink in.
kiney 5y ago
How is persistent storage handled on microk8s (or k3s)?
- traspler 5y ago
  If you want more than just local disk (directory/path on the node the pods currently reside on) you will have to deploy your own storage provider to handle PVCs (on k3s at least). According to Microk8s docs, it's the same there.
  - kiney 5y ago
    ok. thanks. Do you have any recommendations for lightweight solutions? I ran a ceph cluster a few years ago but remember it being quite a bit of work.
    - traspler 5y ago
      I only know of Rook, Longhorn and Portworx. I heard Longhorn is heavy on the CPU, but I‘ve not used any of them yet.
  - spicypony 5y ago
    OpenEBS is a plug-in too which opens up lots of options
stubish 5y ago
Now with Windows installer. This could make life a lot easier in many dev environments.
sidcool 5y ago
We evaulated K3S vs MicroK8S and sided with K3S. MicroK8S seems little complicated to me.
gitgud 5y ago
Interesting that this is supported and maintained by Canonical (Ubuntu). Maybe it will be shipped inside Ubuntu one day.
- timClicks 5y ago
  In some sense, it already is. Run `snap install microk8s` and it's installed.
conjectures 5y ago
Is this a minikube alternative?
- achalkias 5y ago
  It is, and it doesn't require a VM to run the K8s cluster.
baybal2 5y ago
I see nothing of particular utility to iot here. I will call that marketing. After shipping a number of MQTT backends for deployments of few million devices each, I'd say the most troublesome part was getting the anycast network setup, uplinks, routing, and getting the "messaging CDN" second to it. It's very hard to do without having an own ASN in which you have complete freedom. Even in the case of 8 million shipped units fire/burglary alarm client, with ~4M constantly on units, which all have to send frequent pings to signal that they are online, I haven't seen any need in any kind of sophisticated clustering.
ojhughes 5y ago
I moved from k3s to microk8s for local development. I gave up on k3s because I needed calico CNI and it was a pain to set up, on microk8s it's just `microk8s enable calico`. I also found k3s a bit too opinionated with the default Traefik ingress and service-lb.
- hardwaresofton 5y ago
  Traefik is probably the best Ingress out there capability wise for now I think. I've written a bit on it[0] before, but IMO that choice is a good one. Even used it to do some fun external-in SSH connections[1]. I also use it to run a multi-tenant email setup (haraka @ the edge + maddy). It's not like NGINX can't run SMTP expose other ports, but Traefik is easier to manage -- CRDs instead of a ConfigMap update. [0]: https://vadosware.io/post/ingress-controller-considerations-in-early-2021/ https://vadosware.io/post/ingress-controller-considerations-... [1]: https://vadosware.io/post/stuffing-both-ssh-and-https-on-port-443-with-stunnel-ssh-and-traefik/ https://vadosware.io/post/stuffing-both-ssh-and-https-on-por...
  - ojhughes 5y ago
    Maybe, but I prefer to decide myself and in my case I need to test out different ingress controllers with our product. I had troubles getting Nginx ingress to work on k3s. FWIW project contour is also quite nice and has dynamic config reload https://projectcontour.io/ https://projectcontour.io/
  - SOLAR_FIELDS 5y ago
    Is Traefik difficult to initially configure? I use the default ingress-nginx controller on my home lab setup (separate from the “official”, non default & non-free nginx-ingress, absolutely terrible name choices) and it seems to be ok for smaller use cases. It’s not without its idiosyncrasies though. At previous employment with large scale clusters (thousands of nodes) Traefik seemed to be heavily preferred by the SRE’s in my org.
    - hardwaresofton 5y ago
fy20 5y ago
I tried a few incarnations of self-hosted k8s a few years ago, and the biggest problem I had was persistent storage. If you are using a cloud service they will integrate k8s into whatever persistent storage they offer, but if you are self-hosting you are left on your own, it seems most people end up using something like nfs or hostPath - but that ends up being a single point of failure. Have there been any developments on this recently, aimed at people wanted to run k8s on a few RaspberryPi nodes?
- rvdmei 5y ago
  I guess easiest would be Longhorn on top of k3s
  - nullify88 5y ago
    I've found Ceph is more tolerant to failures and staying available. Longhorn was certainly easier to setup and has lower operating requirements, but we encountered outages.
- lvncelot 5y ago
  I've had good experiences using the Rook operator for creating a CephFS cluster. I know that you can run it on k3s, but I don't know whether RaspberryPi nodes are sufficient. Maybe the high RAM Raspi 4 ones.
  - nullify88 5y ago
    I've had good experiences with Rook on k3s in production. Not on raspis though.
  - sethammons 5y ago
    We do this at Twilio SendGrid
- aloknnikhil 5y ago
  Have you tried using a CSI driver to help you do this? https://kubernetes-csi.github.io/docs/drivers.html https://kubernetes-csi.github.io/docs/drivers.html A brief description of what CSI is - https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/ https://kubernetes.io/blog/2019/01/15/container-storage-inte...
PYTHONDJANGO 5y ago
Is K8S still eating so many CPU cycles while idle? Last year I checked and every physical machine in a K8S cluster was burning CPU at 20-30% - with zero payload, just to keep itself up! Don´t you feel that this is totally inacceptable in a world with well understood climate challenges?
- jscheel 5y ago
  It still sucks unfortunately
WolfOliver 5y ago
Can it replace Docker on my local dev machine?
- mdaniel 5y ago
  The classic "it depends" answer applies here, along two axes: container assembly, and container execution. If you mean "docker" the binary used to build container images, you don't even need that right now -- there are multiple projects that will build container images without involving docker or dockerd. If you mean "dockerd" the container management engine that one controls via "docker" to start and stop containers, then yes microk8s will help as they appear to use containerd inside the snap (just like kind and likely all such "single binary kubernetes" setups do): https://github.com/ubuntu/microk8s/blob/master/docs/build.md#building-a-custom-microk8s-package https://github.com/ubuntu/microk8s/blob/master/docs/build.md...
  - WolfOliver 5y ago
    Do you think Docker will disappear and everybody will some kind of "single binary kubernetes" on their machine + A tool to build the images.
    - mdaniel 5y ago
      Unlikely, although I for sure don't trust dockerd for my actual clusters anymore. I still use docker, and its dind friend, to build images because the whole world uses Dockerfile and the compatibility story between those alternative tools and the sea of Dockerfiles out there is ... tricky I would love for something with more sanity to catch on, because Dockerfile as a format has terrible DX, but "wishes horses etc etc."
tasubotadas 5y ago
TBH, I am not that a big fan of microk8s. I have it deployed on VPS and it's far from stable. The server itself, is, probably, overprovision, but I still struggle with responsiveness, logging, and ingress/service management. What's also funny, that using the Ubuntu's ufw service is not that seamless together with microk8s. I am think of moving now to k3s. The only thing that's holding me back is that k3s doesn't use nginx for ingress so I'll need to change some configs. Also, the local storage options are not that clear.
- pas 5y ago
  You can easily deploy the Nginx ingress controller on k3s.
rubyist5eva 5y ago
Is Snap the only way to install this on linux? This is a non-starter for me.
- SemiNormal 5y ago
  Doesn't look like it. https://github.com/ubuntu/microk8s/issues/988#issuecomment-682448812 https://github.com/ubuntu/microk8s/issues/988#issuecomment-6...
  - mkesper 5y ago
    You probably meant "it looks like it" or "it doesn't look like there's another (supported) way". Really sad, and it's a "classic" snap so you don't get any isolation benefits but performance penalties and lots of mounted snap filesystems.
    - achalkias 5y ago
      The MicroK8s team is actively working on a "strict" snap https://github.com/ubuntu/microk8s/issues/2053 https://github.com/ubuntu/microk8s/issues/2053. There you will get all the isolation benefits and security enhancements. https://snapcraft.io/docs/snap-confinement https://snapcraft.io/docs/snap-confinement
utf_8x 5y ago
Microk8s is great if you're lazy. I've recently built a small-ish 3 node cluster hosting internal apps for a few hundred users and pretty much the only setup I needed to do was: install it with Snap*, enable a few plugins (storage, traefik, coredns), run the join command on each node and set up a basic loadbalancer using haproxy** and keepalived***. * I don't like Snap. Like, a lot. But unfortunately there aren't any other options at the moment. ** I have HAProxy load-balancing both the k8s API and the ingresses. Both on L4 so I can terminate TLS on the ingress controller and automatically provision Let's Encrypt certs using cert-manager[1]. *** Keepalived[2] juggles a single floating IP between all the nodes so you can just run HAProxy on the microk8s nodes instead of having dedicated external loadbalancers. [1] https://cert-manager.io/docs/ https://cert-manager.io/docs/ [2] https://github.com/acassen/keepalived https://github.com/acassen/keepalived
rsmitty 5y ago
(Full disclaimer - I'm an engineer at Talos, but I believe it's pretty relevant here) If folks are interested in this kind of K8s deployment, they might also be interested at what we're doing at Talos (https://talos.dev https://talos.dev). We have full support for all of these same environments (we have a great community of k8s-at-home folks running with Raspberry Pis) and a bunch of tooling to make bare metal easier with Cluster API. You can also do the minikube type of thing by running Talos directly in Docker or QEMU with `talosctl`. Talos works with an API instead of SSH/Bash, so there's some interesting things about ease of use when operating K8s that are baked in like built-in etcd backup/restore, k8s upgrades, etc. We're also right in the middle of building out our next release that will have native Wireguard functionality and enable truly hybrid K8s clusters. This should be a big deal for edge deployments and we're super excited about it.
- c7DJTLrn 5y ago
  A few questions if I may. I administer a few thousand machines which are provisioned using Puppet, and I've had my eye on a more immutable style of administration integrating K8S. - How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP. - How does Talos handle upgrades? Can it self upgrade once deployed? - What hardware can Talos run on? Does it work well with virtualisation? - To what degree can Talos dynamically configure itself? What I mean by this is if that a new disk is attached, can it partition it and start storing things on it? - How resilient is Talos to things like filesystem corruption? - What are the minimum hardware requirements? Please forgive my laziness but maybe other HNers will have the same questions.
  - rsmitty 5y ago
    Hey, thanks for the questions. I'll try to answer them in-line: - How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP. For networking in particular, you can configure interfaces directly at boot by using kernel args. But that being said, Talos is entirely driven by a machine config file and there are several different ways of getting Talos off the ground, be it with ISO or any of our cloud images. Generally you can bring your own pre-defined machine configs to get everything configured from the start or you can boot the ISO and configure it via our interactive installer once the machine is online. We also have folks that make heavy use of Cluster API and thus the config generation is all handled automatically based on the providers being used. - How does Talos handle upgrades? Can it self upgrade once deployed? Upgrades can be kicked off manually with `talosctl` or can be done automatically with our upgrade operator. We're currently in the process of revamping the upgrade operator to be smarter, however so it's in flux a bit. As with everything in Talos, upgrades are controllable by the API. Kubernetes upgrades can also be performed across the cluster directly with `talosctl`. We’ve tried to bake in a lot of these common operations tasks directly into the system to make it easier for everyone. - What hardware can Talos run on? Does it work well with virtualisation? Pretty much anything ARM64 or AMD64 will work. We have folks that run in cloud, bare metal servers, Raspberry Pis, you name it. We publish images for all of these with each release. Talos works very well with virtualization, whether that's in the cloud or with QEMU or VMWare. We've got folks running it everywhere. - To what degree can Talos dynamically configure itself? What I mean by this is if a new disk is attached, can it partition it and start storing things on it? Presently, the machine configuration allows you to specify additional disks to be used for non-Talos functions, including formatting and mounting them. However, this is currently an install-time function. We will be extending this in the future to allow for dynamic provisioning utilizing the new Common Operating System Interface (COSI) spec. This is a general specification which we are actively developing both internally and in collaboration with interested parties across the Kubernetes community. You can check that out here if you have interest: https://github.com/cosi-project/community https://github.com/cosi-project/community - How resilient is Talos to things like filesystem corruption? Like any OS, filesystem corruption can indeed occur. We use standard Linux filesystems which have internal consistency checks, but ultimately, things can go wrong. An important design goal of Talos, however, is that it is designed for distributed systems and, as such, is designed to be thrown away and replaced easily when something goes awry. We also try to make it very easy to backup the things that matter from a Kubernetes perspective like etcd. - What are the minimum hardware requirements? Tiny. We run completely in RAM and Talos is less than 100MB. But keep in mind that you still have to run Kubernetes, so there's some overhead there as well. You’ll have container images which need to be downloaded, both for the internal Kubernetes components and for your own applications. We're roughly the same as whatever is required for something like K3s, but probably even a bit less since we don’t require a full Linux distro to get going.
weavie 5y ago
It hadn't occurred to me that there were multiple different kubernetes. I thought it was essentially just a single app, or rather collection of apps. Are these essentially different choices available for people who wish to self host their k8s clusters? Are there any good resources that summarise the different advantages of each choice? What is the same between all the choices, what are the differences?
- seabrookmx 5y ago
  They're Kubernetes distributions. So think of Kubernetes like Linux, and kubeadm, OpenShift, k3s like Ubuntu, Red Hat etc.
  - weavie 5y ago
    What would be the equivalent of the kernel (the bit they all share)?
    - qbasic_forever 5y ago
      The k8s API: https://kubernetes.io/docs/concepts/overview/kubernetes-api/ https://kubernetes.io/docs/concepts/overview/kubernetes-api/ Most distributions will use the stock kubelet process (what runs containers on nodes), but you don't have to--there are kubelet compatible processes to run VMs instead of containers, run webassembly code (krustlet), etc. Everything in k8s can be swapped out and changed, the API spec is the only constant.
- qbasic_forever 5y ago
  Tons of choices, k8s is really just a definition of an API and there lots of things you can plug in at various points to implement parts of the API (or even completely extend it). Read the official docs--they'll walk you through the design and capabilities. As far as what's available, well it's a bit like trying to understand "what's the newest programming languge?" You probably want to plug yourself into the k8s community and start looking at subreddits, SIG meeting notes, community news, etc. New things are popping up literally every day.
jturpin 5y ago
I use MicroK8S for local testing of Kubernetes clusters on my laptop and it works pretty well. I like that I can just run microk8s reset to clear out the state so I can redeploy everything without fear of some lingering configuration laying behind. I have yet to deploy it to an actual server though, though I would definitely be interested if it could do what EKS can do to some capacity (mainly creating EBS volumes and load balancers).
- mkesper 5y ago
  Are you serious? Microk8s reset takes ages. It's faster to uninstall the whole thing. You also can just delete whole workspaces, that should normally be enough.
  - achalkias 5y ago
    Indeed, sometimes it can be faster to nuke the cluster but this is also unsafe. microk8s reset asks Kubernetes to delete the resources created, and that can take time.
  - jturpin 5y ago
    Yeah man, I'm serious. It took 5 minutes 30 seconds on my machine to run microk8s reset. This is well within the tolerance window for something I choose to run every 2 to 6 weeks. This is not part of my normal development strategy. Deleting whole namespaces is not a good idea for things like kube-system, ingress, etc. for obvious reasons.
Whitespace 5y ago
Unobtrusive version: https://microk8s.io/?cp=hide#cookies https://microk8s.io/?cp=hide#cookies
staticassertion 5y ago
If you want "k8s but less" I highly recommend Nomad from Hashicorp and the Hashi-stack in general. It's so damn good.
desktopninja 5y ago
Would be nice too if it could run non containerized applications like Nomad can.

359 points by punnerud 5y ago

How does it compare against minikube? Seems to be based on Snaps vs minikube's VM-based approach. Any other major pros/cons?

GrantZvolsky 5y ago
From my experience of using microk8s for mostly offline development and trying minikube once: Microk8s pros: - bind-mount support => it is possible to mount a project including its node_modules and work on it from the host while it hot-reloads in the pod. - The addons for dns, registry, istio, and metallb just work. - Feels more snappy than minikube. Microk8s cons: - Distributed exclusively via snap => can't be easily installed on nix/nixos.
- m1ghtym0 5y ago
  Thanks!
- panlinux 5y ago
  microk8s also makes it super easy to have a multi-node cluster, even with HA.
Fiahil 5y ago
I thought minikibe would not cluster multiple machine ?
- daitangio 5y ago
  Yes, it seems not possibile. You can EMULATE a multi mode cluster but inside a big host machine. Minikube is very effective on my huble opinion to STUDY K8s becauese it is always strong aligned with K8s releases. I do not think minikube is a good choice for production but hey, I could be wrong... someone want to share any experience?
  - m1ghtym0 5y ago
    Yes, definitely not production. Using them (MicroK8s, Minikube, K3s) for local testing mostly.
hodgesrm 5y ago
I used Minikube constantly for development. The best way to run it is directly on Linux, in which case it does not require a VM and is very stable. VMs on the other hand seem to cause a lot of problems--it's not especially stable on Mac OS and does not seem to survive reboots very well. (YMMV.)

polskibus 5y ago

I thought Kubernetes is not great for environments with poor network connectivity, which is quite common when dealing with Edge and IoT scenarios. Has that changed?

alexellisuk 5y ago
Yes it's changed massively, but it's recent - only since 2019. I am the creator of the k3sup (k3s installer) tool that was mentioned and have a fair amount of experience with K3s on Raspberry Pi too. You might also like my video "Exploring K3s with K3sup" - https://www.youtube.com/watch?v=_1kEF-Jd9pw https://www.youtube.com/watch?v=_1kEF-Jd9pw https://k3sup.dev/ https://k3sup.dev/
- ngrilly 5y ago
  What has been changed to improve k8s for applications with poor connectivity between workers on the edge and the control plane in the cloud?
  - smarterclayton 5y ago
    Other than bugs around bad connections causing hangs (the kubelet is less vulnerable to pathological failures in networking causing it to stall), nothing significant. Kube is designed for nodes to have continuous connectivity to the control plane. If connectivity is disrupted and the machine restarts, none of the workloads will be restarted until connectivity is restored. I.e. if you can have up to 10m of network disruption then at worst a restart / reboot will take 10m to restore the apps on that node. Many other components (networking, storage, per node workers) will likely also have issues if they aren’t tested in those scenarios (i’ve seen some networking plugins hang or otherwise fail). That said, there are lots of people successfully running clusters like this as long as worst case network disruption is bounded, and it’s a solvable problem for many of them. I doubt we’ll see a significant investment in local resilience in Kubelet from the core project (because it’s a lot of work), but I do think eventually it might get addressed in the community (lots of OpenShift customers have asked for that behavior). The easier way today is run edge single node clusters, but then you have to invent new distribution and rollout models on top (ie gitops / custom controllers) instead of being able to reuse daemonsets. We are experimenting in various ecosystem projects with patterns would let you map a daemonset on one cluster to smaller / distributed daemonsets on other clusters (which gets you local resilience).

Kostic 5y ago

From all of the low-ops K8s distributions, k3s[0] is the best from perspective of inital setup, maintenance and usage on less powerful hardware. There are even now higher-level tools such as k3os and k3sup to further reduce the initial deployment pains. MicroK8s prides with 'No APIs added or removed'. That's not that positive in my book. K3s on the other hand actively removes the alpha APIs to reduce the binary size and memory usage. Works great if you only use stable Kubernetes primitives. [0] https://k3s.io/ https://k3s.io/

alexellisuk 5y ago
Thanks for mentioning K3sup [0] I used Microk8s on a client project late last year and it was really painful, but I am sure it serves a particular set of users who are very much into the Snap/Canonical ecosystem. In contrast, K3s is very light-weight and can be run in a container via the K3d project. If folks want to work with K8s upstream or development patches against Kubernetes, they will probably find that KinD is much quicker and easier. Minikube has also got a lot of love recently, and can run without having a dependency on Virtual Box too. [0] https://k3sup.dev/ https://k3sup.dev/ [1] https://kind.sigs.k8s.io/docs/user/quick-start/ https://kind.sigs.k8s.io/docs/user/quick-start/
stavros 5y ago
This looks interesting, what control plane do the nodes usually connect to? I'm trying to see the use case for me, where I have a main NAS in my house and a few disparate Raspberry Pis, but I'm not sure if I would run the control plane on the NAS or if I would use a hosted one somewhere else.
rcarmo 5y ago
I've had a number of issues with k3s on very low spec hardware (typically ARM), where it would take up to 25-50% of CPU just sitting idle with no pods. Stopped using it for those scenarios a year ago, wonder if that's fixed.
- Fiahil 5y ago
  I had the same issue, it wasn't fixed on my last upgrade. I just let it eat some cpu : my pi is somewhat busy anyway

hardwaresofton 5y ago

Deploying k8s has gotten a lot easier these days -- some alternatives in this space: - https://docs.k0sproject.io https://docs.k0sproject.io (https://github.com/k0sproject/k0s https://github.com/k0sproject/k0s) - https://k3s.io https://k3s.io (https://github.com/k3s-io/k3s/ https://github.com/k3s-io/k3s/) k0s is my personal favorite and what I run, the decisions they have made align very well with how I want to run my clusters versus k3s which is similar but slightly different. Of course you also can't go wrong with kubeadm[0][1] -- it was good enough to use minimally (as in you could imagine sprinkling a tiny bit of ansible and maintaining a cluster easily) years ago, and has only gotten better. [0]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/ https://kubernetes.io/docs/reference/setup-tools/kubeadm/ [1]: https://github.com/kubernetes/kubeadm https://github.com/kubernetes/kubeadm

ofrzeta 5y ago
I had a good time with Kubespray. Essentially you just need to edit the Ansible inventory and assign the appropriate roles.
- cyberpunk 5y ago
  Sure, if it works, upgrades are somewhat fraught though (I mean, upgrading a 20 node cluster is an hour long ansible run, or it was when we were using it) We switched to rke, it’s much better.
  - ofrzeta 5y ago
    An hour to upgrade a 20 node cluster doesn't seem unreasonable for me - when you are doing a graceful upgrade that includes moving workloads between nodes. I don't know anything about rke. Might be interesting but it seems different enough from upstream Kubernetes that you have to learn new things. Seems to me a bit similar to Openshift 4 where the cluster manages the underlying machines.
    - cyberpunk 5y ago
      RKE is minutes on the same cluster, and a one click rollback too. It’s just K8s packaged as containers really.

kissgyorgy 5y ago

If anybody seriously believe that Kubernetes is good on Edge and IoT, our industry is in deep trouble. I usually like Canonical, but this is next level bullshit.

qbasic_forever 5y ago
Why? In its smallest form k8s is a few go processes handling a proxy and some container scheduling/runtime logic. Sure if you run an entire control plane on the edge you're adding more complexity... but you don't have to do that, and control planes are complex beasts by their nature.
- kissgyorgy 5y ago
  IoT devices doesn't even have the memory required for running a k8s worker process. They should run a low-cost, very small embedded system. Most of them doesn't even have a Linux kernel.
Jonnax 5y ago
Care to explain why? And what industry you're talking about? Kubernetes has made efforts for nodes to work with poor network connectivity. The node requirements aren't that big. A lot of use cases are relatively easy to containerise. And edge / IoT devices are getting more powerful as well. They aren't talking about consumer IoT here as well. It has become a meme that Kubernetes is complicated. But it solves a lot of orchestration problems that would need to be implemented in other ways.
- stonemetal12 5y ago
  IDK about OP, but when I think IoT, I think small single purpose devices: a ring door bell, a "smart" thermostat or fire alarm, a security camera. There at most a couple of processes running, what is there to orchestrate on an IoT doorbell? On the backend where you have services being fed, processing, and presenting all that data sure Kubernetes that part up, but that doesn't seem to need a special distro of Kubernetes.

holografix 5y ago

Another vote for K3s. Easy to set up on a Pi cluster. Pity Istio doesn’t offer an ARM build

crb 5y ago
There are unofficial builds at https://github.com/querycap/istio https://github.com/querycap/istio, and it seems like it's got to a point (with Envoy now supporting arm64) that it could be upstreamed. I'll keep track of it for you! - istio SC member
Mumps 5y ago
their tutorial [1] is a bit unclear on which Pi's can be used - I have a stack of pi zeroes not doing anything at the moment. Do you happen to know if they could handle it (just for a learning project at home). The docs say recommended 4gb memory, but could I run a huge swap partition for that? [1] https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-raspberry-pi#1-overview https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-ra...

kalev 5y ago

Having a RaspberryPi laying idle in a drawer somewhere, I wonder if others have installed microk8s or k3s on it. What kind of workloads are you running on it? Whoever has OpenFaas installed on your rpi, what type of functions are you running?

rcarmo 5y ago
I mostly gave up on containers on Pis due to the overhead (disk space, deployment hassles with private registries, etc.). Built my own mini-Heroku and run all my services/functions there: https://github.com/piku https://github.com/piku
- agucova 5y ago
  This is honestly just awesome. I love how it just uses git, no web interface or complex configuration needed.
  - rcarmo 5y ago
    Thanks! I got fed up with massive manifests and was enamoured with Heroku's approach, so I looked at uwsgi, realised it was a damn good process supervisor, and just did it.
- pixelmonkey 5y ago
  +1 to piku. I use it on a homelab server (not a Raspberry Pi) and just love how simple it is. It sets up a Python deploy environment for web apps that is exactly how I'd personally hand-configure the same in production (nginx, uwsgi, git). The best part of the project is that the entire thing is 1,000 lines of open source and readable Python code, so there's truly no magic whatsoever.
  - rcarmo 5y ago
    My blog used to run on it until I turned it into a static site, but the builder “function” now happily lives alongside a Node-RED install and a bunch of other things on a tinier VM.

puppet-master 5y ago

Looking forward to this being decoupled from snapd eventually. Until then, not the fientist hope I'd touch this when any alternative exists where snapd can be avoided

alias_neo 5y ago
I feel like snapd must be hindering the uptake of microK8s, surely? Canonical is pushing it hard, but snapd is a hard no. Every time I see it mentioned, I check to see if they ditched snapd yet, alas, today is not the that day.

gnfargbl 5y ago

Plain k8s has a fearsome reputation as being complex to deploy, which I don't think is quite deserved. It isn't totally straightforward, but the documentation does tend to make it sound a bit worse than it actually is. I run a couple of small clusters and my Ansible script for installing them is pretty much: * Set up the base system. Set up firewall. Add k8s repo. Keep back kubelet & kubeadm. * Install and configure docker. * On one node, run kubeadm init. Capture the output. * Install flannel networking. * On the other nodes, run the join command that is printed out by kubeadm init. Running in a multi-master setup requires an extra argument to kubeadm init. There are a couple of other bits of faffing about to get metrics working, but the documentation covers that pretty clearly. I'm definitely not knocking k3s/microk8s, they're a great and quick way to experiment with Kubernetes (and so is GKE).

professor_v 5y ago
I remember about 5 years ago I tried to deploy it on CoreOS using the available documentation and literally couldn't get it working. I haven't done a manual deployment since. I hope it got significantly better and I may be an idiot but the reputation isn't fully undeserved. The problem back then was also that this was usually the first thing you had to do to try it out. Doing a complicated deployment without knowing much about it doesn't make it any easier.
- Uberphallus 5y ago
  Same here. I just wanted to play with it for my toy projects and personal services, so I didn't really push a whole lot, but it just felt like there were too many moving parts to figure out. I didn't need autoscaling or most of the advanced features of k8s, so I just went back to my libvirt-based set of scripts.
- ClumsyPilot 5y ago
  I run kubernetes on a home server, but it took me a couple weeks of testing and trial and error to arrive at a setup I was happy with, and I already had experience of K8S in the cloud. At the time I was stuck without a work laptop, so had time to self-educate, but normally I wouldnt have that kind of time to sink in.

kiney 5y ago

How is persistent storage handled on microk8s (or k3s)?

traspler 5y ago
If you want more than just local disk (directory/path on the node the pods currently reside on) you will have to deploy your own storage provider to handle PVCs (on k3s at least). According to Microk8s docs, it's the same there.
- kiney 5y ago
  ok. thanks. Do you have any recommendations for lightweight solutions? I ran a ceph cluster a few years ago but remember it being quite a bit of work.
  - traspler 5y ago
    I only know of Rook, Longhorn and Portworx. I heard Longhorn is heavy on the CPU, but I‘ve not used any of them yet.
- spicypony 5y ago
  OpenEBS is a plug-in too which opens up lots of options

stubish 5y ago

Now with Windows installer. This could make life a lot easier in many dev environments.

sidcool 5y ago

We evaulated K3S vs MicroK8S and sided with K3S. MicroK8S seems little complicated to me.

gitgud 5y ago

Interesting that this is supported and maintained by Canonical (Ubuntu). Maybe it will be shipped inside Ubuntu one day.

timClicks 5y ago
In some sense, it already is. Run `snap install microk8s` and it's installed.

conjectures 5y ago

Is this a minikube alternative?

achalkias 5y ago
It is, and it doesn't require a VM to run the K8s cluster.

baybal2 5y ago

I see nothing of particular utility to iot here. I will call that marketing. After shipping a number of MQTT backends for deployments of few million devices each, I'd say the most troublesome part was getting the anycast network setup, uplinks, routing, and getting the "messaging CDN" second to it. It's very hard to do without having an own ASN in which you have complete freedom. Even in the case of 8 million shipped units fire/burglary alarm client, with ~4M constantly on units, which all have to send frequent pings to signal that they are online, I haven't seen any need in any kind of sophisticated clustering.

ojhughes 5y ago

I moved from k3s to microk8s for local development. I gave up on k3s because I needed calico CNI and it was a pain to set up, on microk8s it's just `microk8s enable calico`. I also found k3s a bit too opinionated with the default Traefik ingress and service-lb.

hardwaresofton 5y ago
Traefik is probably the best Ingress out there capability wise for now I think. I've written a bit on it[0] before, but IMO that choice is a good one. Even used it to do some fun external-in SSH connections[1]. I also use it to run a multi-tenant email setup (haraka @ the edge + maddy). It's not like NGINX can't run SMTP expose other ports, but Traefik is easier to manage -- CRDs instead of a ConfigMap update. [0]: https://vadosware.io/post/ingress-controller-considerations-in-early-2021/ https://vadosware.io/post/ingress-controller-considerations-... [1]: https://vadosware.io/post/stuffing-both-ssh-and-https-on-port-443-with-stunnel-ssh-and-traefik/ https://vadosware.io/post/stuffing-both-ssh-and-https-on-por...
- ojhughes 5y ago
  Maybe, but I prefer to decide myself and in my case I need to test out different ingress controllers with our product. I had troubles getting Nginx ingress to work on k3s. FWIW project contour is also quite nice and has dynamic config reload https://projectcontour.io/ https://projectcontour.io/
- SOLAR_FIELDS 5y ago
  Is Traefik difficult to initially configure? I use the default ingress-nginx controller on my home lab setup (separate from the “official”, non default & non-free nginx-ingress, absolutely terrible name choices) and it seems to be ok for smaller use cases. It’s not without its idiosyncrasies though. At previous employment with large scale clusters (thousands of nodes) Traefik seemed to be heavily preferred by the SRE’s in my org.
  - hardwaresofton 5y ago

fy20 5y ago

I tried a few incarnations of self-hosted k8s a few years ago, and the biggest problem I had was persistent storage. If you are using a cloud service they will integrate k8s into whatever persistent storage they offer, but if you are self-hosting you are left on your own, it seems most people end up using something like nfs or hostPath - but that ends up being a single point of failure. Have there been any developments on this recently, aimed at people wanted to run k8s on a few RaspberryPi nodes?

rvdmei 5y ago
I guess easiest would be Longhorn on top of k3s
- nullify88 5y ago
  I've found Ceph is more tolerant to failures and staying available. Longhorn was certainly easier to setup and has lower operating requirements, but we encountered outages.
lvncelot 5y ago
I've had good experiences using the Rook operator for creating a CephFS cluster. I know that you can run it on k3s, but I don't know whether RaspberryPi nodes are sufficient. Maybe the high RAM Raspi 4 ones.
- nullify88 5y ago
  I've had good experiences with Rook on k3s in production. Not on raspis though.
- sethammons 5y ago
  We do this at Twilio SendGrid
aloknnikhil 5y ago
Have you tried using a CSI driver to help you do this? https://kubernetes-csi.github.io/docs/drivers.html https://kubernetes-csi.github.io/docs/drivers.html A brief description of what CSI is - https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/ https://kubernetes.io/blog/2019/01/15/container-storage-inte...

PYTHONDJANGO 5y ago

Is K8S still eating so many CPU cycles while idle? Last year I checked and every physical machine in a K8S cluster was burning CPU at 20-30% - with zero payload, just to keep itself up! Don´t you feel that this is totally inacceptable in a world with well understood climate challenges?

jscheel 5y ago
It still sucks unfortunately

WolfOliver 5y ago

Can it replace Docker on my local dev machine?

mdaniel 5y ago
The classic "it depends" answer applies here, along two axes: container assembly, and container execution. If you mean "docker" the binary used to build container images, you don't even need that right now -- there are multiple projects that will build container images without involving docker or dockerd. If you mean "dockerd" the container management engine that one controls via "docker" to start and stop containers, then yes microk8s will help as they appear to use containerd inside the snap (just like kind and likely all such "single binary kubernetes" setups do): https://github.com/ubuntu/microk8s/blob/master/docs/build.md#building-a-custom-microk8s-package https://github.com/ubuntu/microk8s/blob/master/docs/build.md...
- WolfOliver 5y ago
  Do you think Docker will disappear and everybody will some kind of "single binary kubernetes" on their machine + A tool to build the images.
  - mdaniel 5y ago
    Unlikely, although I for sure don't trust dockerd for my actual clusters anymore. I still use docker, and its dind friend, to build images because the whole world uses Dockerfile and the compatibility story between those alternative tools and the sea of Dockerfiles out there is ... tricky I would love for something with more sanity to catch on, because Dockerfile as a format has terrible DX, but "wishes horses etc etc."

tasubotadas 5y ago

TBH, I am not that a big fan of microk8s. I have it deployed on VPS and it's far from stable. The server itself, is, probably, overprovision, but I still struggle with responsiveness, logging, and ingress/service management. What's also funny, that using the Ubuntu's ufw service is not that seamless together with microk8s. I am think of moving now to k3s. The only thing that's holding me back is that k3s doesn't use nginx for ingress so I'll need to change some configs. Also, the local storage options are not that clear.

pas 5y ago
You can easily deploy the Nginx ingress controller on k3s.

rubyist5eva 5y ago

Is Snap the only way to install this on linux? This is a non-starter for me.

SemiNormal 5y ago
Doesn't look like it. https://github.com/ubuntu/microk8s/issues/988#issuecomment-682448812 https://github.com/ubuntu/microk8s/issues/988#issuecomment-6...
- mkesper 5y ago
  You probably meant "it looks like it" or "it doesn't look like there's another (supported) way". Really sad, and it's a "classic" snap so you don't get any isolation benefits but performance penalties and lots of mounted snap filesystems.
  - achalkias 5y ago
    The MicroK8s team is actively working on a "strict" snap https://github.com/ubuntu/microk8s/issues/2053 https://github.com/ubuntu/microk8s/issues/2053. There you will get all the isolation benefits and security enhancements. https://snapcraft.io/docs/snap-confinement https://snapcraft.io/docs/snap-confinement

utf_8x 5y ago

Microk8s is great if you're lazy. I've recently built a small-ish 3 node cluster hosting internal apps for a few hundred users and pretty much the only setup I needed to do was: install it with Snap*, enable a few plugins (storage, traefik, coredns), run the join command on each node and set up a basic loadbalancer using haproxy** and keepalived***. * I don't like Snap. Like, a lot. But unfortunately there aren't any other options at the moment. ** I have HAProxy load-balancing both the k8s API and the ingresses. Both on L4 so I can terminate TLS on the ingress controller and automatically provision Let's Encrypt certs using cert-manager[1]. *** Keepalived[2] juggles a single floating IP between all the nodes so you can just run HAProxy on the microk8s nodes instead of having dedicated external loadbalancers. [1] https://cert-manager.io/docs/ https://cert-manager.io/docs/ [2] https://github.com/acassen/keepalived https://github.com/acassen/keepalived

rsmitty 5y ago

(Full disclaimer - I'm an engineer at Talos, but I believe it's pretty relevant here) If folks are interested in this kind of K8s deployment, they might also be interested at what we're doing at Talos (https://talos.dev https://talos.dev). We have full support for all of these same environments (we have a great community of k8s-at-home folks running with Raspberry Pis) and a bunch of tooling to make bare metal easier with Cluster API. You can also do the minikube type of thing by running Talos directly in Docker or QEMU with `talosctl`. Talos works with an API instead of SSH/Bash, so there's some interesting things about ease of use when operating K8s that are baked in like built-in etcd backup/restore, k8s upgrades, etc. We're also right in the middle of building out our next release that will have native Wireguard functionality and enable truly hybrid K8s clusters. This should be a big deal for edge deployments and we're super excited about it.

c7DJTLrn 5y ago
A few questions if I may. I administer a few thousand machines which are provisioned using Puppet, and I've had my eye on a more immutable style of administration integrating K8S. - How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP. - How does Talos handle upgrades? Can it self upgrade once deployed? - What hardware can Talos run on? Does it work well with virtualisation? - To what degree can Talos dynamically configure itself? What I mean by this is if that a new disk is attached, can it partition it and start storing things on it? - How resilient is Talos to things like filesystem corruption? - What are the minimum hardware requirements? Please forgive my laziness but maybe other HNers will have the same questions.
- rsmitty 5y ago
  Hey, thanks for the questions. I'll try to answer them in-line: - How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP. For networking in particular, you can configure interfaces directly at boot by using kernel args. But that being said, Talos is entirely driven by a machine config file and there are several different ways of getting Talos off the ground, be it with ISO or any of our cloud images. Generally you can bring your own pre-defined machine configs to get everything configured from the start or you can boot the ISO and configure it via our interactive installer once the machine is online. We also have folks that make heavy use of Cluster API and thus the config generation is all handled automatically based on the providers being used. - How does Talos handle upgrades? Can it self upgrade once deployed? Upgrades can be kicked off manually with `talosctl` or can be done automatically with our upgrade operator. We're currently in the process of revamping the upgrade operator to be smarter, however so it's in flux a bit. As with everything in Talos, upgrades are controllable by the API. Kubernetes upgrades can also be performed across the cluster directly with `talosctl`. We’ve tried to bake in a lot of these common operations tasks directly into the system to make it easier for everyone. - What hardware can Talos run on? Does it work well with virtualisation? Pretty much anything ARM64 or AMD64 will work. We have folks that run in cloud, bare metal servers, Raspberry Pis, you name it. We publish images for all of these with each release. Talos works very well with virtualization, whether that's in the cloud or with QEMU or VMWare. We've got folks running it everywhere. - To what degree can Talos dynamically configure itself? What I mean by this is if a new disk is attached, can it partition it and start storing things on it? Presently, the machine configuration allows you to specify additional disks to be used for non-Talos functions, including formatting and mounting them. However, this is currently an install-time function. We will be extending this in the future to allow for dynamic provisioning utilizing the new Common Operating System Interface (COSI) spec. This is a general specification which we are actively developing both internally and in collaboration with interested parties across the Kubernetes community. You can check that out here if you have interest: https://github.com/cosi-project/community https://github.com/cosi-project/community - How resilient is Talos to things like filesystem corruption? Like any OS, filesystem corruption can indeed occur. We use standard Linux filesystems which have internal consistency checks, but ultimately, things can go wrong. An important design goal of Talos, however, is that it is designed for distributed systems and, as such, is designed to be thrown away and replaced easily when something goes awry. We also try to make it very easy to backup the things that matter from a Kubernetes perspective like etcd. - What are the minimum hardware requirements? Tiny. We run completely in RAM and Talos is less than 100MB. But keep in mind that you still have to run Kubernetes, so there's some overhead there as well. You’ll have container images which need to be downloaded, both for the internal Kubernetes components and for your own applications. We're roughly the same as whatever is required for something like K3s, but probably even a bit less since we don’t require a full Linux distro to get going.

weavie 5y ago

It hadn't occurred to me that there were multiple different kubernetes. I thought it was essentially just a single app, or rather collection of apps. Are these essentially different choices available for people who wish to self host their k8s clusters? Are there any good resources that summarise the different advantages of each choice? What is the same between all the choices, what are the differences?

seabrookmx 5y ago
They're Kubernetes distributions. So think of Kubernetes like Linux, and kubeadm, OpenShift, k3s like Ubuntu, Red Hat etc.
- weavie 5y ago
  What would be the equivalent of the kernel (the bit they all share)?
  - qbasic_forever 5y ago
    The k8s API: https://kubernetes.io/docs/concepts/overview/kubernetes-api/ https://kubernetes.io/docs/concepts/overview/kubernetes-api/ Most distributions will use the stock kubelet process (what runs containers on nodes), but you don't have to--there are kubelet compatible processes to run VMs instead of containers, run webassembly code (krustlet), etc. Everything in k8s can be swapped out and changed, the API spec is the only constant.
qbasic_forever 5y ago
Tons of choices, k8s is really just a definition of an API and there lots of things you can plug in at various points to implement parts of the API (or even completely extend it). Read the official docs--they'll walk you through the design and capabilities. As far as what's available, well it's a bit like trying to understand "what's the newest programming languge?" You probably want to plug yourself into the k8s community and start looking at subreddits, SIG meeting notes, community news, etc. New things are popping up literally every day.

jturpin 5y ago

I use MicroK8S for local testing of Kubernetes clusters on my laptop and it works pretty well. I like that I can just run microk8s reset to clear out the state so I can redeploy everything without fear of some lingering configuration laying behind. I have yet to deploy it to an actual server though, though I would definitely be interested if it could do what EKS can do to some capacity (mainly creating EBS volumes and load balancers).

mkesper 5y ago
Are you serious? Microk8s reset takes ages. It's faster to uninstall the whole thing. You also can just delete whole workspaces, that should normally be enough.
- achalkias 5y ago
  Indeed, sometimes it can be faster to nuke the cluster but this is also unsafe. microk8s reset asks Kubernetes to delete the resources created, and that can take time.
- jturpin 5y ago
  Yeah man, I'm serious. It took 5 minutes 30 seconds on my machine to run microk8s reset. This is well within the tolerance window for something I choose to run every 2 to 6 weeks. This is not part of my normal development strategy. Deleting whole namespaces is not a good idea for things like kube-system, ingress, etc. for obvious reasons.

Whitespace 5y ago

Unobtrusive version: https://microk8s.io/?cp=hide#cookies https://microk8s.io/?cp=hide#cookies

staticassertion 5y ago

If you want "k8s but less" I highly recommend Nomad from Hashicorp and the Hashi-stack in general. It's so damn good.

desktopninja 5y ago

Would be nice too if it could run non containerized applications like Nomad can.