12 ms·
Domesticating Kubernetes
- sdan 7y agoI can attest to RPI i/o speed: its horrible. Combined with the fact that I have to build Docker containers on the RPIs themselves(because of arm), its more of a hassle than a cool add-on (to be clear I'm running Docker Swarm in a way that isn't too different from the setup of OP). Only reason I can see RPIs in kubernetes is if you're exclusively using ARM everywhere and/or are running some distributed cluster among different locations (like Chick-fil-a).
- GordonS 7y agoYou can actually build ARM images on your x64 machine using newer versions of Docker with BuildKit built into it. I think it uses QEMU under the hood. I recently did this at work, using a single Dockerfile to built multi-arch images for ARM and x64 - it's pretty nice!
- sdan 7y agoPreviously I've done this, but found it takes a bunch of time. I'll give it another go, though and see how it goes!
- aequitas 7y agoYou don't have to build on the RPi's themselves: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images https://docs.docker.com/buildx/working-with-buildx/#build-mu...
- charlieegan3 7y agoI use drone.io to build all my arm images for my pi cluster.
- globular-toast 7y agoDon't build stuff on the Pi! You just need to set up a cross-compiler toolchain and compile it on your much more capable AMD64 machine.
- reaktivo 7y agoThis is not much of a problem if you switch out of the 32bit Raspbian. Most images will work out of the box on ARM64
- wlll 7y agoIs there an RPi alternative (ie. similar size, form etc.) that has better IO/Network speed? I used RPis for my robot (https://sendc.at/dl/Kjjbt3dij6T733YgWsyv3p3Vv5GPGKO1q5IEFStVngrkI6EVupm8dtT https://sendc.at/dl/Kjjbt3dij6T733YgWsyv3p3Vv5GPGKO1q5IEFStV...) with daughter boards for motor control) and they wre pretty convenient, but never really considered running anything server-like on them because they have reportedly woeful network performance.
- dabeeeenster 7y agoI have this https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram/ https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram/ It's pretty powerful for the cost and has no fan etc. Arm64.
- wlll 7y agoThose look sweet, thanks!
- jsight 7y agoThe pi 4 has gigabit ethernet. I'm not sure why you'd think it has "woeful network performance". The 3B+ could only do a couple hundred megabit, which was somewhat mediocre, though.
- wlll 7y agoIf they've fixed it with the 4 then that's great, ISTR in previous versions the Ethernet was backed by the shared USB controller (https://raspberrypi.stackexchange.com/questions/45130/why-do-the-usb-ports-and-ethernet-port-share-the-same-controller https://raspberrypi.stackexchange.com/questions/45130/why-do...), hence my question.
- jsight 7y agoThe shared USB controller wasn't the issue, the lack of USB 3 was.
- louwrentius 7y agoThe RPi 4 can saturate gigabit with ease, it's in a different league compared to the RPi 3 and lower, avoid those like the plague.
- sandGorgon 7y agoyou can run a kubernetes stack using k3s on a rpi stack within minutes. and it runs far better than all of these. https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ here's a live walkthrough - https://www.youtube.com/watch?v=DjpVtNjiXSU https://www.youtube.com/watch?v=DjpVtNjiXSU
- detaro 7y ago> and it runs far better than all of these. in what way?
- pas 7y agok3s is a lightweight stripped down version of k8s, so it's smaller and faster, and tuned for more resource scarce environments, like an RPi.
- rkachowski 7y agoThe author mentions the differences between k3s and micro k8s in the post and concludes that due to the quirks of k3s (subset of features running in a single binary) it's only really preferable if you want to run rancher.
- sandGorgon 7y agoThat is not true. We have personally run k3s on AWS including cluster-scaler and spot instances. Last I checked, K3s is a fully certified distro of k8s.
- breakingcups 7y agoThe author disagrees with you on both points, he asserts rpi's are way slower than "old" pc's and laptops and that MicroK8S is preferable over K3S. He actually provides arguments & benchmarks to back up those opinions. Could you do the same for yours?
- sandGorgon 7y ago
- tachion 7y agoWhat I miss from all of these tutorials is one very important piece - how to handle network routing and dns automation within your home network, that's in typical scenario is being handled by the ingress/cloud controller. Without having automated (or easy enough) way of reaching the apps you're deploying there, each of these clusters is pretty much useless for users except for maybe learning basics of k8s, what's easier done with minikube.
- charlieegan3 7y agoI use dynamic DNS with my Edgemax router. Something like inlets operator or cloudflare Argo are other options.
- jimmcslim 7y agoI think the OP might be more asking about how to get local DNS automatically provisioned with the IP address of a service/container that has been deployed to the LAN.
- tachion 7y agoThat's exactly what I meant. I know how to do it myself, but then again I'm not the target for these articles, but when I read them with the mindset of someone who is supposed to have a use of them, that's always the big missing bit for me.
- jimmcslim 7y agoDid davestephen's comment help though? Apart from suggesting a .local domain... that's a bad idea, try .lan instead. Then you could type (for Plex, for example) plex.k8s.lan Maybe we need the equivalent of traefik for DNS? https://news.ycombinator.com/item?id=23040478 https://news.ycombinator.com/item?id=23040478
- tachion 7y agoI know how to solve this problem, so I wasn't commenting on his comment, but rather on the fact that none of these tutorials so far solve this problem for new users that they target. Aside from that, I don't think what he proposes is a great solution, because what we'd need is the automated way for deployments to get that DNS created (or announced) for their IP when they get it or when it changes. Having it done manually and being static is vastly different in usability from k8s does with ingress/cloud controllers.
- rtempaccount1 7y agoI think what type of k8s environment you use very much depends on what you're looking to get out of it. If it's experience deploying applications into containerized environments, then micro-k8s and k3s seem like reasonable choices, you don't really care about the setup of the underlying components, just that they present the k8s API. If you're looking for experience of managing k8s clusters, then either the distribution you're looking to run in prod. or something like kubeadm are perhaps a better option. kubeadm is very "vanilla" in terms of how it's deployed so it's quite representative of production (on-prem) deployments, perhaps unlike k3s which makes changes to how k8s works. If you're looking to quickly test things in k8s, I'd recommend kind as the easiest way to stand up and remove clusters quickly. And if you're looking for something to run your home services long term, I would recommend not using Kubernetes :) (unless you have a really complex home network which might justify adding k8s to the mix)
- charlieegan3 7y ago> if you're looking for something to run your home services long term, I would recommend not using Kubernetes If we disregard one’s experience with Kubernetes as a factor, are there any other reasons you see to not use k8s at home?
- vetinari 7y agoDo you really have that many computers at home, so you need container orchestration for that? A single machine is capable of handling most home workloads. So outside of playing with Kubernetes for experience, why would you do that?
- tuananh 7y agoi have to agree with you on this. i've been using k8s at work for over 4 years but operating k8s for home workload is still too much; unless, it's for tinkering/testing.
- rtempaccount1 7y agocomplexity and overhead. Maintaining Kubernetes is non-trivial (IMO ofc). It has a 9 month support lifecycle, so you need to redeploy everything regularly. There are also API deprecations to deal with periodically, which if you're not keeping up with k8s developments, could be disruptive. From a complexity perspective, you're adding more places for things to break. Instead of (say) running your apps in containers with Docker, you add pods, services and ingress as layers, which is more places for things to go wrong.
- guiriduro 7y agoOr... you could just install a Docker swarm in 10 minutes https://www.bretfisher.com/10-minutes-to-highly-available-docker/ https://www.bretfisher.com/10-minutes-to-highly-available-do... No good reasons to use K8s outside multi-team enterprise stateful multi-tier architectures.
- guiriduro 7y ago...over simpler approaches, I should qualify.
- charlieegan3 7y agoKubernetes provides a standard way to deploy applications. If I’m familiar with deploying applications on Kubernetes at work, why not have the same for my side projects at home?
- politelemon 7y agoThe same can be said of deployment mechanism that one is accustomed to. GP's point is around time to setup and complexity involved.
- sdan 7y agoSwarm is dead in the water. No progress and there's a ton of basic features that aren't being supported. In my use I have to literally remove every single node and reset the cluster if 1 goes down (I run into some weird grpc error via swarm). Combined with weak GPU support.... its there for basic use, but not for anything "intermediate" or higher.
- tr33house 7y agoHappy to see this. I think more orgs should be running more on-prem or colocated in a data center somewhere. Public clouds can be a rip off for larger projects
- mrweasel 7y agoThere’s a massive difference between running a Kubernetes setup, like the one in the article, for you home lab, and the attempting to run Kubernetes on-prem for a business. I would recommend against the latter, unless your sure you can afford it, in term of staffing.
- rb808 7y agoI've given up running my own k8s at home a couple of times. It does seem to be getting easier, but then upgrading and maintaining breaks me again. + the other stuff like running your own container registry. Then with new centos podman breaks everything that worked OK in docker. I hate this stuff, so many problems its worth ignoring the whole stack.
- StreamBright 7y agoI do not blame you. I try to avoid it as long as it is possible. The amount of time, energy and effort wasted on this is just insane.
- brianwawok 7y agoHosted k8s has changed my life. Let me focus on my business and not my infra.
- StreamBright 7y agoHave you had any serious yet? If you are on the happy path does not really matter what you use.
- brianwawok 7y agoAny serious what? I am running a mildly successful SaaS business on hosted kubernetes, yes.
- StreamBright 7y agoSorry, HN made me retry couple of times. Any serious troubles with Kubernetes. Have you had any of those?
- brianwawok 7y agoNo I have not. Going on 3 years now?
- tuananh 7y agoquestion: if not k8s, what everyone has been using to orchestrate containers at home? Ideally, I just want something easy to setup, low maintenance cost and easy to backup and restore if sth went wrong. i've been using k8s at work for over 4 years but for home usage, it's just too much.
- v77 7y agodocker-compose
- tuananh 7y agobut docker swarm is kind of dead, isn't it? i need to orchestrate on multiple hosts.
- dantheman0207 7y agoNomad
- StavrosK 7y agoCan you post a short, high-level overview of how you use Nomad? What does it do for you exactly?
- StavrosK 7y agohttp://dokku.viewdocs.io/dokku/ http://dokku.viewdocs.io/dokku/
- NickBusey 7y agoHomelabOS
- deleted 7y ago[deleted]
- oppositelock 7y agoWhy would you do this for simple home apps? k8s is complete overkill. At home, there is little reason to drag in all that complexity to host a blog or whatever. Kubernetes drags in a tremendous amount of complexity, background knowledge, and strange constraints, even on small installations. I really do not see its benefit for small apps and projects. Are you hosting your blog? You don't need k8s. Are you running a small app server at home? Nope, don't need it. Are you running an auto-scaling app with many hundreds or thousands of worker instances, and need to support non-disruptive roll-in upgrades across multiple clouds and have many services which need to scale independently? Maybe you need k8s, but only in some circumstances. For the record, I run many k8s clusters across several clouds, using 1,000 - 50,000 cores at any one time, so I've dealt with quite a bit of k8s complexity, and I'm still on the fence whether it's the right answer for us, but it's allowed us to standardize our software to k8s and only worry about getting that running well in each deployment, which put the cross-cloud work on the infrastructure teams versus the software development teams. The price you pay for this is that you still need to do non-k8s stuff if you need some cloud specific resources more than simple compute and routing, so you have both k8s and cloud specific code.
- Bombthecat 7y agoLearning?
- oppositelock 7y agoThat is a fine reason!
- brooksgarrett 7y agoI'll second you. I full time admin our K8s infrastructure and clusters. It's a great solution for our production workloads and I've bootstrapped RancherOS/k3s a handful of times to look at using it at home but K8s is purpose built for cloud. I can get all the same benefits with significantly reduced complexity with Docker + Systemd at home.
- MuffinFlavored 7y ago
- soulnothing 7y agoI recently gave up on home network clustering/kubernetes. I've since moved to digital ocean. I was working on data heavy apps for testing. I had several SSD zfs pools which was the driving reason, and a lot of RAM to work with. My setup did work initially. I had a dedicated server acting as the metal lb, about 20$ a month. That was then connected through a wireguard tunnel to my home dmz network. Which backed to a dual xeon v2 work station. The latency was very good under 20ms, and really good speeds. I'm lucky to have fios in my area. The Xeon workstation died, I then backed up to several think pads. Those were not performant either. So I got several HP t610 nodes, raspberry pi speed with sata 3. Rook took all CPU, and to boot they were running at 90c constantly. Even after re paste and with fan mods. I didn't want five little space heaters next to my desk. After all this I ditched the home setup. I had gotten parts for a local Epyc server after my xeon's died. But sold them due the current situation. In the end I had wanted to start a series of blogs on Kubernetes and micro service development. To help me learn and flesh out my under standing of Kubernetes. I don't feel I wasted several months setting up Kube. I now know about under the cover stuff. Having deployed OkD, Rancher, kubespray, and kube adm. The initial wire guard set up helped cement a lot of the internal networking model. I was mildly acquainted with having worked on openvswitch and open stack before. If you're doing local. I would really recommend a Ryzen 1600af (6c/12t zen+ 65w), either asrock rack x470, or a basic b450. Both can take ECC, and should land a little node under 500ish or so. There are also SFF PCs Lenovo ThinkCentre M93, comes to mind. But at that price 90$ a node, I'd rather move up the stack a little bit. If you're waiting to buy local. DigitalOcean has been very well priced. If you want to learn the internals, grab a dedicated server and set up KVM, to get familiar with the internals. On the upper end you can get a new ryzen 3rd gen dedicated for 90$ a month. I look at it as a 90$ class I take once. I'm not saying Kube is always right. But I avoided it for a long time, backing up to docker swarm. Now, I feel at a base it's not much more complicated than docker swarm, now that I've done this deep dive. Kube adm is on ease of use with docker swarm to me. Add in Metal LB, and Gitlab to help manage the cluster, you have a personal little cloud. It's also good to know for future job searches.
- zerpelin68 7y agoAll of these trap your mind in the k8s spiral arm and watch how adroit I am at building something no one would want to troubleshoot articles are hipster reminders of why I only create occasional throwaways for this site.
- nojvek 7y agoI’ve spent a decent amount of time fighting k8s at work. The GCP k8s which abstracts away a bunch of things, but still it can be a bit crazy. Docker compose is really great but only works on a single machine. Also docker compose makes rolling updates a PITA. What I really want is something like Cloud Run but locally tunable. Google Cloud Run is essentially: here is a docker image, when you get a http request, spin up a container, when there are no requests for 10 minutes, spin it down. If there are more than 80 req/s, replicate to handle the spike. Now I want to say, here’s a bunch of Ubuntu machines with X cores, Y RAM and Y SSD storage. Go run these images on it and auto scale it up and down. I know k8s was meant to solve this problem but the layers of abstraction on top of abstractions are insane. I just want to specify my compute/storage pool of machines, my docker images, how they should connect to each other and scale up/down. Boom! It just works. Is there something that does this?
- maztaim 7y agoCurios how the benchmark compares when adding SSD on RPI 4? https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd-flash-drives/ https://jamesachambers.com/raspberry-pi-4-usb-boot-config-gu... I don't think it beats anything, but I am sure IO improves fairly significantly. One of the things I found to be a problem is that most container images found on different registries are built for x86_64. You would need to rebuild those containers yourself on the RPI.