5 ms·

It really doesn't take much experience with kubernetes before it becomes fairly easy. And then using it for trivial things like this isn't a big deal. My home

It really doesn't take much experience with kubernetes before it becomes fairly easy. And then using it for trivial things like this isn't a big deal. My home "server" runs plex, sab, mongo, unifi, and various other things in a single node k8s with a local zfs volume provisioner. The previous revision of this server I'd switched to using docker for everything and was annoyed with the upgrade process with just docker alone. With k8s, I just use :latest for most images and an upgrades happen every time I restart a pod or reboot the machine. I've been working with k8s since petSets and so this is all NBD to me. Linux had a learning curve too and ya'll got over it.

tenacious_tuna 6y ago
A friend and I have recently stood up homelabs for funsies. Mine's a cluster of x64 boxes, his is a small pile of raspi4s. He went for a k8s cluster off the bat--having worked with cloud stuff for a while, he had the familiarity to hit the ground running. I was immediately overcome by all the abstractions. I had no idea where to look to figure things out, and was mostly relying on advice from my friend. I dind't know what an ingress controller was, much less how to configure one--but I knew I wanted each 'service' to have its own IP I could route to from my network. Overall it felt like I had SO MUCH TO LEARN at *every point* it was difficult to get even close to my actual goals (CI/CD for a personal site + some hosted game servers). I eventually went with the philosophy of "build something now, and move towards perfection later" / "don't let best be the enemy of good" and begain spooling up LXCs and VMs to do the work I needed, planning to move things into k8s later when I better understood the actual things I wanted to move. (Plus then I got some satisfaction out of actually accomplishing the goals I wanted, instead of just banging my head on k8s documentation and learning all the abstractions.) As an example, I've not used docker in any meaningful capacity. For anything. No idea how to make a docker image. To k8s the CI for my docker site, I needed to know how to : 1. install the dependencies, which requires compiling a plugin for pandoc, which requires installing haskell and cabal. this is expensive, so I'd prefer to get the pre-res set up once... but that doesn't seem to be how docker works? Do I need an image repository? can I use DockerHub? I've seen HN talk about how docker is trying to monetize, should I run my own repository? Can I do that on my cluster? I'll need an ingress controller to route to it... I don't even know what that is. 2. I need some way to pass the built website files to the contianer I actually want to host them. I think that means I need an NFS share of somekind to store the files, so one coantainer can load them and another can read them. Do I hos tthat on my NAS? I could put an NFS share in the cluster, maybe. No idea how to get Docker to mount one, or k8s to host one. All the examples I seem to find deal more with connecting to remote services on a host than mounting local storage. is it even local storage? 3. Everyone says infrastructure as code is good, so I guess I'll follow this flux tutorial--only to find out the one I followed is out of date, and I should follow their NEW one. But they still assume I know way more baout k8s than I actually do, but still, I'll spend the few hours to get this operational, so then in theory everything else I deploy can be IaC'd, which is just good practice. At this point I'm so many layers of abstraction deep, I have no idea what I'm actually doing or how concepts relate to each other, and I'm no closer to actually having my goal. So last night I spent 2 hours spinning up VMs on my cluster and installing dependencies, configuring an nginx proxy, and now I actually have my personal blog self-hosted and updatable. Way more "progress" than the 10ish hours I've sunk into building a k8s cluster already. There's something to be said for limiting the number of abstractions you're dealing with.
- nrmitchi 6y ago
  > I didn't know what an ingress controller was I've seen this comment a lot when discussion introductions to Kubernetes, and it is probably one of the biggest "first step" problems I see, and a perfectly legitimate complaint. If there was one thing that Kubernetes could address to help "onboarding", it might be thing. Kubernetes is essentially a collection of controllers, that each control a different aspect of the system. These are all "internal", and you don't need to know/understand them in order to stand up Kubernetes in the happy-path. If you're using a managed solution, these are all managed by your provider anyways. The ingress controllers are the exception. It brings the concept of "controller" out of the "Kubernetes administration" space, and into the "Kubernetes user" space. It opens up a whole can of worms around "What exactly is a controller?" that you shouldn't have to understand in order to get started using Kubernetes.
  - tenacious_tuna 6y ago
    Part of the issue, I suspect, is I was blending k8s use and k8s administration by running my own k8s cluster on my own equipment--that's likely an extra layer of complexity. (of course, the whole point of the homelab endeavor I set out on was to be fully self-hosted to learn all these underlying concetps--but definitely doesn't help that I layered some extra abstractions on top of the soup that k8s already is.)
- uberduper 6y ago
  You've listed a dozen or so other technologies you've had to learn in the past, which in aggregate required much more effort and experience to master. Much of which provides a good foundation for understanding the concepts that kubernetes builds upon. I feel like this was more of a hyperbolic rant.
  - tenacious_tuna 6y ago
    > I feel like this was more of a hyberbolic rant. Reasonable--it definitely is a bit of a rant. I'm not sure it's hyperbolic, personally, but clearly I have a bias. Overall, my acute frustration with k8s coming from a native-destkop-development background is I simply don't have the scaffolding to be effective relatively quickly. The learning curve is steep enough that I get discouraged before I actually begin making progress on my own goals, and I just feel like I'm trying to do things The Right Way, without understanding what I'm doing, or why, or actually accomplishing my original goals. As you said, this is true of concepts I've had to learn in the past, but I could learn all those concepts in isolation, then apply them together. K8s I feel like I have to understand a much larger chunk of before I hit critical mass and can start being effective. i.e. python, I can start with, like, sqlite, before I move to an external hosted DB. K8s I feel like I have to understand waaaay more components--and they don't directly build. Like Docker-compose _seems like_ a stepping stone to k8s, but I've been told is a false path.
nonameiguess 6y ago
You can use Flux and it will just poll your source control and image registries to see if any updates to the images or configs are available and reconcile automatically. No need to even restart pods. Totally commodity continuous deployment. It's wonderful.
tail_exchange 6y ago
Kubernetes is incredibly daunting for someone who only used VMs, but after spending some time with it, it's not really that hard. I also unironically run my personal stuff on K8s just because it's familiar to me, and it is also a good exercise for a very useful skill. People complain about the price, but you can easily get a 3-node cluster on GCP for less than $15/month.
- Hamuko 6y ago
  Yeah, I've been learning Kubernetes recently and a simple setup isn't too difficult. However, there's a lot of concepts involved with Kubernetes to grasp and I've definitely not learned everything there is. For example, Jobs still baffle me in regards to what is the purpose outside of CronJobs.
  - mpb88 6y ago
    I use Jobs for helm hooks like integration testing and DB migrations associated with deployments. They work great for one off workloads like that.
    - thedevelopnik 6y ago
      You just beat me to it!
  - thedevelopnik 6y ago
    Jobs can get useful in more complex workflows. The most common use I’ve had for them us running a smoke test suite or a database migration as part of the larger workflow for a big Rails app.
    - Hamuko 6y ago
      But why wouldn't I just run a database migration as part of my CI flow? I don't understand why I would add a persistent element to my Kubernetes cluster for just a one-off task, since Jobs do not actually automatically remove themselves from the cluster. And as far as I know, you can't even reuse the already created Job, so if I want to migrate the database again, I need to make it from scratch.
imiric 6y ago
Did you use Docker Swarm in the previous revision? I've been quite happy with it for a few years now on a single-node cluster and a handful of services. I could easily add another node, but haven't had the need to. The setup was quite simple and it requires practically no maintenance. If I have to reboot everything starts up automatically, image upgrades are a breeze, and I really have no issues with it. Sure it doesn't have all the bells and whistles of a k8s cluster, but it's perfectly fine for personal use. I'm still partly annoyed that Swarm is mostly dead in this space and k8s has undoubtedly "won". It's only a matter of time before Docker Inc. fully abandons it. Such a shame.
- deleted 6y ago
  [deleted]
- KronisLV 6y ago
  Agreed, while i've dabbled in Kubernetes, most of my homelab and hybrid cloud infrastructure (think ~10-20 nodes) runs in Docker Swarm with Portainer for graphical administration. A while back i also tried Nomad, and while it felt polished, it was also a bit lacking in features (manual encryption config, mostly read-only UI, though that was a while back). It's a shame that Docker Swarm never got more popular, because in my mind it's basically the sweet spot between running containers directly or using Docker Compose and full blown Kubernetes distributions. The Compose manifest format that it uses is really great and clear in my mind, instead of Kubernetes with all of its selectors, object types and so on.

tenacious_tuna 6y ago

A friend and I have recently stood up homelabs for funsies. Mine's a cluster of x64 boxes, his is a small pile of raspi4s. He went for a k8s cluster off the bat--having worked with cloud stuff for a while, he had the familiarity to hit the ground running. I was immediately overcome by all the abstractions. I had no idea where to look to figure things out, and was mostly relying on advice from my friend. I dind't know what an ingress controller was, much less how to configure one--but I knew I wanted each 'service' to have its own IP I could route to from my network. Overall it felt like I had SO MUCH TO LEARN at *every point* it was difficult to get even close to my actual goals (CI/CD for a personal site + some hosted game servers). I eventually went with the philosophy of "build something now, and move towards perfection later" / "don't let best be the enemy of good" and begain spooling up LXCs and VMs to do the work I needed, planning to move things into k8s later when I better understood the actual things I wanted to move. (Plus then I got some satisfaction out of actually accomplishing the goals I wanted, instead of just banging my head on k8s documentation and learning all the abstractions.) As an example, I've not used docker in any meaningful capacity. For anything. No idea how to make a docker image. To k8s the CI for my docker site, I needed to know how to : 1. install the dependencies, which requires compiling a plugin for pandoc, which requires installing haskell and cabal. this is expensive, so I'd prefer to get the pre-res set up once... but that doesn't seem to be how docker works? Do I need an image repository? can I use DockerHub? I've seen HN talk about how docker is trying to monetize, should I run my own repository? Can I do that on my cluster? I'll need an ingress controller to route to it... I don't even know what that is. 2. I need some way to pass the built website files to the contianer I actually want to host them. I think that means I need an NFS share of somekind to store the files, so one coantainer can load them and another can read them. Do I hos tthat on my NAS? I could put an NFS share in the cluster, maybe. No idea how to get Docker to mount one, or k8s to host one. All the examples I seem to find deal more with connecting to remote services on a host than mounting local storage. is it even local storage? 3. Everyone says infrastructure as code is good, so I guess I'll follow this flux tutorial--only to find out the one I followed is out of date, and I should follow their NEW one. But they still assume I know way more baout k8s than I actually do, but still, I'll spend the few hours to get this operational, so then in theory everything else I deploy can be IaC'd, which is just good practice. At this point I'm so many layers of abstraction deep, I have no idea what I'm actually doing or how concepts relate to each other, and I'm no closer to actually having my goal. So last night I spent 2 hours spinning up VMs on my cluster and installing dependencies, configuring an nginx proxy, and now I actually have my personal blog self-hosted and updatable. Way more "progress" than the 10ish hours I've sunk into building a k8s cluster already. There's something to be said for limiting the number of abstractions you're dealing with.

nrmitchi 6y ago
> I didn't know what an ingress controller was I've seen this comment a lot when discussion introductions to Kubernetes, and it is probably one of the biggest "first step" problems I see, and a perfectly legitimate complaint. If there was one thing that Kubernetes could address to help "onboarding", it might be thing. Kubernetes is essentially a collection of controllers, that each control a different aspect of the system. These are all "internal", and you don't need to know/understand them in order to stand up Kubernetes in the happy-path. If you're using a managed solution, these are all managed by your provider anyways. The ingress controllers are the exception. It brings the concept of "controller" out of the "Kubernetes administration" space, and into the "Kubernetes user" space. It opens up a whole can of worms around "What exactly is a controller?" that you shouldn't have to understand in order to get started using Kubernetes.
- tenacious_tuna 6y ago
  Part of the issue, I suspect, is I was blending k8s use and k8s administration by running my own k8s cluster on my own equipment--that's likely an extra layer of complexity. (of course, the whole point of the homelab endeavor I set out on was to be fully self-hosted to learn all these underlying concetps--but definitely doesn't help that I layered some extra abstractions on top of the soup that k8s already is.)
uberduper 6y ago
You've listed a dozen or so other technologies you've had to learn in the past, which in aggregate required much more effort and experience to master. Much of which provides a good foundation for understanding the concepts that kubernetes builds upon. I feel like this was more of a hyperbolic rant.
- tenacious_tuna 6y ago
  > I feel like this was more of a hyberbolic rant. Reasonable--it definitely is a bit of a rant. I'm not sure it's hyperbolic, personally, but clearly I have a bias. Overall, my acute frustration with k8s coming from a native-destkop-development background is I simply don't have the scaffolding to be effective relatively quickly. The learning curve is steep enough that I get discouraged before I actually begin making progress on my own goals, and I just feel like I'm trying to do things The Right Way, without understanding what I'm doing, or why, or actually accomplishing my original goals. As you said, this is true of concepts I've had to learn in the past, but I could learn all those concepts in isolation, then apply them together. K8s I feel like I have to understand a much larger chunk of before I hit critical mass and can start being effective. i.e. python, I can start with, like, sqlite, before I move to an external hosted DB. K8s I feel like I have to understand waaaay more components--and they don't directly build. Like Docker-compose _seems like_ a stepping stone to k8s, but I've been told is a false path.

nonameiguess 6y ago

You can use Flux and it will just poll your source control and image registries to see if any updates to the images or configs are available and reconcile automatically. No need to even restart pods. Totally commodity continuous deployment. It's wonderful.

tail_exchange 6y ago

Kubernetes is incredibly daunting for someone who only used VMs, but after spending some time with it, it's not really that hard. I also unironically run my personal stuff on K8s just because it's familiar to me, and it is also a good exercise for a very useful skill. People complain about the price, but you can easily get a 3-node cluster on GCP for less than $15/month.

Hamuko 6y ago
Yeah, I've been learning Kubernetes recently and a simple setup isn't too difficult. However, there's a lot of concepts involved with Kubernetes to grasp and I've definitely not learned everything there is. For example, Jobs still baffle me in regards to what is the purpose outside of CronJobs.
- mpb88 6y ago
  I use Jobs for helm hooks like integration testing and DB migrations associated with deployments. They work great for one off workloads like that.
  - thedevelopnik 6y ago
    You just beat me to it!
- thedevelopnik 6y ago
  Jobs can get useful in more complex workflows. The most common use I’ve had for them us running a smoke test suite or a database migration as part of the larger workflow for a big Rails app.
  - Hamuko 6y ago
    But why wouldn't I just run a database migration as part of my CI flow? I don't understand why I would add a persistent element to my Kubernetes cluster for just a one-off task, since Jobs do not actually automatically remove themselves from the cluster. And as far as I know, you can't even reuse the already created Job, so if I want to migrate the database again, I need to make it from scratch.

imiric 6y ago

Did you use Docker Swarm in the previous revision? I've been quite happy with it for a few years now on a single-node cluster and a handful of services. I could easily add another node, but haven't had the need to. The setup was quite simple and it requires practically no maintenance. If I have to reboot everything starts up automatically, image upgrades are a breeze, and I really have no issues with it. Sure it doesn't have all the bells and whistles of a k8s cluster, but it's perfectly fine for personal use. I'm still partly annoyed that Swarm is mostly dead in this space and k8s has undoubtedly "won". It's only a matter of time before Docker Inc. fully abandons it. Such a shame.

deleted 6y ago
[deleted]
KronisLV 6y ago
Agreed, while i've dabbled in Kubernetes, most of my homelab and hybrid cloud infrastructure (think ~10-20 nodes) runs in Docker Swarm with Portainer for graphical administration. A while back i also tried Nomad, and while it felt polished, it was also a bit lacking in features (manual encryption config, mostly read-only UI, though that was a while back). It's a shame that Docker Swarm never got more popular, because in my mind it's basically the sweet spot between running containers directly or using Docker Compose and full blown Kubernetes distributions. The Compose manifest format that it uses is really great and clear in my mind, instead of Kubernetes with all of its selectors, object types and so on.