4 ms·

A lot of the conversation about "simplicity" with kubernetes ignores that there are MANY different distributions of k8s. I would argue that k3s on a vm is not

A lot of the conversation about "simplicity" with kubernetes ignores that there are MANY different distributions of k8s. I would argue that k3s on a vm is not too much harder to setup than a normal reverse proxy setup. And the configs are a bit easier to find on a whim when coming back after months. Obviously a full self hosted setup of k8s is pretty large task but in other forms its a competitive option in the "simple" area.

marcyb5st 1y ago
Last time I tried what stumped me was getting "ReadWriteMany" volumes working. My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them. I tried Longhorn, but even if I manage to get a volume showing up in the UI, it was never detected as healthy and after a while I refactored the workflow to use Apache Beam so that I could drop databases and volumes and run everything from within a single beefy VM. Is it still an issue (it was a while ago TBH)?
- JeffMcCune 1y ago
  Use a bucket for this. Or if you’re in a home lab use a boring old NFS server outside k8s
  - marcyb5st 1y ago
    Thanks. The Apache beam code in the end doesn't need a bucket but the HDD attached to the VM was more than enough to store everything and then moving things out of it as the final step.
- acedTrex 1y ago
  Ya this kind of usecase is for sure straying into the "this is more of a pain than its worth" with regards to k8s. Esp when you can just do basic nfs mounts in nix anyways. If you are already familiar with k8s volumes and csi's its not a huge problem but if you arent its not worth learning if your goal is simple. At least in my opinion.
  - marcyb5st 1y ago
    Thank you for taking the time to confirm this. Yeah, that is also my opinion after that experience. Also, it was really frustrating as the number of moving parts that have to play nice to each other is high.
    - acedTrex 1y ago
      K8s and its derivatives ultimately were designed to declaratively orchestrate stateless apis/middlewares. It of course can quite competently do many other things. But it does what it was BUILT to do originally the best and the simplest. Many hobby projects are just that, simple apis and can benefit from the simple declarative yamls that exist in say a sqlite db on the vm vs random config in /etc, /var and ip table scripts etc. But trying to do anything out of that simple mold? just do it the simpler and easier traditional way.
packetlost 1y ago
K8s is chock full of complexity though. Complexity that the vast majority of applications do not, and will not need
- frollogaston 1y ago
  I only used it once forever ago, and all I remember is being very confused about pod vs service and using a lot of magic strings to get load-balancing to work. Sure I could figure it out, but didn't want to.
  - pnt12 1y ago
    Kubernetes is really complex but I'm surprised by this - for a simple setup, I think those 2 resources are not that difficult. I'd describe a really simple setup as this: Pod: you put 1 container inside 1 pod - you can basically replace the word "container" with "pod". Let's say you have 1 backend in python and 1 frontend in React: you deploy 1 pod for your backend, and 1 pod for your frontend. The simplest way to deploy pods is not with "Pod" but with "Deployment" (1). Service: Kubernetes really doesn't want you to connect directly to pods, it wants you to have a service in between. This is sensible because pods are ephemeral - they can be killed due to crashes, updates, etc. You define a service, point towards the pod, and all internal requests should be sent to the service, not to the pod. (2) (1) https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ https://kubernetes.io/docs/concepts/workloads/controllers/de... (2) https://kubernetes.io/docs/concepts/services-networking/service/ https://kubernetes.io/docs/concepts/services-networking/serv... Note: technically, a pod is a group of containers. It can get more complex than this, but it's not unusual to have a group of 1 pod, especially in simpler setups.
    - frollogaston 1y ago
      The thing about managing pods using deployments rings a bell, but most official Kubernetes tutorials/examples (at least back then) were declaring "kind: Pod" directly which gave mixed signals, maybe because deployment was a newer feature. Per the service doc you linked, services connect directly to pods rather than deployments, using a seemingly magic "kubernetes.io/service-name" label. In the deployments example, what gets me is they called 4 different things "nginx." It's hard to tell what does what. Like I've read 6 times what matchLabels does, and it's still not clear if that's supposed to mirror template.metadata.labels.app, or what happens if it doesn't. Sure I'd figure it out if I set up the real thing and messed with the config to see what it does, but it shouldn't take that. Of course it's easy to follow tutorials without fully understanding how stuff works, and I did. Slightly more advanced things were a lot of trial and error. At some point I lost interest because I didn't really need this.

marcyb5st 1y ago

Last time I tried what stumped me was getting "ReadWriteMany" volumes working. My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them. I tried Longhorn, but even if I manage to get a volume showing up in the UI, it was never detected as healthy and after a while I refactored the workflow to use Apache Beam so that I could drop databases and volumes and run everything from within a single beefy VM. Is it still an issue (it was a while ago TBH)?

JeffMcCune 1y ago
Use a bucket for this. Or if you’re in a home lab use a boring old NFS server outside k8s
- marcyb5st 1y ago
  Thanks. The Apache beam code in the end doesn't need a bucket but the HDD attached to the VM was more than enough to store everything and then moving things out of it as the final step.
acedTrex 1y ago
Ya this kind of usecase is for sure straying into the "this is more of a pain than its worth" with regards to k8s. Esp when you can just do basic nfs mounts in nix anyways. If you are already familiar with k8s volumes and csi's its not a huge problem but if you arent its not worth learning if your goal is simple. At least in my opinion.
- marcyb5st 1y ago
  Thank you for taking the time to confirm this. Yeah, that is also my opinion after that experience. Also, it was really frustrating as the number of moving parts that have to play nice to each other is high.
  - acedTrex 1y ago
    K8s and its derivatives ultimately were designed to declaratively orchestrate stateless apis/middlewares. It of course can quite competently do many other things. But it does what it was BUILT to do originally the best and the simplest. Many hobby projects are just that, simple apis and can benefit from the simple declarative yamls that exist in say a sqlite db on the vm vs random config in /etc, /var and ip table scripts etc. But trying to do anything out of that simple mold? just do it the simpler and easier traditional way.

packetlost 1y ago

K8s is chock full of complexity though. Complexity that the vast majority of applications do not, and will not need

frollogaston 1y ago
I only used it once forever ago, and all I remember is being very confused about pod vs service and using a lot of magic strings to get load-balancing to work. Sure I could figure it out, but didn't want to.
- pnt12 1y ago
  Kubernetes is really complex but I'm surprised by this - for a simple setup, I think those 2 resources are not that difficult. I'd describe a really simple setup as this: Pod: you put 1 container inside 1 pod - you can basically replace the word "container" with "pod". Let's say you have 1 backend in python and 1 frontend in React: you deploy 1 pod for your backend, and 1 pod for your frontend. The simplest way to deploy pods is not with "Pod" but with "Deployment" (1). Service: Kubernetes really doesn't want you to connect directly to pods, it wants you to have a service in between. This is sensible because pods are ephemeral - they can be killed due to crashes, updates, etc. You define a service, point towards the pod, and all internal requests should be sent to the service, not to the pod. (2) (1) https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ https://kubernetes.io/docs/concepts/workloads/controllers/de... (2) https://kubernetes.io/docs/concepts/services-networking/service/ https://kubernetes.io/docs/concepts/services-networking/serv... Note: technically, a pod is a group of containers. It can get more complex than this, but it's not unusual to have a group of 1 pod, especially in simpler setups.
  - frollogaston 1y ago
    The thing about managing pods using deployments rings a bell, but most official Kubernetes tutorials/examples (at least back then) were declaring "kind: Pod" directly which gave mixed signals, maybe because deployment was a newer feature. Per the service doc you linked, services connect directly to pods rather than deployments, using a seemingly magic "kubernetes.io/service-name" label. In the deployments example, what gets me is they called 4 different things "nginx." It's hard to tell what does what. Like I've read 6 times what matchLabels does, and it's still not clear if that's supposed to mirror template.metadata.labels.app, or what happens if it doesn't. Sure I'd figure it out if I set up the real thing and messed with the config to see what it does, but it shouldn't take that. Of course it's easy to follow tutorials without fully understanding how stuff works, and I did. Slightly more advanced things were a lot of trial and error. At some point I lost interest because I didn't really need this.