4 ms·

I am all for this trend. Microk8s and k3s are both a joy to develop with (though I have ran into a few bugs with k3s so tend to prefer Microk8s). How many folk

I am all for this trend. Microk8s and k3s are both a joy to develop with (though I have ran into a few bugs with k3s so tend to prefer Microk8s). How many folks are running self-managed k8s in production though, out of curiosity? It seems so economical to deploy k3s or use something like Rancher on dirt cheap VPS's from some place like Hetzner -- but what's the ops burden and failure risk like? Never tried it myself because it seems intimidating. Use managed k8s services.

tuananh 6y ago
> (though I have ran into a few bugs with k3s so tend to prefer Microk8s). i tried both in my homelab and this is my experience as well. microk8s seems to has less bug for me.
kavalg 6y ago
Maybe not the answer you are looking for, but for small and not that critical workloads, I've so far been happy with docker swarm workloads on cheap cloud/VPS.
- nurgasemetey 6y ago
  We run our platform in different cities and they are all on-prem(custom server or managed hypervisor). After hearing horror stories about kubernetes on prem, we are happy that we decided to go with docker swarm.
  - jakupovic 6y ago
    K8s is the new platform, you're just delaying the inevitable. Get with the program or suffer when your lack of knowledge makes you deficient in the platform uptake.
    - akvadrako 6y ago
      However, being an early adopter is not usually wise. Better to wait until lots of other people have found the bugs and improved the documentation.
dhagz 6y ago
My employer has over 2000 clusters running k3s (I'm on the team that manages them), it's been...okay. We're on an old version though, so a lot of the k3s-specific issues we're running into are mostly fixed in more recent releases. The issues we run into more often are network or power related than k3s itself. Sometimes with the OS image the devices are running.
- e12e 6y ago
  > My employer has over 2000 clusters running k3s Oh wow.. Why so many clusters, and what kind of resources per cluster? Are you running 100 000s of physical servers? Or are you just using k3s over three physical nodes as a way to achieve hardware redundancy and rolling hardware replacement?
  - dhagz 6y ago
    We're providing a "smart kitchen" solution in restaurants. Each restaurant has three Intel NUCs, so a little over 6K-7K devices. We're using k3s partly for the hardware redundancy, and partly for the ease of managing the services running at the edge - there's a local OAuth provider, MQTT server, along with some other applications that need to be up a majority of the time. There's a cloud component to all of the software running at the edge as well, and since that's run on k8s, we wanted something similar but more lightweight at the edge.
    - e12e 6y ago
      Interesting. Do you happen to have a blog post or something discussing how you arrived at that architecture? Do you frequently see a nuc dying, but the cluster staying up?
      - dhagz 6y ago
        Yeah, there are some posts out there. I'll have to find them and edit them in. Yes, in fact - we've seen clusters still running when two of the NUCs have disappeared from the cluster.

tuananh 6y ago

> (though I have ran into a few bugs with k3s so tend to prefer Microk8s). i tried both in my homelab and this is my experience as well. microk8s seems to has less bug for me.

kavalg 6y ago

Maybe not the answer you are looking for, but for small and not that critical workloads, I've so far been happy with docker swarm workloads on cheap cloud/VPS.

nurgasemetey 6y ago
We run our platform in different cities and they are all on-prem(custom server or managed hypervisor). After hearing horror stories about kubernetes on prem, we are happy that we decided to go with docker swarm.
- jakupovic 6y ago
  K8s is the new platform, you're just delaying the inevitable. Get with the program or suffer when your lack of knowledge makes you deficient in the platform uptake.
  - akvadrako 6y ago
    However, being an early adopter is not usually wise. Better to wait until lots of other people have found the bugs and improved the documentation.

dhagz 6y ago

My employer has over 2000 clusters running k3s (I'm on the team that manages them), it's been...okay. We're on an old version though, so a lot of the k3s-specific issues we're running into are mostly fixed in more recent releases. The issues we run into more often are network or power related than k3s itself. Sometimes with the OS image the devices are running.

e12e 6y ago
> My employer has over 2000 clusters running k3s Oh wow.. Why so many clusters, and what kind of resources per cluster? Are you running 100 000s of physical servers? Or are you just using k3s over three physical nodes as a way to achieve hardware redundancy and rolling hardware replacement?
- dhagz 6y ago
  We're providing a "smart kitchen" solution in restaurants. Each restaurant has three Intel NUCs, so a little over 6K-7K devices. We're using k3s partly for the hardware redundancy, and partly for the ease of managing the services running at the edge - there's a local OAuth provider, MQTT server, along with some other applications that need to be up a majority of the time. There's a cloud component to all of the software running at the edge as well, and since that's run on k8s, we wanted something similar but more lightweight at the edge.
  - e12e 6y ago
    Interesting. Do you happen to have a blog post or something discussing how you arrived at that architecture? Do you frequently see a nuc dying, but the cluster staying up?
    - dhagz 6y ago
      Yeah, there are some posts out there. I'll have to find them and edit them in. Yes, in fact - we've seen clusters still running when two of the NUCs have disappeared from the cluster.