3 ms·
My employer has over 2000 clusters running k3s (I'm on the team that manages them), it's been...okay. We're on an old version though, so a lot of the k3s-specif
by dhagz 6y ago
My employer has over 2000 clusters running k3s (I'm on the team that manages them), it's been...okay. We're on an old version though, so a lot of the k3s-specific issues we're running into are mostly fixed in more recent releases. The issues we run into more often are network or power related than k3s itself. Sometimes with the OS image the devices are running.
- e12e 6y ago> My employer has over 2000 clusters running k3s Oh wow.. Why so many clusters, and what kind of resources per cluster? Are you running 100 000s of physical servers? Or are you just using k3s over three physical nodes as a way to achieve hardware redundancy and rolling hardware replacement?
- dhagz 6y agoWe're providing a "smart kitchen" solution in restaurants. Each restaurant has three Intel NUCs, so a little over 6K-7K devices. We're using k3s partly for the hardware redundancy, and partly for the ease of managing the services running at the edge - there's a local OAuth provider, MQTT server, along with some other applications that need to be up a majority of the time. There's a cloud component to all of the software running at the edge as well, and since that's run on k8s, we wanted something similar but more lightweight at the edge.
- e12e 6y agoInteresting. Do you happen to have a blog post or something discussing how you arrived at that architecture? Do you frequently see a nuc dying, but the cluster staying up?
- dhagz 6y agoYeah, there are some posts out there. I'll have to find them and edit them in. Yes, in fact - we've seen clusters still running when two of the NUCs have disappeared from the cluster.