3 ms·

You don't even end up spending time on Kubernetes, because k8s is just part of the solution, a container scheduler. You have to bring logs, monitoring, a contai

You don't even end up spending time on Kubernetes, because k8s is just part of the solution, a container scheduler. You have to bring logs, monitoring, a container registry ,as well as a CI system with custom jobs and do integration of everything.

deathanatos 4y ago
This is true. But I had to bring all those things anyways, when I didn't run on k8s. I still needed some form of all of that. (Though "container registry" might be "package store", or something, depending on specifics of the implementation. Some form of artifact store.) And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.) ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
- dosethree 4y ago
  For sure, I had all this stuff before k8s as well

deathanatos 4y ago

This is true. But I had to bring all those things anyways, when I didn't run on k8s. I still needed some form of all of that. (Though "container registry" might be "package store", or something, depending on specifics of the implementation. Some form of artifact store.) And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.) ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.

dosethree 4y ago
For sure, I had all this stuff before k8s as well