This was a fascinating read. > I’ve had concerns about the sidecars in the past, however after this event I am convinced that having each workload ship with th

This was a fascinating read. > I’ve had concerns about the sidecars in the past, however after this event I am convinced that having each workload ship with their own logging and metric sidecars is the right thing. If these had been a shared resource to the cluster, the problem almost certainly would have been exacerbated and much harder to fix. While I'm all for the sidecars model, I'm not sure that I agree with this... If logging/metrics ingestion was a shared resource that applications were calling out to directly (basically a shim for kafka 99% of the time), it seems they could have built in a layer to dump writes to disk in the case kafka was unreachable. Even if they didn't and the shared service just started failing, that's a single (likely highly scaled) service failing (or best case 503ing and dropping logging+metrics where necessary) not service downtime/the cascading failures. Maybe I'm misunderstanding something about this quote. Maybe it's just me, but workloads going down due to a sidecar that handles logging/metrics doesn't sound like the right prioritization to me. Also, like other people have said, resource limits & using the kubernetes compute reservation systems[0] also seems like a key bit that was missing from this. [0]: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ https://kubernetes.io/docs/tasks/administer-cluster/reserve-...