Interesting to learn about last gasp packets, seems like a useful mechanism for any system. I generally find that observability into kubernetes is not that grea

Interesting to learn about last gasp packets, seems like a useful mechanism for any system. I generally find that observability into kubernetes is not that great especially for analytical queries. Answering the question of “where where my pods running for a replicaset at time X” is surprisingly not easy to answer with just events. Also, events are designed to have a short ttl so you need to extract them out of etcd and into some sort of long term storage and indexed in a way which you can run useful queries. At larger scales kube-state-metrics will blow up about any prometheus server outside of a full blown billion metric stream scale mimir deployment.