13 ms·
Watch out for Grafana Cloud and that. All it takes is one dumbass to not understand cardinality of labels and histograms and you have problems. Also on a Kuber
by minkles 2y ago
Watch out for Grafana Cloud and that. All it takes is one dumbass to not understand cardinality of labels and histograms and you have problems.
Also on a Kubernetes cluster we hit 1.8 million series out of the box on prometheus so you can imagine how much much that would have cost. And we had one service double that due to some poor understanding by an engineer.
Anything that isn't constrained cost is a major financial risk.
- aliasxneo 2y agoYeah, their prepackaged Kubernetes exports seem a bit excessive to me. We stood up a brand new EKS cluster, and it came in at 50k metrics, which was odd considering that's the exact limit before you get charged for additional metrics. It felt intentional.