4 ms·

Grafana is really nice when you can go 100% all in on it. One thing I have found is that certain parts of grafana aren’t really designed for very large k8s clu

Grafana is really nice when you can go 100% all in on it. One thing I have found is that certain parts of grafana aren’t really designed for very large k8s clusters. For example if I want to utilize the k8s_sd_config for scraping metrics from pods the agent needs to watch ALL pods in the cluster which for clusters with hundreds of thousands of pods is a no go. I ended up recently writing my own daemonset agent which queries the kubelet API to get pods on the local node and uses that information to set up the scrape routines. There are other solutions such as scrape managers which do the watching and then tell the agents what endpoints to scrape, but that seems overly complex, and those features are in beta and not in the happy path. Another thing I would like to do more easily is have different relabel configs for different pods/targets. If you are running a multitennant k8s cluster it is useful to allow tenants to define their own scrape configs because they may want to relabel/rename/drop their metrics. Again, I found the grafana solutions either too complex or they don’t solve my problem so I wrote into my own solution where tenants define their scrape configs in k8s config maps and then the agent consumes those configmaps to set up scraping.

nijave 3y ago
Iirc this is a limitation with Prometheus, too, but they give you a switch to change the behavior It's possible Grafana Agent inherited (potentially without exposing the corresponding configuration switch to change it) Edit, it might also be a limitation with k8s, too. I think watches don't support annotations > Again, I found the grafana solutions either too complex or they don’t solve my problem so I wrote into my own solution where tenants define their scrape configs in k8s config maps and then the agent consumes those configmaps to set up scraping. I think the Prometheus Operator supports this. I also setup something similar with weaveworks tf-controller (or whatever they renamed it to). It can data source configmaps and shove that into other places

nijave 3y ago

Iirc this is a limitation with Prometheus, too, but they give you a switch to change the behavior It's possible Grafana Agent inherited (potentially without exposing the corresponding configuration switch to change it) Edit, it might also be a limitation with k8s, too. I think watches don't support annotations > Again, I found the grafana solutions either too complex or they don’t solve my problem so I wrote into my own solution where tenants define their scrape configs in k8s config maps and then the agent consumes those configmaps to set up scraping. I think the Prometheus Operator supports this. I also setup something similar with weaveworks tf-controller (or whatever they renamed it to). It can data source configmaps and shove that into other places