4 ms·

I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable. I would even pay money fo

by cfors 5y ago

I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable. I would even pay money for an actual offering that worked.

lukeqsee 5y ago
Grafana Cloud is the best ROI money my startup spends every month.
- igetspam 5y ago
  What's your spend? We're way into five digits and we're not getting thar in ROI. We're heavy on metrics, which come at a huge cost.
gotjosh- 5y ago
Hey there! I work with alerting in general at Grafana - what are the pain points of dashboards and alerts as code you're currently experiencing? Would love to deliver / capitalise on the feedback.
- wernerb 5y ago
  Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
  - wernerb 5y ago
    I can't edit my above comment anymore but I see that at least alerting is now a separate system in grafana 8! Great, we will take a look again!
    - gotjosh- 5y ago
      Thanks for sharing! We built Grafana 8 alerting to address all of the problems you've mentioned. Appreciate the second look and please let me know if you have any additional feedback.
- cfors 5y ago
  For one, I'm not convinced that the Grafana 8 Alerting API Swagger docs are up to date or ready for the public [0]. I've literally copied an alert's json format, and then tried to post it back and never got it to work. Here's an example from my bash history: > curl -X POST -H "Authorization: Bearer $GRAFANA_API_KEY" -H "accept: application/json" -d @rule.json some_endpoint/api/ruler/grafana/api/v1/rules/test1 I spent a solid day trying to play around with this to get it to work. Because of this the alerts are impossible to code review or store in a git source. Which stinks because Grafana's datasource API's would be amazing to use for alerting. But they're either unusable because anybody can change them or the administrator could bork them at any given point (which has happened before), or just undocumented to the point where they are useless. That's not even to begin on dealing with the "big blob of json" problem [1] that was clearly important enough to be given an entire spot at GrafanaCon, but even Grafonnet is not supported with Grafana 8. There is apparently some CUE way of doing this, but I can't seem to find any official documentation on that. Anyways, I've moved back to alertmanager for the time being. edit: is all of grafana labs downvoting the GP? this is very honest and candid feedback here. [0]: https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json https://editor.swagger.io/?url=https://raw.githubusercontent... [1]: https://grafana.com/go/grafanaconline/2021/dashboards-as-code/ https://grafana.com/go/grafanaconline/2021/dashboards-as-cod...
- wtfishackernews 5y ago
  It's currently impossible to write alert rules for Prometheus vectors. https://github.com/grafana/grafana/issues/35663 https://github.com/grafana/grafana/issues/35663 Missing basic functionality like that is a dealbreaker.
  - KyleBrandt 5y ago
    Hoping to see cleaner ways to integrate across data sources, but developing that contract is going to take some time I think. In the meantime, should be able to get this supported with prometheus data source in a Grafana managed alert: https://github.com/grafana/grafana/pull/44865 https://github.com/grafana/grafana/pull/44865
wernerb 5y ago
Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
- mikewave 5y ago
  The new Grafana alerts do absolutely nothing to help with this. I'm at the point where I would pay 5 figures a year for something purely to do better alerting inside or alongside Grafana. Clicking alerts together is a nightmare when I have a ton of identical systems I need to configure. Same for dashboards - the limitations of the current mechanism are too severe. I'd build my own templating mechanism for it, but I still want the alerts visible in Grafana itself. Zabbix has the power to do all this but with a UX that is not ideal....
  - treffer 5y ago
    Well, been down the Prometheus/Alertmanager path and it is pretty workable there. You would get all the templating and grouping by labels. Dashboards can showed Prometheus alerts through annotations on the graphs. So you get a visual feedback what was broken when. Firing alerts are also a metric on Prometheus, so you can list those (or do other stuff with it). It's not a UI thing though. More the lower layers to get stuff for Grafana.
chrismarlow9 5y ago
Grafana would do well to look into the thinkorswim desktop platform and the ability to write code around metrics. They are entirely different use cases but I feel the desired goals are the same, which is making the most of an ocean of metrics. Financial world crushes at this for obvious reasons. Tech world? not so much.
- rixed 5y ago
  Why the desktop platform specifically, given they also appear to have a web version?

by cfors 5y ago

I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable. I would even pay money for an actual offering that worked.

lukeqsee 5y ago

Grafana Cloud is the best ROI money my startup spends every month.

igetspam 5y ago
What's your spend? We're way into five digits and we're not getting thar in ROI. We're heavy on metrics, which come at a huge cost.

gotjosh- 5y ago

Hey there! I work with alerting in general at Grafana - what are the pain points of dashboards and alerts as code you're currently experiencing? Would love to deliver / capitalise on the feedback.

wernerb 5y ago
Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
- wernerb 5y ago
  I can't edit my above comment anymore but I see that at least alerting is now a separate system in grafana 8! Great, we will take a look again!
  - gotjosh- 5y ago
    Thanks for sharing! We built Grafana 8 alerting to address all of the problems you've mentioned. Appreciate the second look and please let me know if you have any additional feedback.
cfors 5y ago
For one, I'm not convinced that the Grafana 8 Alerting API Swagger docs are up to date or ready for the public [0]. I've literally copied an alert's json format, and then tried to post it back and never got it to work. Here's an example from my bash history: > curl -X POST -H "Authorization: Bearer $GRAFANA_API_KEY" -H "accept: application/json" -d @rule.json some_endpoint/api/ruler/grafana/api/v1/rules/test1 I spent a solid day trying to play around with this to get it to work. Because of this the alerts are impossible to code review or store in a git source. Which stinks because Grafana's datasource API's would be amazing to use for alerting. But they're either unusable because anybody can change them or the administrator could bork them at any given point (which has happened before), or just undocumented to the point where they are useless. That's not even to begin on dealing with the "big blob of json" problem [1] that was clearly important enough to be given an entire spot at GrafanaCon, but even Grafonnet is not supported with Grafana 8. There is apparently some CUE way of doing this, but I can't seem to find any official documentation on that. Anyways, I've moved back to alertmanager for the time being. edit: is all of grafana labs downvoting the GP? this is very honest and candid feedback here. [0]: https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json https://editor.swagger.io/?url=https://raw.githubusercontent... [1]: https://grafana.com/go/grafanaconline/2021/dashboards-as-code/ https://grafana.com/go/grafanaconline/2021/dashboards-as-cod...
wtfishackernews 5y ago
It's currently impossible to write alert rules for Prometheus vectors. https://github.com/grafana/grafana/issues/35663 https://github.com/grafana/grafana/issues/35663 Missing basic functionality like that is a dealbreaker.
- KyleBrandt 5y ago
  Hoping to see cleaner ways to integrate across data sources, but developing that contract is going to take some time I think. In the meantime, should be able to get this supported with prometheus data source in a Grafana managed alert: https://github.com/grafana/grafana/pull/44865 https://github.com/grafana/grafana/pull/44865

wernerb 5y ago

Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.

mikewave 5y ago
The new Grafana alerts do absolutely nothing to help with this. I'm at the point where I would pay 5 figures a year for something purely to do better alerting inside or alongside Grafana. Clicking alerts together is a nightmare when I have a ton of identical systems I need to configure. Same for dashboards - the limitations of the current mechanism are too severe. I'd build my own templating mechanism for it, but I still want the alerts visible in Grafana itself. Zabbix has the power to do all this but with a UX that is not ideal....
- treffer 5y ago
  Well, been down the Prometheus/Alertmanager path and it is pretty workable there. You would get all the templating and grouping by labels. Dashboards can showed Prometheus alerts through annotations on the graphs. So you get a visual feedback what was broken when. Firing alerts are also a metric on Prometheus, so you can list those (or do other stuff with it). It's not a UI thing though. More the lower layers to get stuff for Grafana.

chrismarlow9 5y ago

Grafana would do well to look into the thinkorswim desktop platform and the ability to write code around metrics. They are entirely different use cases but I feel the desired goals are the same, which is making the most of an ocean of metrics. Financial world crushes at this for obvious reasons. Tech world? not so much.

rixed 5y ago
Why the desktop platform specifically, given they also appear to have a web version?