8 ms·

Grafana Incident: Smart incident management for your teams

250 points by matryer 5y ago

antod 5y ago
Will it always be a Grafana Cloud only offering?
- shamiln 5y ago
  Seems like the industry is headed in that direction.
  - deleted 5y ago
    [deleted]
- netingle 5y ago
  For now, yes. Long term we're trying to offer everything we do both on premise and in the cloud. It's a bit tricky, so we can't say when....
  - zbhoy 5y ago
    Have you heard of Replicated.com before? They might be able to get y'all to both on premise and in the cloud at the same time easier
  - chosenken 5y ago
    Would it be possible to have a split offering, with both on prem and cloud? In my mind I would prefer to have things like Prometheus, Logs, and Metrics stored on prem mainly due to the volume of logs and metrics we create. Then use Grafana cloud for Grafana Dashboards, Loki logs, and incident management that pull directly from my on prem data stores. I bring this up as it may be cost prohibitive for us to store our metrics in the cloud ( we make so many metrics and logs! ) but I would love to off load hosting the front end. Grafana cloud takes care of managing and maintaining Grafana Dashboard and backend database, Authentication, updates, ect. I'm fine hosting Prometheus and Loki locally, have been for a long time! I just get annoyed having to host Grafana and setting it up, the database up, configuring auth, etc.
    - bboreham 5y ago
      I’m pretty sure that is doable today: Hosted Grafana with data sources pointing at your on-prem Prometheus and Loki. https://grafana.com/docs/grafana-cloud/fundamentals/gs-visualize/ https://grafana.com/docs/grafana-cloud/fundamentals/gs-visua... (I work for Grafana Labs, but not on this part)
  - mikewave 5y ago
    Is there any hope of a Grafana Cloud data access proxy that runs on prem and enables us to give the Cloud access to databases we cannot expose?
    - netingle 5y ago
      Yes! It’s something we’ve be mulling for a while, and I was just talking to one of the PMs about it this morning. This year for sure I hope.
  - BeefWellington 5y ago
    > It's a bit tricky, so we can't say when.... I'm curious about this part, and I can absolutely understand if you don't want to answer but I do have the following question: Why is it tricky to ensure an application can run on a cloud deployed system or a local Kubernetes/Docker Swarm/newfangle containerization mechanism of choice/etc. system? Specifically I'm wondering what barriers you're running into that are pushing the focus to go cloud only.
- matryer 5y ago
  Yeah, building for Grafana Cloud has big dev benefits too. We can iterate quickly, run live experiments, and build a more complicated stack (e.g. for ML tasks). We're going to be integrating more and more with the rest of Grafana too. All of this is much easier to do in one place.
  - encryptluks2 5y ago
    It also has drawbacks like being locked into Saas products that you don't have a lot of insight to.
jtlisi 5y ago
This looks really sharp! Love the opinionated approach to how to handle incidents with assigned roles!
- amelius 5y ago
  It seems like this is a special case of project management software. If the existing products can't handle incidents then that software should be improved, not new software written. It's the best way to ensure that everybody on the team knows how to use the software when it's most urgently needed. E.g. would you change your favorite editor to a different one, in case of an incident? Probably not. So why change project management systems?
  - hughrr 5y ago
    Zero here!
    - matryer 5y ago
      You must have solid tech. :)
  - deleted 5y ago
    [deleted]
  - JshWright 5y ago
    While you certainly could cobble together incident response workflows in something like Jira, I think it makes more sense to extend the monitoring and paging tooling (in large part due to the reason you mention— familiarity with the tools that you're using as part of that response).
    - jrowley 5y ago
      Jira now has OpsGenie so you don’t have to cobble anything together, in theory.
  - bastardoperator 5y ago
JshWright 5y ago
This is timely... I just started building out an internal "chatops" solution that leans heavily on OnCall. Looks like I may be able to set that aside. If this is implemented as cleanly as OnCall, I have high hopes. It isn't without bugs, but it's already miles ahead of solutions like Pager Duty (in my opinion).
- bloodyplonker22 5y ago
  PagerDuty is a product that has not evolved much at all in the last 10 years, unfortunately.
- btables 5y ago
  I'd checkout FireHydrant, but I'm biased ;)
  - JshWright 5y ago
    Yeah, there are definitely already products in this space, but we're already invested in Grafana, so it makes sense to lean in that direction, even if it meant a little custom work on our end (though it looks like that may not be necessary now)
- motakuk 5y ago
  Please reach out to me, it would be awesome to learn your experience of using our API and make sure we're aware of all bugs you noticed (and fixing them!), matvey.kukuy @grafana.com
  - JshWright 5y ago
    I guess "bugs" is a strong word, more like "suboptimal UX", but I'll definitely reach out with details.
encryptluks2 5y ago
So does Grafana actually believe in open source or not?
sjwhitworth 5y ago
- capableweb 5y ago
  You'd do your job as a CEO better if you didn't spam competitors HN threads with your own product, unless you have something relevant to bring to the table. This comment just looks like a shameless plug because you're in the same sector. One way you could approach is to highlight what you think is good with Grafanas implementation, and what could be better, and then contrast that with your own offering, without sounding like a salesman.
  - capableweb 5y ago
    Seems post was deleted, not a great look for the CEO of incident.io
- burkaman 5y ago
  This is just incredibly rude. Please don't do it again.
cfors 5y ago
I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable. I would even pay money for an actual offering that worked.
- lukeqsee 5y ago
  Grafana Cloud is the best ROI money my startup spends every month.
  - igetspam 5y ago
    What's your spend? We're way into five digits and we're not getting thar in ROI. We're heavy on metrics, which come at a huge cost.
- gotjosh- 5y ago
  Hey there! I work with alerting in general at Grafana - what are the pain points of dashboards and alerts as code you're currently experiencing? Would love to deliver / capitalise on the feedback.
  - wernerb 5y ago
    Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
    - wernerb 5y ago
palijer 5y ago
>Automatically create the online meeting spaces for collaboration >Manage TODO items so nothing falls through the cracks I work in incident response, and I feel a huge misunderstanding of incident response products fail to understand that companies already have established tools for collaborations and meetings and for capturing planned work. I find adding these things is seen as nice and inclusive and it is easier to sell a product that does a lot, but it turns into complete bloat and makes adoption harder, and makes it harder to support a larger product.
- dharmab 5y ago
  I've used products in this space that would integrate with your existing video, chat and ticketing tools.
- buscoquadnary 5y ago
  I think the problem is trying to present an abstraction layer to management, because we have those same features of todo lists, and recording information, in Jira and ServiceNow and like a dozen other pieces that's purpose is to coordinate and track work, and often they are unpopular with developers because they end up trying to provide an abstraction layer to the Execs to replace their management by spreadsheets, but unfortunately as anyone who has worked in software for long enough can tell you, abstractions are leaky. Hence the dissatisfaction with a lot of these tools.
- aantix 5y ago
  Interesting take.. What do you think is the solution - when an enterprise already has Jira, Github and Confluence, how do you think a product like Grafana Incident should integrate with these somewhat overlapping products?
  - ethbr0 5y ago
    This feels like a central question of post-cloud / post-SaaS outsourcing. In the end, it boils down to two options: offer deep APIs into your product, or don't. IMHO, what needs to happen to support the former is for every SaaS purchase to include full technical due diligence on external integration capabilities. Integration needs to start being a headline feature in purchasing. And less an afterthought when a horrified engineer looks at some new enterprise product that's already being adopted.
dijit 5y ago
It’s funny what process can do. 13 years ago I was working on a SaaS eCommerce platform and it feels like this tool is a relatively minor improvement over what we had built on top of IRC. That said; it’s pretty cool and I’m definitely going to evaluate it: as our current PagerDuty integration is not nearly as clean as this.
deleted 5y ago
[deleted]
- deleted 5y ago
  [deleted]
firstSpeaker 5y ago
In most of places I have been involved with ServiceNow has been the core of incident management. From alerts playbook to follow up on systems/components uptime and daily/monthly/yearly SLA breaches. Any system that is offered for enterprises should somehow integrate into that solution. Generally speaking, I can say that ServiceNow is horrible to work with, use, manage; but it looks like it is the solution that is dominant in enterprises.

250 points by matryer 5y ago

antod 5y ago

Will it always be a Grafana Cloud only offering?

shamiln 5y ago
Seems like the industry is headed in that direction.
- deleted 5y ago
  [deleted]
netingle 5y ago
For now, yes. Long term we're trying to offer everything we do both on premise and in the cloud. It's a bit tricky, so we can't say when....
- zbhoy 5y ago
  Have you heard of Replicated.com before? They might be able to get y'all to both on premise and in the cloud at the same time easier
- chosenken 5y ago
  Would it be possible to have a split offering, with both on prem and cloud? In my mind I would prefer to have things like Prometheus, Logs, and Metrics stored on prem mainly due to the volume of logs and metrics we create. Then use Grafana cloud for Grafana Dashboards, Loki logs, and incident management that pull directly from my on prem data stores. I bring this up as it may be cost prohibitive for us to store our metrics in the cloud ( we make so many metrics and logs! ) but I would love to off load hosting the front end. Grafana cloud takes care of managing and maintaining Grafana Dashboard and backend database, Authentication, updates, ect. I'm fine hosting Prometheus and Loki locally, have been for a long time! I just get annoyed having to host Grafana and setting it up, the database up, configuring auth, etc.
  - bboreham 5y ago
    I’m pretty sure that is doable today: Hosted Grafana with data sources pointing at your on-prem Prometheus and Loki. https://grafana.com/docs/grafana-cloud/fundamentals/gs-visualize/ https://grafana.com/docs/grafana-cloud/fundamentals/gs-visua... (I work for Grafana Labs, but not on this part)
- mikewave 5y ago
  Is there any hope of a Grafana Cloud data access proxy that runs on prem and enables us to give the Cloud access to databases we cannot expose?
  - netingle 5y ago
    Yes! It’s something we’ve be mulling for a while, and I was just talking to one of the PMs about it this morning. This year for sure I hope.
- BeefWellington 5y ago
  > It's a bit tricky, so we can't say when.... I'm curious about this part, and I can absolutely understand if you don't want to answer but I do have the following question: Why is it tricky to ensure an application can run on a cloud deployed system or a local Kubernetes/Docker Swarm/newfangle containerization mechanism of choice/etc. system? Specifically I'm wondering what barriers you're running into that are pushing the focus to go cloud only.
matryer 5y ago
Yeah, building for Grafana Cloud has big dev benefits too. We can iterate quickly, run live experiments, and build a more complicated stack (e.g. for ML tasks). We're going to be integrating more and more with the rest of Grafana too. All of this is much easier to do in one place.
- encryptluks2 5y ago
  It also has drawbacks like being locked into Saas products that you don't have a lot of insight to.

jtlisi 5y ago

This looks really sharp! Love the opinionated approach to how to handle incidents with assigned roles!

amelius 5y ago
It seems like this is a special case of project management software. If the existing products can't handle incidents then that software should be improved, not new software written. It's the best way to ensure that everybody on the team knows how to use the software when it's most urgently needed. E.g. would you change your favorite editor to a different one, in case of an incident? Probably not. So why change project management systems?
- hughrr 5y ago
  Zero here!
  - matryer 5y ago
    You must have solid tech. :)
- deleted 5y ago
  [deleted]
- JshWright 5y ago
  While you certainly could cobble together incident response workflows in something like Jira, I think it makes more sense to extend the monitoring and paging tooling (in large part due to the reason you mention— familiarity with the tools that you're using as part of that response).
  - jrowley 5y ago
    Jira now has OpsGenie so you don’t have to cobble anything together, in theory.
- bastardoperator 5y ago

JshWright 5y ago

This is timely... I just started building out an internal "chatops" solution that leans heavily on OnCall. Looks like I may be able to set that aside. If this is implemented as cleanly as OnCall, I have high hopes. It isn't without bugs, but it's already miles ahead of solutions like Pager Duty (in my opinion).

bloodyplonker22 5y ago
PagerDuty is a product that has not evolved much at all in the last 10 years, unfortunately.
btables 5y ago
I'd checkout FireHydrant, but I'm biased ;)
- JshWright 5y ago
  Yeah, there are definitely already products in this space, but we're already invested in Grafana, so it makes sense to lean in that direction, even if it meant a little custom work on our end (though it looks like that may not be necessary now)
motakuk 5y ago
Please reach out to me, it would be awesome to learn your experience of using our API and make sure we're aware of all bugs you noticed (and fixing them!), matvey.kukuy @grafana.com
- JshWright 5y ago
  I guess "bugs" is a strong word, more like "suboptimal UX", but I'll definitely reach out with details.

encryptluks2 5y ago

So does Grafana actually believe in open source or not?

sjwhitworth 5y ago

capableweb 5y ago
You'd do your job as a CEO better if you didn't spam competitors HN threads with your own product, unless you have something relevant to bring to the table. This comment just looks like a shameless plug because you're in the same sector. One way you could approach is to highlight what you think is good with Grafanas implementation, and what could be better, and then contrast that with your own offering, without sounding like a salesman.
- capableweb 5y ago
  Seems post was deleted, not a great look for the CEO of incident.io
burkaman 5y ago
This is just incredibly rude. Please don't do it again.

cfors 5y ago

I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable. I would even pay money for an actual offering that worked.

lukeqsee 5y ago
Grafana Cloud is the best ROI money my startup spends every month.
- igetspam 5y ago
  What's your spend? We're way into five digits and we're not getting thar in ROI. We're heavy on metrics, which come at a huge cost.
gotjosh- 5y ago
Hey there! I work with alerting in general at Grafana - what are the pain points of dashboards and alerts as code you're currently experiencing? Would love to deliver / capitalise on the feedback.
- wernerb 5y ago
  Alert templating. Grafana is fussy about configuring alerts on dashboards that have variables. What this means is if you have 30 clusters and want to use a single dashboard with a drop-down variable seefting your cluster you cannot define alerts on it. It will refuse to do it. Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json! What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
  - wernerb 5y ago

palijer 5y ago

>Automatically create the online meeting spaces for collaboration >Manage TODO items so nothing falls through the cracks I work in incident response, and I feel a huge misunderstanding of incident response products fail to understand that companies already have established tools for collaborations and meetings and for capturing planned work. I find adding these things is seen as nice and inclusive and it is easier to sell a product that does a lot, but it turns into complete bloat and makes adoption harder, and makes it harder to support a larger product.

dharmab 5y ago
I've used products in this space that would integrate with your existing video, chat and ticketing tools.
buscoquadnary 5y ago
I think the problem is trying to present an abstraction layer to management, because we have those same features of todo lists, and recording information, in Jira and ServiceNow and like a dozen other pieces that's purpose is to coordinate and track work, and often they are unpopular with developers because they end up trying to provide an abstraction layer to the Execs to replace their management by spreadsheets, but unfortunately as anyone who has worked in software for long enough can tell you, abstractions are leaky. Hence the dissatisfaction with a lot of these tools.
aantix 5y ago
Interesting take.. What do you think is the solution - when an enterprise already has Jira, Github and Confluence, how do you think a product like Grafana Incident should integrate with these somewhat overlapping products?
- ethbr0 5y ago
  This feels like a central question of post-cloud / post-SaaS outsourcing. In the end, it boils down to two options: offer deep APIs into your product, or don't. IMHO, what needs to happen to support the former is for every SaaS purchase to include full technical due diligence on external integration capabilities. Integration needs to start being a headline feature in purchasing. And less an afterthought when a horrified engineer looks at some new enterprise product that's already being adopted.

dijit 5y ago

It’s funny what process can do. 13 years ago I was working on a SaaS eCommerce platform and it feels like this tool is a relatively minor improvement over what we had built on top of IRC. That said; it’s pretty cool and I’m definitely going to evaluate it: as our current PagerDuty integration is not nearly as clean as this.

deleted 5y ago

[deleted]

deleted 5y ago
[deleted]

firstSpeaker 5y ago

In most of places I have been involved with ServiceNow has been the core of incident management. From alerts playbook to follow up on systems/components uptime and daily/monthly/yearly SLA breaches. Any system that is offered for enterprises should somehow integrate into that solution. Generally speaking, I can say that ServiceNow is horrible to work with, use, manage; but it looks like it is the solution that is dominant in enterprises.