6 ms·

Show HN: Using machine learning to recommend dashboards during incidents

52 points by shinryudbz 9y ago

shinryudbz 9y ago
Hi everyone, I'm one of the founders of Overseer. We built this tool because we noticed that engineers had to dig through many dashboards when diagnosing an incident. We felt this process could be streamlined through the use of machine learning. When we first started working on this project over a year ago, we weren't sure if the algorithms would work, or if our insights would be of value to anyone. We were also struggling to figure out how to make it easier for people to try the product without having to change their existing workflow. Since then, we've made huge improvements to the algorithms, deployed the tech for several large customers, and demonstrated value. Now I'd love to get a bit more feedback from you guys and see if we're going in the right direction! So here's how the tool works: 1 - We pull down your dashboards from your existing monitoring tool (e.g. Datadog/Wavefront/Librato) using your API key. 2 - We integrate with your PagerDuty account via a Webhook to notify us when an incident has triggered. 3 - When our Webhook is invoked, we will use machine learning to rank your dashboards, rank the metrics on those dashboards, and notify you via Slack/Email of the top dashboards/top metrics on those dashboards to look at. For this demo, we only expose the Wavefront plugin, and you'll be able to configure it on the initial page. To integrate with PagerDuty, you'll need a URL to our end point, and we'll need an email address where we can send the analysis. You can configure that by clicking on your user name (on the top right) and doing the following: 1. Clicking on "Generate API Key" and jotting down the generated Webhook URL. PagerDuty will need that. 2. Filling out the "Organization Email" text box. We will send your our analysis there! Given that we'll be dealing with potentially sensitive data, we reluctantly decided to add a layer of security and have folks register with us first - this allows us to protect your data better. My apologies for the inconvenience. I'd love to see what the HN community thinks and how we can make it better!
- weego 9y ago
  It could also be streamlined by just understanding your key metrics and visualising those over spunking out chart porn. Adding magic into the mix hoping to surface the right thing at the right time but not being entirely sure that would be the case seems like solving a problem of unknowns by hoping that adding new unknowns will cancel out the old ones.
  - josh_overseer 9y ago
    Solid point! I agree that it's super important to know your system well enough to know what your key metrics/kpis are without us. We are no substitute for that. One of the key places we try to fit in is after you may already know something is going on (ex, your kpis are being affected) and you want to drill down deeper to understand what's going on at a more detailed level. With a deep enough understanding of your system, you may have some ideas about where to look for more info, but there can sometimes be so many places to look that it's hard to know where to start. That's where we come in. One of the articles that really influences how I personally think about this stuff is here, it's a great read if you have the time: https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e https://medium.com/@copyconstruct/monitoring-in-the-time-of-... . In the words of the article, we fit in between the overview and debugging part of the monitoring pyramid.
  - rco8786 9y ago
    In a perfect world, sure everyone would know right where to look when triaging an incident. That’s a largely impossible task in real life, at least in my experience with larger teams/systems.
- rhizome 9y ago
  The thing that comes to mind is that maybe the incident should create a notification that also generates its own dashboard specific to the components that are (likely) involved. So rather than scanning or hunting for the right collection of indicators, they're all in the notification (or a URL within it). Note: I'm pretty sure this doesn't exist in a turnkey form.
  - deleted 9y ago
    [deleted]
  - josh_overseer 9y ago
    There are a couple things I think it's possible you're saying: (1) When an incident happens, you should get notified what components are likely involved. That is basically what we do if you hook us up to PagerDuty- when PagerDuty sends out an alert, we will also send out an email that lists what dashboards are likely relevant along with links to Overseer dashboards that show you which metrics are probably relevant. (2) By "the incident should create a notification" you might mean you'd like Overseer to generate an alert. While this would be possible to do, where Overseer really aims to shine is as a triaging tool rather than being the best alerting service.
capkutay 9y ago
Looks interesting...but I don't want to sign up with an email before seeing what it actually does.
- josh_overseer 9y ago
  Hi, I'm also from Overseer :-) . The site we link to has a summary, but here's a more in-depth explanation for you since you're interested. Basically we pull from wherever you keep your ops dashboards and metrics (right now we've only exposed Wavefront as a source), and then pull down some history for the data on those dashboards. We use machine learning to determine how the metrics on your dashboards normally behave, and then start monitoring those dashboards in real time. We use this info to generate "health scores" for your dashboards which you can view any time. Also, if you hook up PagerDuty, when a PagerDuty alert goes off, we can email you info about which dashboards are unhealthy and what metrics on those dashboards are contributing to that strange state.
  - deleted 9y ago
    [deleted]
- josh_overseer 9y ago
  If by "see" you mean screenshots, there are some on this article from when we opened up to CloudWatch users: https://engr.overseerlabs.io/free-public-beta-machine-learning-incident-triage-ops-8ba104ed5e56 https://engr.overseerlabs.io/free-public-beta-machine-learni...
stephengillie 9y ago
Dashboards are largely dangerous. They promote a "Star Trek Bridge" theater - a play where one actor looks at a screen, becomes concerned, and takes action. Often this action will include a report to a senior officer. Business leaders often enjoy seeing their engineers act out this scene. It's gratifying to see someone take expected action, and gratifying to simply look at your direct reports to see if they're working. But this theater play requires engineers to be idly starting at dashboards first, so they are in the right place to see an issue and take action. This leads to bored engineers, complacency, and delays issue resolution. It's also inefficient to pay people to be bored. Instead of having a dashboard display a subset of metrics, have alerting configured on these. Pager duty notifies the same if you're in another app, another castle, another room, or another state - you can pay people to do other things instead of staring at a dashboard all day. A big TV full of metrics is a prop that your actors and engineers will ignore.
- jstanley 9y ago
  You're right, it's inefficient to pay people to look at the screens. But when something's gone wrong and you want to know how to fix it, having your key indicators all in one place can be a big time saver.
  - r3bl 9y ago
    And let us not even mention that those key indicators could trigger all sorts of alarms (like email notifications and SMS notifications) when something goes wrong.
    - stephengillie 9y ago
      Put monitoring ON your key indicators, so your alerts tell you what's wrong, instead of asking you to figure it out. It can be intellectually gratifying to use brute-intelligence to "save the day" during an urgent incident, but those heroics are part of the "Star Trek Bridge" scene play, and wastes crucial time.
- jamesmishra 9y ago
bllguo 9y ago
Hmm.. I'm not convinced that this is a problem that actually needs solving. You admit to having these misgivings when starting out - could you describe in general terms what changed your mind?
- shinryudbz 9y ago
  Great question! Being an engineer myself, this was a personal pain point and I wanted to solve it, but the key question was whether or not machine learning would help. Thus, most of the time was spent deploying the tech with early adopters, refining the algos, and trying to better understand the value. What I learned was that our message resonated with some companies more than others. Working with those guys and getting some proof-points on the value is what kept us going!
singold 9y ago
I like the idea. It is something that we may buy at my job, but it being "cloud based" is a deal breaker for us, any plan for some self hosted version?
- shinryudbz 9y ago
  Absolutely! Currently all our large deployments are on-prem. Please reach out to me and we can discuss further: upal@overseerlabs.io.
rosstex 9y ago
I love the photos of equations. "We use enhanced Bayes' Rule and Joint Summation algorithms to serve you the most relevant and useful dashboards." Probably more truth to it than it seems :)
- shinryudbz 9y ago
  Glad you liked it :) It turns out that modeling operational metrics is a lot harder than I expected, so there was quite a bit of work we had to do to get the algos to work.
  - nerdponx 9y ago
    I was wondering about this. I assume you're treating this info as proprietary, but in case you aren't I'd love to hear more about how you actually implement this.
    - shinryudbz 9y ago
      Well, there's a lot of details and I don't think I can cover it all here, but if you're interested in the general framework that I used to approach this problem, take a look at this blog post I wrote last year: https://medium.com/@upal/how-to-use-machine-learning-to-debug-outages-10x-faster-e480b7e2a907 https://medium.com/@upal/how-to-use-machine-learning-to-debu.... Let me know if you have any feedback!
      - srean 9y ago
        I am a little surprised that you got some milage out of procedures and algorithms that rely on the Gaussian assumption (that includes k-means). From experience, the raw metrics are, shall we say, as violently non-Gaussian as it gets. OK OK you need not tell what you are doing to address that as I myself am being rather economical with the truth. But so glad, so glad that someone is using multivariate analysis, about time too. From someone who dabbles in a similar space I wish you well.
        9y ago
mendeza 9y ago
The fact you put Bayes rule on your page excites me. I am taking a graduate course in Bayesian machine learning, so Bayes is all around me lol. Any use of Gaussian processes or probabilistic graph models? Would love to hear also if you use Bayesian treatment for automatic model selection. It sounds ideal in lecture, but I am interested to see how it is applied in real world settings.
- shinryudbz 9y ago
  So it turns out that these metrics don't exactly follow a Gaussian distribution, so it's hard to get these algos to work right out the box. Additionally, speed was an important component for us (for training and evaluation), so we had to toss out a lot of the fancy, but slower, techniques.

52 points by shinryudbz 9y ago

Hi everyone, I'm one of the founders of Overseer. We built this tool because we noticed that engineers had to dig through many dashboards when diagnosing an incident. We felt this process could be streamlined through the use of machine learning. When we first started working on this project over a year ago, we weren't sure if the algorithms would work, or if our insights would be of value to anyone. We were also struggling to figure out how to make it easier for people to try the product without having to change their existing workflow. Since then, we've made huge improvements to the algorithms, deployed the tech for several large customers, and demonstrated value. Now I'd love to get a bit more feedback from you guys and see if we're going in the right direction! So here's how the tool works: 1 - We pull down your dashboards from your existing monitoring tool (e.g. Datadog/Wavefront/Librato) using your API key. 2 - We integrate with your PagerDuty account via a Webhook to notify us when an incident has triggered. 3 - When our Webhook is invoked, we will use machine learning to rank your dashboards, rank the metrics on those dashboards, and notify you via Slack/Email of the top dashboards/top metrics on those dashboards to look at. For this demo, we only expose the Wavefront plugin, and you'll be able to configure it on the initial page. To integrate with PagerDuty, you'll need a URL to our end point, and we'll need an email address where we can send the analysis. You can configure that by clicking on your user name (on the top right) and doing the following: 1. Clicking on "Generate API Key" and jotting down the generated Webhook URL. PagerDuty will need that. 2. Filling out the "Organization Email" text box. We will send your our analysis there! Given that we'll be dealing with potentially sensitive data, we reluctantly decided to add a layer of security and have folks register with us first - this allows us to protect your data better. My apologies for the inconvenience. I'd love to see what the HN community thinks and how we can make it better!

weego 9y ago
It could also be streamlined by just understanding your key metrics and visualising those over spunking out chart porn. Adding magic into the mix hoping to surface the right thing at the right time but not being entirely sure that would be the case seems like solving a problem of unknowns by hoping that adding new unknowns will cancel out the old ones.
- josh_overseer 9y ago
  Solid point! I agree that it's super important to know your system well enough to know what your key metrics/kpis are without us. We are no substitute for that. One of the key places we try to fit in is after you may already know something is going on (ex, your kpis are being affected) and you want to drill down deeper to understand what's going on at a more detailed level. With a deep enough understanding of your system, you may have some ideas about where to look for more info, but there can sometimes be so many places to look that it's hard to know where to start. That's where we come in. One of the articles that really influences how I personally think about this stuff is here, it's a great read if you have the time: https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e https://medium.com/@copyconstruct/monitoring-in-the-time-of-... . In the words of the article, we fit in between the overview and debugging part of the monitoring pyramid.
- rco8786 9y ago
  In a perfect world, sure everyone would know right where to look when triaging an incident. That’s a largely impossible task in real life, at least in my experience with larger teams/systems.
rhizome 9y ago
The thing that comes to mind is that maybe the incident should create a notification that also generates its own dashboard specific to the components that are (likely) involved. So rather than scanning or hunting for the right collection of indicators, they're all in the notification (or a URL within it). Note: I'm pretty sure this doesn't exist in a turnkey form.
- deleted 9y ago
  [deleted]
- josh_overseer 9y ago
  There are a couple things I think it's possible you're saying: (1) When an incident happens, you should get notified what components are likely involved. That is basically what we do if you hook us up to PagerDuty- when PagerDuty sends out an alert, we will also send out an email that lists what dashboards are likely relevant along with links to Overseer dashboards that show you which metrics are probably relevant. (2) By "the incident should create a notification" you might mean you'd like Overseer to generate an alert. While this would be possible to do, where Overseer really aims to shine is as a triaging tool rather than being the best alerting service.

capkutay 9y ago

Looks interesting...but I don't want to sign up with an email before seeing what it actually does.

josh_overseer 9y ago
Hi, I'm also from Overseer :-) . The site we link to has a summary, but here's a more in-depth explanation for you since you're interested. Basically we pull from wherever you keep your ops dashboards and metrics (right now we've only exposed Wavefront as a source), and then pull down some history for the data on those dashboards. We use machine learning to determine how the metrics on your dashboards normally behave, and then start monitoring those dashboards in real time. We use this info to generate "health scores" for your dashboards which you can view any time. Also, if you hook up PagerDuty, when a PagerDuty alert goes off, we can email you info about which dashboards are unhealthy and what metrics on those dashboards are contributing to that strange state.
- deleted 9y ago
  [deleted]
josh_overseer 9y ago
If by "see" you mean screenshots, there are some on this article from when we opened up to CloudWatch users: https://engr.overseerlabs.io/free-public-beta-machine-learning-incident-triage-ops-8ba104ed5e56 https://engr.overseerlabs.io/free-public-beta-machine-learni...

stephengillie 9y ago

Dashboards are largely dangerous. They promote a "Star Trek Bridge" theater - a play where one actor looks at a screen, becomes concerned, and takes action. Often this action will include a report to a senior officer. Business leaders often enjoy seeing their engineers act out this scene. It's gratifying to see someone take expected action, and gratifying to simply look at your direct reports to see if they're working. But this theater play requires engineers to be idly starting at dashboards first, so they are in the right place to see an issue and take action. This leads to bored engineers, complacency, and delays issue resolution. It's also inefficient to pay people to be bored. Instead of having a dashboard display a subset of metrics, have alerting configured on these. Pager duty notifies the same if you're in another app, another castle, another room, or another state - you can pay people to do other things instead of staring at a dashboard all day. A big TV full of metrics is a prop that your actors and engineers will ignore.

jstanley 9y ago
You're right, it's inefficient to pay people to look at the screens. But when something's gone wrong and you want to know how to fix it, having your key indicators all in one place can be a big time saver.
- r3bl 9y ago
  And let us not even mention that those key indicators could trigger all sorts of alarms (like email notifications and SMS notifications) when something goes wrong.
  - stephengillie 9y ago
    Put monitoring ON your key indicators, so your alerts tell you what's wrong, instead of asking you to figure it out. It can be intellectually gratifying to use brute-intelligence to "save the day" during an urgent incident, but those heroics are part of the "Star Trek Bridge" scene play, and wastes crucial time.
jamesmishra 9y ago

bllguo 9y ago

Hmm.. I'm not convinced that this is a problem that actually needs solving. You admit to having these misgivings when starting out - could you describe in general terms what changed your mind?

shinryudbz 9y ago
Great question! Being an engineer myself, this was a personal pain point and I wanted to solve it, but the key question was whether or not machine learning would help. Thus, most of the time was spent deploying the tech with early adopters, refining the algos, and trying to better understand the value. What I learned was that our message resonated with some companies more than others. Working with those guys and getting some proof-points on the value is what kept us going!

singold 9y ago

I like the idea. It is something that we may buy at my job, but it being "cloud based" is a deal breaker for us, any plan for some self hosted version?

shinryudbz 9y ago
Absolutely! Currently all our large deployments are on-prem. Please reach out to me and we can discuss further: upal@overseerlabs.io.

rosstex 9y ago

I love the photos of equations. "We use enhanced Bayes' Rule and Joint Summation algorithms to serve you the most relevant and useful dashboards." Probably more truth to it than it seems :)

shinryudbz 9y ago
Glad you liked it :) It turns out that modeling operational metrics is a lot harder than I expected, so there was quite a bit of work we had to do to get the algos to work.
- nerdponx 9y ago
  I was wondering about this. I assume you're treating this info as proprietary, but in case you aren't I'd love to hear more about how you actually implement this.
  - shinryudbz 9y ago
    Well, there's a lot of details and I don't think I can cover it all here, but if you're interested in the general framework that I used to approach this problem, take a look at this blog post I wrote last year: https://medium.com/@upal/how-to-use-machine-learning-to-debug-outages-10x-faster-e480b7e2a907 https://medium.com/@upal/how-to-use-machine-learning-to-debu.... Let me know if you have any feedback!
    - srean 9y ago
      I am a little surprised that you got some milage out of procedures and algorithms that rely on the Gaussian assumption (that includes k-means). From experience, the raw metrics are, shall we say, as violently non-Gaussian as it gets. OK OK you need not tell what you are doing to address that as I myself am being rather economical with the truth. But so glad, so glad that someone is using multivariate analysis, about time too. From someone who dabbles in a similar space I wish you well.
      - 9y ago

mendeza 9y ago

The fact you put Bayes rule on your page excites me. I am taking a graduate course in Bayesian machine learning, so Bayes is all around me lol. Any use of Gaussian processes or probabilistic graph models? Would love to hear also if you use Bayesian treatment for automatic model selection. It sounds ideal in lecture, but I am interested to see how it is applied in real world settings.

shinryudbz 9y ago
So it turns out that these metrics don't exactly follow a Gaussian distribution, so it's hard to get these algos to work right out the box. Additionally, speed was an important component for us (for training and evaluation), so we had to toss out a lot of the fancy, but slower, techniques.