4 ms·

Show HN: Grafana Live Streaming Datasource Implementation

120 points by seanlaff 8y ago

dkhenry 8y ago
I saw this presentation at grafanacon and it was awesome. I really hope the team takes this and turns it into a first class feature.
deleted 8y ago
[deleted]
kthejoker2 8y ago
Very cool, but real time visualization use cases are surprisingly hard to concoct. Almost any signal worth capturing visually at one second or sooner intervals would be better served through automation. For example, all SCADA systems already have command and control alerting for instant degradation response.
- kthejoker2 8y ago
  And not helped by placating BI vendors conflating "real time" (with a precise technical definition) with something akin to "zero friction" self service analytics platforms (you can start to answer a question as soon as you can conceive it.) Cf https://www.sisense.com/blog/dont-real-time-analytics-insights-already-late/ https://www.sisense.com/blog/dont-real-time-analytics-insigh...
- iluminae 8y ago
  Also think about having a dashboard refresh at 5s with 50k datapoints, literally transferring gigabytes of data to every browser looking at that dashboard. If you streamed the data (even at slow rates) it would be a massive traffic savings.
  - dkhenry 8y ago
    This was the thing I liked the best about it. Even if you wanted to not make it "real time" making it so I didn't have to transfer all my data every refresh would be benefit enough.
  - minimaxir 8y ago
    Recently I was looking into Plot.ly Dash (https://plot.ly/products/dash/ https://plot.ly/products/dash/) for a Python-based real-time updating dashboard. ...until I noticed that it transfers the entire chart and all the datapoints each update. (example: https://dash-gallery.plotly.host/dash-live-model-training/ https://dash-gallery.plotly.host/dash-live-model-training/)
    - doctorpangloss 8y ago
      It's funny, we already have a format for transferring graphical data in real time that the web has optimized relentlessly: video! You'd think someone would just run a daemon local to the data that live-streams a compressed video feed rendering the graph, instead of relentlessly and repeatedly reinventing some notion of lossy-but-useful realtime data transferring technology.
    - nicolaskruchten 8y ago
      I'll be honest, before I started working at Plotly I had the same reaction to Dash at first glance, but as I worked with it/on it, I realized that most of the time, this client-server chattiness is not much of a problem from a user perspective, and is a great developer-productivity tradeoff. More to your point, though, this is something we're trying to address in collaboration with the Dash community, in this PR for example if you want to follow along: https://github.com/plotly/dash-core-components/pull/461 https://github.com/plotly/dash-core-components/pull/461
- seanlaff 8y ago
  Agreed, watching dashboards is no substitute for alerting. However, one of the motivators for doing this is to reduce pressure on the metric backend. It's common to have Grafana reports refresh at 5s intervals, which if you have sub-second data, means that you request the same (potentially tens of thousands of) points over and over again. Some have worked around this by adding a caching layer (see https://github.com/Comcast/trickster https://github.com/Comcast/trickster), but streaming feels more elegant :)
  - rixed 8y ago
    Exactly this. Especially in an environment where you have tens or hundredths of people/screens with the dashboard opened. Non streaming dashboards are just stupidly hammering the backends (and the network transferring the same data over and over). So this has nothing to do with how cool it looks (also dashboard users rarely complain that they refresh or scroll too fast). Working on such a backend at the moment, I was missing this feature so much I would probably have started to work on this on my own at some point.
rixed 8y ago
After having looked at what this actually is, I'm a bit disappointed. I was hopping that they wound change the various panels data API so that they would request only the new time slices; ie. instead of querying the last hour of data every minute, query the last minute every minute and reuse the previous 49 data points. But that's not what this is. Instead, you need a specific panel and a specific data source for streaming to work. :-(
- seanlaff 8y ago
  This datasource doesn't require any custom panels to function. To your point though, it would be nice if grafana supported incremental time range querying out of the box, since many backends don't support streaming data.
  - nopzor 8y ago
    Something we are looking at. Problem is complicated when the query applies time series transformations such as moving averages, where requesting only incremental data can be problematic.

120 points by seanlaff 8y ago

I saw this presentation at grafanacon and it was awesome. I really hope the team takes this and turns it into a first class feature.

deleted 8y ago

[deleted]

kthejoker2 8y ago

Very cool, but real time visualization use cases are surprisingly hard to concoct. Almost any signal worth capturing visually at one second or sooner intervals would be better served through automation. For example, all SCADA systems already have command and control alerting for instant degradation response.

kthejoker2 8y ago
And not helped by placating BI vendors conflating "real time" (with a precise technical definition) with something akin to "zero friction" self service analytics platforms (you can start to answer a question as soon as you can conceive it.) Cf https://www.sisense.com/blog/dont-real-time-analytics-insights-already-late/ https://www.sisense.com/blog/dont-real-time-analytics-insigh...
iluminae 8y ago
Also think about having a dashboard refresh at 5s with 50k datapoints, literally transferring gigabytes of data to every browser looking at that dashboard. If you streamed the data (even at slow rates) it would be a massive traffic savings.
- dkhenry 8y ago
  This was the thing I liked the best about it. Even if you wanted to not make it "real time" making it so I didn't have to transfer all my data every refresh would be benefit enough.
- minimaxir 8y ago
  Recently I was looking into Plot.ly Dash (https://plot.ly/products/dash/ https://plot.ly/products/dash/) for a Python-based real-time updating dashboard. ...until I noticed that it transfers the entire chart and all the datapoints each update. (example: https://dash-gallery.plotly.host/dash-live-model-training/ https://dash-gallery.plotly.host/dash-live-model-training/)
  - doctorpangloss 8y ago
    It's funny, we already have a format for transferring graphical data in real time that the web has optimized relentlessly: video! You'd think someone would just run a daemon local to the data that live-streams a compressed video feed rendering the graph, instead of relentlessly and repeatedly reinventing some notion of lossy-but-useful realtime data transferring technology.
  - nicolaskruchten 8y ago
    I'll be honest, before I started working at Plotly I had the same reaction to Dash at first glance, but as I worked with it/on it, I realized that most of the time, this client-server chattiness is not much of a problem from a user perspective, and is a great developer-productivity tradeoff. More to your point, though, this is something we're trying to address in collaboration with the Dash community, in this PR for example if you want to follow along: https://github.com/plotly/dash-core-components/pull/461 https://github.com/plotly/dash-core-components/pull/461
seanlaff 8y ago
Agreed, watching dashboards is no substitute for alerting. However, one of the motivators for doing this is to reduce pressure on the metric backend. It's common to have Grafana reports refresh at 5s intervals, which if you have sub-second data, means that you request the same (potentially tens of thousands of) points over and over again. Some have worked around this by adding a caching layer (see https://github.com/Comcast/trickster https://github.com/Comcast/trickster), but streaming feels more elegant :)
- rixed 8y ago
  Exactly this. Especially in an environment where you have tens or hundredths of people/screens with the dashboard opened. Non streaming dashboards are just stupidly hammering the backends (and the network transferring the same data over and over). So this has nothing to do with how cool it looks (also dashboard users rarely complain that they refresh or scroll too fast). Working on such a backend at the moment, I was missing this feature so much I would probably have started to work on this on my own at some point.

rixed 8y ago

After having looked at what this actually is, I'm a bit disappointed. I was hopping that they wound change the various panels data API so that they would request only the new time slices; ie. instead of querying the last hour of data every minute, query the last minute every minute and reuse the previous 49 data points. But that's not what this is. Instead, you need a specific panel and a specific data source for streaming to work. :-(

seanlaff 8y ago
This datasource doesn't require any custom panels to function. To your point though, it would be nice if grafana supported incremental time range querying out of the box, since many backends don't support streaming data.
- nopzor 8y ago
  Something we are looking at. Problem is complicated when the query applies time series transformations such as moving averages, where requesting only incremental data can be problematic.