4 ms·

Time Series Density Plot

40 points by luu 6y ago

loehnsberg 6y ago
Aren’t these spaghetti just plots with transparency. Or am I missing something? Another nice way to illustrate density are fan charts. While spaghetti plots show individual time series, fan charts just show quantiles. Spaghetti plots therefore also reveal information of the conditional distribution of a time series which fan charts do not. The latter are just easier to read.
- yurivish 6y ago
  Hi, author here. You can think of density plots as a way to visualize the data traditionally visualized with line charts or spaghetti plots, but in a way that scales more effectively to large datasets. Traditional graphics pipelines aren't built to handle more than 256 incremental levels of opacity since it's an 8-bit channel, and when you have a thousand time series you can no longer distinguish between levels of density. Conditional information (following a single line) becomes difficult as well since there are so many lines. Density plots solve the overdraw problem by accumulating density into an offscreen buffer before rendering, which allows the color scale to be tuned to the amount of overlap in the specific dataset. Interactive selection/hover techniques can be used to recover conditional information. This example is part of a density plot library published earlier this week (https://observablehq.com/@twitter/density-plot-introduction?collection=@twitter/density https://observablehq.com/@twitter/density-plot-introduction?...) and one of the references there is to an opacity-based approach I've seen used for 2D point clouds, though I haven't seen an analogous demo for curves: https://observablehq.com/@rreusser/selecting-the-right-opacity-for-2d-point-clouds https://observablehq.com/@rreusser/selecting-the-right-opaci...
cozzyd 6y ago
so... a finely-binned colormap? I'm not sure it's not quite right either, because the weight of each series in a bin should depend on the derivative but it doesn't look like it does. This sort of plot always has a conflict between showing the distribution of each time domain sample and showing the time behavior of each curve. For the former, showing quantile contours may be clearer (though not as pretty!). For the latter different colors with transparency might be better, though it becomes unintelligible at some point.
- yurivish 6y ago
  The weight does depend on the derivative: https://arxiv.org/abs/1808.06019 https://arxiv.org/abs/1808.06019 (This is mentioned in the intro post for this library: https://observablehq.com/@twitter/density-plot-introduction?collection=@twitter/density https://observablehq.com/@twitter/density-plot-introduction?...)
  - cozzyd 6y ago
    hmm, it's not obvious (to me!) in the single-line case, but that might be because of the high-frequency noise that makes the derivative sort of random.
    - yurivish 6y ago
      Here's a quick fork with one line, no noise, and an increasing-frequency sine wave: https://observablehq.com/d/ed17fc04bb7eb887 https://observablehq.com/d/ed17fc04bb7eb887
      - cozzyd 6y ago
        Ah cool! I suppose one can incorporate uncertainty on f(t) by having g(y,t) = gausy(0,\sigma(t)) * \delta(y-f(t)) (which would also effectively antialias in the few-lines case). Also I guess you have to be careful not to plot a Weierstrass Function!
  - floatrock 6y ago
    Wasn't immediately clear to me why the derivative comes into play, but let me see if I understand the gist of it. I think this is something interesting I haven't thought about in dataviz before... So in a traditional 2D heatmap, we discretize the space into squares or bins, then color the bin "proportional" to the number of points that fall inside the bin. If we're making a 2D heatmap of a timeseries, though, we don't really want to count up the discrete points that make up the timeseries because our binning may not line up with our timeseries sampling frequency. Especially not if we want a very high-resolution heatmap, where each bin is sized to be 1 pixel. So the solution isn't to count the number of points (from the timeseries data series) in each bin, but rather to count the number of lines that get projected/drawn into each bin (pixel). But we need to go a step further... if we think of each bin as a hypothetical square, the shortest line through a square is a vertical or horizontal line. And the longest line through a square is a diagonal line. So if we want to represent "how much" of the line goes through a square, we need to measure its slope. Hence, derivative. So by 2D heatmapping lines instead of points, we're not just counting how many lines fall into each bin, but we also need to weigh each bin-line occurrence by it's arc length, which we use the derivative as a proxy for. Neat! (I think...)
saltypal 6y ago
We use a homegrown gnuplot version of this at $work, and I can speak from experience that you can immediately spot problems (timeouts, as a set of ghostly horizontal lines, or outlier requests) that would be absolutely hidden in a traditional p50/p95/p99 chart. I wish more (hosted) monitoring systems would shift to this type of visualization, since computers are fast now (I'm told!)
tantalor 6y ago
"Here's how to..." call a function in a library that does all the work for you. Expected a bit more.
- yurivish 6y ago
  This page was specifically created as a library example. Here's the full collection: https://observablehq.com/collection/@twitter/density https://observablehq.com/collection/@twitter/density If you're interested in the technique behind the plot you can read more about how that works in this paper: https://arxiv.org/abs/1808.06019 https://arxiv.org/abs/1808.06019
atemerev 6y ago
One of my favorite plot types. Very simple and immediately conveying the necessary information. My colleagues in the lab dislike it, though, they prefer something more traditional with error bars, as they are more quantitatively inclined.
mighdoll 6y ago
Nice work! Density plots aren't just pretty. I think they show a more accurate story for any time series plot with more than a few thousand points. The reduceOp caught my eye in your API. What uses have you imagined/tried for changing the density accumulation function from x + y?
- yurivish 6y ago
  Thanks! I've used it when the data is itself binned frequency data, which is one way to reduce data volume (e.g. binning 2 weeks of minutely time series data into (hour, percentile) buckets and plotting them as weighted points). When the number of the bins in the data is not an exact integer multiple of the number of bins in the histogram, adjacent data bins can get mapped to the same histogram bin, resulting in e.g. 2x the data volume in some rows/columns of the histogram. When a bit of loss of fidelity is acceptable the two solutions I've used are to render the histogram at an exact factor of the number of data bins then set the canvas dimensions to the desired size (relying on the browser to downsample the resulting image), or to use `max` as the reduceOp and render the histogram directly at the intended size.

40 points by luu 6y ago

Aren’t these spaghetti just plots with transparency. Or am I missing something? Another nice way to illustrate density are fan charts. While spaghetti plots show individual time series, fan charts just show quantiles. Spaghetti plots therefore also reveal information of the conditional distribution of a time series which fan charts do not. The latter are just easier to read.

yurivish 6y ago
Hi, author here. You can think of density plots as a way to visualize the data traditionally visualized with line charts or spaghetti plots, but in a way that scales more effectively to large datasets. Traditional graphics pipelines aren't built to handle more than 256 incremental levels of opacity since it's an 8-bit channel, and when you have a thousand time series you can no longer distinguish between levels of density. Conditional information (following a single line) becomes difficult as well since there are so many lines. Density plots solve the overdraw problem by accumulating density into an offscreen buffer before rendering, which allows the color scale to be tuned to the amount of overlap in the specific dataset. Interactive selection/hover techniques can be used to recover conditional information. This example is part of a density plot library published earlier this week (https://observablehq.com/@twitter/density-plot-introduction?collection=@twitter/density https://observablehq.com/@twitter/density-plot-introduction?...) and one of the references there is to an opacity-based approach I've seen used for 2D point clouds, though I haven't seen an analogous demo for curves: https://observablehq.com/@rreusser/selecting-the-right-opacity-for-2d-point-clouds https://observablehq.com/@rreusser/selecting-the-right-opaci...

cozzyd 6y ago

so... a finely-binned colormap? I'm not sure it's not quite right either, because the weight of each series in a bin should depend on the derivative but it doesn't look like it does. This sort of plot always has a conflict between showing the distribution of each time domain sample and showing the time behavior of each curve. For the former, showing quantile contours may be clearer (though not as pretty!). For the latter different colors with transparency might be better, though it becomes unintelligible at some point.

yurivish 6y ago
The weight does depend on the derivative: https://arxiv.org/abs/1808.06019 https://arxiv.org/abs/1808.06019 (This is mentioned in the intro post for this library: https://observablehq.com/@twitter/density-plot-introduction?collection=@twitter/density https://observablehq.com/@twitter/density-plot-introduction?...)
- cozzyd 6y ago
  hmm, it's not obvious (to me!) in the single-line case, but that might be because of the high-frequency noise that makes the derivative sort of random.
  - yurivish 6y ago
    Here's a quick fork with one line, no noise, and an increasing-frequency sine wave: https://observablehq.com/d/ed17fc04bb7eb887 https://observablehq.com/d/ed17fc04bb7eb887
    - cozzyd 6y ago
      Ah cool! I suppose one can incorporate uncertainty on f(t) by having g(y,t) = gausy(0,\sigma(t)) * \delta(y-f(t)) (which would also effectively antialias in the few-lines case). Also I guess you have to be careful not to plot a Weierstrass Function!
- floatrock 6y ago
  Wasn't immediately clear to me why the derivative comes into play, but let me see if I understand the gist of it. I think this is something interesting I haven't thought about in dataviz before... So in a traditional 2D heatmap, we discretize the space into squares or bins, then color the bin "proportional" to the number of points that fall inside the bin. If we're making a 2D heatmap of a timeseries, though, we don't really want to count up the discrete points that make up the timeseries because our binning may not line up with our timeseries sampling frequency. Especially not if we want a very high-resolution heatmap, where each bin is sized to be 1 pixel. So the solution isn't to count the number of points (from the timeseries data series) in each bin, but rather to count the number of lines that get projected/drawn into each bin (pixel). But we need to go a step further... if we think of each bin as a hypothetical square, the shortest line through a square is a vertical or horizontal line. And the longest line through a square is a diagonal line. So if we want to represent "how much" of the line goes through a square, we need to measure its slope. Hence, derivative. So by 2D heatmapping lines instead of points, we're not just counting how many lines fall into each bin, but we also need to weigh each bin-line occurrence by it's arc length, which we use the derivative as a proxy for. Neat! (I think...)

saltypal 6y ago

We use a homegrown gnuplot version of this at $work, and I can speak from experience that you can immediately spot problems (timeouts, as a set of ghostly horizontal lines, or outlier requests) that would be absolutely hidden in a traditional p50/p95/p99 chart. I wish more (hosted) monitoring systems would shift to this type of visualization, since computers are fast now (I'm told!)

tantalor 6y ago

"Here's how to..." call a function in a library that does all the work for you. Expected a bit more.

yurivish 6y ago
This page was specifically created as a library example. Here's the full collection: https://observablehq.com/collection/@twitter/density https://observablehq.com/collection/@twitter/density If you're interested in the technique behind the plot you can read more about how that works in this paper: https://arxiv.org/abs/1808.06019 https://arxiv.org/abs/1808.06019

atemerev 6y ago

One of my favorite plot types. Very simple and immediately conveying the necessary information. My colleagues in the lab dislike it, though, they prefer something more traditional with error bars, as they are more quantitatively inclined.

mighdoll 6y ago

Nice work! Density plots aren't just pretty. I think they show a more accurate story for any time series plot with more than a few thousand points. The reduceOp caught my eye in your API. What uses have you imagined/tried for changing the density accumulation function from x + y?

yurivish 6y ago
Thanks! I've used it when the data is itself binned frequency data, which is one way to reduce data volume (e.g. binning 2 weeks of minutely time series data into (hour, percentile) buckets and plotting them as weighted points). When the number of the bins in the data is not an exact integer multiple of the number of bins in the histogram, adjacent data bins can get mapped to the same histogram bin, resulting in e.g. 2x the data volume in some rows/columns of the histogram. When a bit of loss of fidelity is acceptable the two solutions I've used are to render the histogram at an exact factor of the number of data bins then set the canvas dimensions to the desired size (relying on the browser to downsample the resulting image), or to use `max` as the reduceOp and render the histogram directly at the intended size.