4 ms·
I've seen this called window alignment. You can either pick a default and stick to it everywhere, e.g. truncate to the nearest minute or Nth-minute, but this o
by benlivengood 6y ago
I've seen this called window alignment. You can either pick a default and stick to it everywhere, e.g. truncate to the nearest minute or Nth-minute, but this only works for divisors of an hour for example so that it's easy to align different sources at any starting point.
Resampling is pretty effective; linear interpolation is usually good enough because the data is already downsampled, presumably with an averaging or aggregating step that can be interpolated over. The mean-value theorem is your friend if you're looking at derivatives of time series data.
The type of metric also matters; gauges can be simply interpolated. Counters make more sense to re-bucket into the chosen sampling period and offset, and for your specific question what you can do is switch from downsampled data to "realtime" data exactly at the bucket boundary so that you avoid the half-full bucket problem.
A good way to treat counter timeseries is as bundles of updates (deltas) that arrive at a certain cadence. That makes the transition from different cadences smooth by resampling to avoid overlapping updates, e.g. if you have a downsampled bucket counting events from 1:00 to 1:10 then resample other timeseries so that buckets begin exactly at 1:10 or end exactly at 1:00 instead of trying to let them overlap.
- dabreegster 6y agoI haven't had nearly enough caffeine today to process this properly, but thank you -- it partly makes sense, and it's plenty to go off of later!
- benlivengood 6y agoAfter re-reading your questions 1 and 2 I think interpolation basically answers them both. You want smooth and stable values for your counters at any time t, and the way to get that is define an interpolation function from (timeseries, absolute time) to value, or recreate a sufficient range of higher resolution timeseries values that can be graphed directly. Linear interpolation will give you a line chart. Curve-fitting methods will smooth the charts out, but since you are working with counters you can put some constraints on the interpolation. Since each bucket represents a change over a period of time it represents an area under the curve for the same interval which should match any graphic representation. https://stackoverflow.com/questions/34983360/create-a-piecewise-smooth-function-which-preserves-local-integrals-from-data https://stackoverflow.com/questions/34983360/create-a-piecew... is basically the question of how to interpolate between data points while preserving that area, which might be overkill for your application but is possible.
- bigger_cheese 6y agoInterpolation is what I have seen used at industrial Plant I work at to trend high frequency data onto a HMI for example. Reinsch Spline is commonly used for this.