4 ms·
ASAP: Automatic Smoothing for Attention Prioritization in Time Series
- SEJeff 9y agoBlog post with additional info on why this is useful: http://dawn.cs.stanford.edu/2017/08/07/asap/ http://dawn.cs.stanford.edu/2017/08/07/asap/
- aashu_dwivedi 9y agoIs there a python implementation of it as well?
- SEJeff 9y agohttps://github.com/stanford-futuredata/ASAP/blob/master/ASAP.py https://github.com/stanford-futuredata/ASAP/blob/master/ASAP...
- aashu_dwivedi 9y agoJust tried it. I was using the seasonal_decompose from statsmodel in python for a project. I have to say ASAP looks great!
- SEJeff 9y agoIt really is a gem for analysis of time series data. I was lucky enough to watch Kexin's talk live @ Monitorama on ASAP. Really great stuff.
- jnordwick 9y agoDoes he explain how this is better than any other methods?
- SEJeff 9y agoI don't believe she did. She investigated the specific problem of helping users to identify deviations in time series by applying smoothing. Here is a link to her presentation: https://vimeo.com/221062931 https://vimeo.com/221062931 It is pretty fascinating, I do recommend giving it a watch.
- krong 9y agoThanks!
- srean 9y agoI upvoted the authors comment linking the Arxiv paper, that said, I think the presentation oversold the claim that smoothing is always a good thing to do. In particular, showing that excel, grafana, graphite etc, do not smooth by default, (apparently because it is a hard) looked like a low point of the talk, sorry. Smoothing has its uses but its hardly the panacea. What is an appropriate smoothing is very context sensitive and I would like the consumer of the plots be in charge of that. Sometimes smoothing away the high frequencies or transients are exactly the wrong thing to do (for example, when you care about transient spikes) I have seen more than enough damage done by plotting tools sneaking in a smoothing without informing the consumer. Especially so in the monitoring space. When the user expects raw plots it better be raw, and raw is a good default. Smoothening should be invoked explicitly.
- krong 9y agoThanks for the comment. I strongly agree that smoothing is not always beneficial and that the users should be informed and have control of the amount of the smoothing that is being applied to their plots. We developed ASAP with the intend to complement, instead of replacing the raw plots. We view ASAP more as a "smart default" - if you were to expose a slide bar of different moving average window sizes that users could apply to their visualizations, where should a reasonable default be on that slide bar? In addition, ASAP does not always choose to smooth the visualizations. In some cases (e.g. Figure C.1 in the arXiv paper) where the raw plots already have a few well-defined outlying regions and smoothing would only decrease the extent of such deviations, ASAP leaves the visualization unsmoothed.
- jnordwick 9y agoThe examples they give are periodic so removing the periodicity is the big win, not removing the noise. How did ASAP perform better than ARIMA and other similar algos should be the main question, not how does ASAP compare to nothing.
- aashu_dwivedi 9y agoIsn't Arima used for forecasting? Pardon me I'm not very informed about it but how is Arima used for smoothing?
- jnordwick 9y agoVarious ARIMA methods are used to deseason data for forecasting, but yes a better more directly comparable example might have been methods like STAMP and STL.
- polskibus 9y agoIn short - is this a moving average with window range automatically tuned for optimum detail to smoothness ratio?
- deleted 9y ago[deleted]
- aashu_dwivedi 9y agoYes.
- hcrisp 9y agoA moving window average is a poor digital filter in the frequency domain [0]. You can get better results with more control on which frequencies are removed by implementing a FIR or IIR (especially Butterworth) digital filter. [0] https://ptolemy.eecs.berkeley.edu/eecs20/week12/freqResponseRA.html https://ptolemy.eecs.berkeley.edu/eecs20/week12/freqResponse...
- alexbeloi 9y agoYes, optimizes window size for moving average Optimal detail is measure by: * minimizing variance of 1-step difference (point-to-point fluctuation) * maintaining kurtosis (outliers not lost) +optimizations to reduce search space
- hammock 9y agoComments are saying yes- but is it once you find the optimal window size, you fix it? Or does the window size vary over the data? Because if the latte wouldn't that be disingenuous to outliers and such?