5 ms·
Anomaly Detection of Time Series Data Using Machine Learning and Deep Learning
- gargravarr 9y agoThe first few words of this title sound way too sci-fi. Very disappointed when I read the whole thing together. Edit: supposed to be a light-hearted observation, I haven't read the article!
- baldeagle 9y agoTLDR: This is a high level overview of all the terms in the topic (Anomaly, Time Series, Deep Learning). If you have a passing familiarity with the problem space, there is nothing new here. There is an ad for xenonstack services at the bottom.
- VHRanger 9y ago> This is because the oscillations are dependable upon the business cycle. No, no, no, no. The "business cycle" is a misnomer; it's not a cycle. It doesn't fit any descriptions of a "cycle" except for the fact that it sometimes goes up and sometimes goes down. Saying that it's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible
- nerdponx 9y agoDo they just mean mean "seasonal/cyclic variations in business"?
- VHRanger 9y agoYes, there are seasonal variations in business (eg. agriculture!) but those aren't the "business cycle" which refers to the boom/bust recurrence (like the 1990s boom and 2000 tech bubble pop, the 2000-2008 boom and financial crisis, etc.)
- kgwgk 9y agoHave you tried to look "cycle" up in a dictionary?
- VHRanger 9y agonoun: cycle; plural noun: cycles 1.a series of events that are regularly repeated in the same order. So no, the "business cycle" is not a cycle. In the sense that you won't successfully fit a sin curve through the GDP that predicts timing or magnitude of significant deviations from the log GDP time series trend.
- kgwgk 9y agoYou forgot to include the example: "the recurrent cycle of harvest failure, food shortages, and price increases" Can you predict the timing and magnitude of food shortages and prices?
- krona 9y agoSaying that it's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible Just because something is hard doesn't mean it's impossible; You can predict anything with a degree of certainty. Then there are things like uncertainty and radical (Knightian) uncertainty. Any chartered actuary would tell you the same if you could afford one, and they will know quite a bit about business cycles and how to model them, too.
- VHRanger 9y agoExcept you won't in macroeconomics. The simple problem in macroeconomics is lack of data -- history happened only once so you can't test counterfactuals. You can try to build a model, but the confidence intervals for the next big crash are going to be either useless (way too wide) or bullshit if what you're trying to predict is the next market crash. Building a classifier around it machine learning style will not fundamentally help, either. You can try it: here is an open source, serious, full scale macroeconomic model by the NY Fed [1]. Try to predict the next large crash with it and report back [1] https://github.com/FRBNY-DSGE/DSGE.jl https://github.com/FRBNY-DSGE/DSGE.jl
- neffy 9y agoThis would be the same macro-economic theory that even macro-economists will now grant is fundamentally flawed. The rest is left as an exercise to the reader.
- ares2012 9y agoBusiness cycles are often a component of what is a composite function of the time series. So the time series itself might not have perfect cycles but the business cycle is usually one of the principal components of the time series, which is what I think they were referring to. For example, the price of commodities does depend on the growing season (which is predictable) but other factors which are not cyclical so that the price itself is not easily predicted.
- pheldagryph 9y agoThe business cycle is perfectly apparent... in retrospect. If it was apparent in... forespect... we'd all be rich!
- adrianN 9y agoNo, information that is available to anyone has no value.
- Xcelerate 9y agoIf you're interested in more research level techniques, check out some of the work that's been done using minimum description length (MDL). It basically seeks to minimize the size of the model plus the size of the residual data (that the model doesn't predict) and uses Huffman compression as a proxy for the Kolmogorov complexity of a string: "Discovering the intrinsic cardinality and dimensionality of time series using MDL" https://pdfs.semanticscholar.org/2049/50b3cd9cf2eef52f957df626e01cad7adcef.pdf https://pdfs.semanticscholar.org/2049/50b3cd9cf2eef52f957df6... A few papers that build on that one show how to incorporate anomaly detection (in the sense that the anomaly is incompressible).
- boltzmannbrain 9y agoAnd more on research-level techniques: Ahmad et al. "Unsupervised real-time anomaly detection for streaming data": http://www.sciencedirect.com/science/article/pii/S0925231217309864 http://www.sciencedirect.com/science/article/pii/S0925231217... Anomaly detection benchmark (dataset and evaluation): https://github.com/numenta/NAB https://github.com/numenta/NAB
- Omnipresent 9y agoSince this article is very high level, are there other resources that show how to apply some (or all) of these topics in actual code or on a sample set of data?
- myth_drannon 9y agohttp://machinelearningmastery.com/blog/ http://machinelearningmastery.com/blog/ has some good tutorials on time series and ML in general