6 ms·

I find it interesting that Computer Scientists are basically rediscovering statistics. Now when predicting time series, an issue is that most model (like ARIMA

by zwaps 8y ago

I find it interesting that Computer Scientists are basically rediscovering statistics. Now when predicting time series, an issue is that most model (like ARIMA, GARCH etc.) are short-memory processes. When you look at the full-series prediction of LSTMs, you observe the same thing. So in terms of Time Series, Machine Learning is currently in the mid to late 80's compared to Financial Econometrics. So if you are a CS, you should now probably take a look at fractional GARCH models and incorporate this into the LSTM logic. If the statistic issues are the same, then this may give you that hot new paper.

jkabrg 8y ago
Nassim Taleb had some negative things to say about GARCH. "GARCH does not work out of sample. It is a good story, but I was unable to use it in predicting squared deviations or mean deviations" I haven't found it in Rob J Hyndman's forecasting tutorial either. How does it fare in the Makridakis competitions?
- VHRanger 8y ago
  You shouldn't listen to N. Taleb on technical matters. He's been a classic mold crank for the last decade or so when it comes to anything serious, relegated instead to writing fluffy books on whatever he thinks is important.
- zwaps 8y ago
  GARCH, like I said, is a short memory process and is inherently inadequate for (longer) out of sample predictions. Doing this is possible, but not really correct. Taleb is basically right, of course what he says is probably inflammatory and half wrong, as usual. Don't forget that most econometrics models are also concerned with identification and causality, less with prediction.
RA_Fisher 8y ago
It's been amazing to watch CS (really the Python community, save statsmodels and patsy) discover statistics. For a while I thought perhaps it was me and statistics that was "behind." Over time I realized that it was mostly re-invention of old ideas: one-hot encoding = dummy variables, neural networks approximating polynomial regression, etc. I decided to double-down on statistics and it's really paid off. NN / random forests and the stats-founded but CS-led approaches are very general models. That leaves statisticians a big opening because a more specific model can be chosen to obtain more accurate predictions. These days I'm positioning myself to clean-up the messes / save broken ML models. Turns out [stats] theory is very practical. :-)
- Eridrus 8y ago
  I'm curious what resources you found useful to learn stats modelling and what sorts of approaches have been useful. On one hand, it's almost a tautoloy that specific models should be better than general models, but I worked on some 2d time series classification with a statistician and afterwards, for kicks, I replaced the entire thing with a CNN+LSTM and it worked just as well as the whole complicated model he had come up with.
  - RA_Fisher 8y ago
    I highly recommend this econometrics text for getting started with statistics: https://www.amazon.com/Principles-Econometrics-5th-Carter-Hill-ebook/dp/B07BNZ6MWM https://www.amazon.com/Principles-Econometrics-5th-Carter-Hi... For modeling I found Wooldridge's panel and cross-section data book very useful: https://www.amazon.com/Econometric-Analysis-Cross-Section-Panel/dp/0262232588 https://www.amazon.com/Econometric-Analysis-Cross-Section-Pa... Greene is a really useful reference text: https://www.amazon.com/Econometric-Analysis-8th-William-Greene/dp/0134461363 https://www.amazon.com/Econometric-Analysis-8th-William-Gree... For advanced stats theory, I recommend Casella and Berger https://www.amazon.com/Statistical-Inference-George-Casella/dp/0534243126 https://www.amazon.com/Statistical-Inference-George-Casella/... Hope that helps! The more specific a model can be made to the problem at hand, the better it'll perform. Supervised ML models are great starting / baseline models.
    - tomrod 8y ago
      I second Wooldridge. Greene I found to be much denser without providing much additional insight. It is a popular MS/PhD entry text though. I add any of Ken Train's work to this mix, especially on estimating discrete choice theory.
- curiousgal 8y ago
  Because saying "relevant username" is frowned upon I'll just point out that R A Fisher is "a genius who almost single-handedly created the foundations for modern statistical science"[0] 0.https://en.m.wikipedia.org/wiki/Ronald_Fisher https://en.m.wikipedia.org/wiki/Ronald_Fisher
laichzeit0 8y ago
I don't think it's entirely fair to say "Computer Scientists are basically rediscovering statistics". LSTMs are used beyond just time series prediction. It is also quite common in language modelling tasks, which is also a sequence modelling task, and where it works quite well. I'm not familiar at all with using GARCH/ARIMA for something like this. Also, with neural networks it's very easy and natural to build complex models where different "layers" perform different tasks. So an LSTM can very easily be extended to work bi-directionally (taking data from the beginning of the sequence, and the end of the sequence), adding things like attention, using word-vectors before the recurrent network or just using a character model. What are the statistical equivalents for this? Because most of the papers on this topic seem to come from Computer Science. Take a look at the epilogue of [1] for a thorough discussion on where statistical theory needs to catch up. [1] Computer Age Statistical Inference - Efron, Hastie.
- srean 8y ago
  > What are the statistical equivalents for this? That would be nonparametric statistics.
  - sidr 8y ago
    > That would be nonparametric statistics. No, it wouldn't. Firstly, nonparametrics in general can be a little misleading. The most common instantiations place function ("process") priors on modeling decisions that are otherwise found through trial and error. Those process priors do have their own parameters though. But more importantly, LSTMs and neural networks are very much parametric - their success come from the advances in computing and optimization that have enabled estimating these parameters in very complicated model structures.
    - srean 8y ago
      Why are you talking about priors ? Nonparametric vs parametric is an axis completely orthogonal to Bayesian vs Frequentist. We weren't talking about the "success" though, I was responding to the question "where in the body of stats literature would a neural net model lie". I argue that would be non-parametric stats. In parametric stats the limit (#params/#data) goes to 0. For models where this is not the case, statisticians and probabilists call them non-parametric (and in certain cases semi-parametric models). Neural net, especially the deep kind (and certainly not the single layer kind) have the property that #params/#data is finite and large.
md2be 8y ago
I agree with your sentiments, but there is a contribution that the CS departments made that the statistics, math, Econ (as in econometrics departments) seemed to have overlooked. I remember going to each of these departments in 2002 and asking them why don’t we split the data sets to train and update the coefficients and automate the process. The answer was always the same “that’s trivial and adds nothing to the field”.
- zwaps 8y ago
  I think that ML is very useful, but remember that forecasting is really not the main objective of econometric models. Basically, forecasting implies you have a good handle on all properties of the relevant distributions, which in my opinion is a lost cause in social sciences (think external validity). Instead, econometrics is nowadays mainly concerned with the identification of causal effect using non-parametric or semi-parametric approaches. Basically, you can believably estimate the directionality of some mechanism, but you probably never have the data or model to make a good out of sample prediction. You can, but it's basically implied that approaches that consistently estimate some marginal of a conditional expectation will NOT be that useful to predict a whole stochastic process. Also, using training and test sets kind of predicates that your process is very stable. Otherwise the "test" set is not really a good test, is it? Again, in social sciences these things are hard to argue. You usually wanna generalize some mechanism from this industry to that industry, not find a good predictor in the same industry. Test datasets still run on the same data! ML is successful because in practice we DO care about prediction. This allows us to do all the cool things. Because econometrics/stats is so conservative and comes from a causal standpoint, people are just really shy to develop a model for prediction (not everywhere true, but that's the gist). For ML, the primary question is basically how good the thing predicts. When I first tried scikit learn way back, I was so confused it didn't offer standard errors or some other statistical measure. But then I saw how ingrained the in-sample, out-sample process is and I thought well - that's really useful. tl;dr: Stats and ML have different objectives, but there is a lot to learn in stats for ML
- digitalzombie 8y ago
  > why don’t we split the data sets to train and update the coefficients and automate the process. What you just stated is just a pipeline. You can just split the data and train it and automate with tree ensemble that aren't boosting that is if you're talking about doing in parallel. If you're just saying split and do as batch process in different time interval you can do that with nonparametric bayesian. CS contribution in creating Deep learning and having it be the best accurate algo for certain data domain is pretty nice. But again Stat care a lot more than prediction.
unhammer 8y ago
Apropos, here's a "Time series shootout: ARIMA vs. LSTM" : https://www.youtube.com/watch?v=h9QWefYBfJg&list=PL51yKFtVfMeCI5OVzqToxFoDfIsYfCS-r&index=15 https://www.youtube.com/watch?v=h9QWefYBfJg&list=PL51yKFtVfM...

by zwaps 8y ago

jkabrg 8y ago

Nassim Taleb had some negative things to say about GARCH. "GARCH does not work out of sample. It is a good story, but I was unable to use it in predicting squared deviations or mean deviations" I haven't found it in Rob J Hyndman's forecasting tutorial either. How does it fare in the Makridakis competitions?

VHRanger 8y ago
You shouldn't listen to N. Taleb on technical matters. He's been a classic mold crank for the last decade or so when it comes to anything serious, relegated instead to writing fluffy books on whatever he thinks is important.
zwaps 8y ago
GARCH, like I said, is a short memory process and is inherently inadequate for (longer) out of sample predictions. Doing this is possible, but not really correct. Taleb is basically right, of course what he says is probably inflammatory and half wrong, as usual. Don't forget that most econometrics models are also concerned with identification and causality, less with prediction.

RA_Fisher 8y ago

It's been amazing to watch CS (really the Python community, save statsmodels and patsy) discover statistics. For a while I thought perhaps it was me and statistics that was "behind." Over time I realized that it was mostly re-invention of old ideas: one-hot encoding = dummy variables, neural networks approximating polynomial regression, etc. I decided to double-down on statistics and it's really paid off. NN / random forests and the stats-founded but CS-led approaches are very general models. That leaves statisticians a big opening because a more specific model can be chosen to obtain more accurate predictions. These days I'm positioning myself to clean-up the messes / save broken ML models. Turns out [stats] theory is very practical. :-)

Eridrus 8y ago
I'm curious what resources you found useful to learn stats modelling and what sorts of approaches have been useful. On one hand, it's almost a tautoloy that specific models should be better than general models, but I worked on some 2d time series classification with a statistician and afterwards, for kicks, I replaced the entire thing with a CNN+LSTM and it worked just as well as the whole complicated model he had come up with.
- RA_Fisher 8y ago
  I highly recommend this econometrics text for getting started with statistics: https://www.amazon.com/Principles-Econometrics-5th-Carter-Hill-ebook/dp/B07BNZ6MWM https://www.amazon.com/Principles-Econometrics-5th-Carter-Hi... For modeling I found Wooldridge's panel and cross-section data book very useful: https://www.amazon.com/Econometric-Analysis-Cross-Section-Panel/dp/0262232588 https://www.amazon.com/Econometric-Analysis-Cross-Section-Pa... Greene is a really useful reference text: https://www.amazon.com/Econometric-Analysis-8th-William-Greene/dp/0134461363 https://www.amazon.com/Econometric-Analysis-8th-William-Gree... For advanced stats theory, I recommend Casella and Berger https://www.amazon.com/Statistical-Inference-George-Casella/dp/0534243126 https://www.amazon.com/Statistical-Inference-George-Casella/... Hope that helps! The more specific a model can be made to the problem at hand, the better it'll perform. Supervised ML models are great starting / baseline models.
  - tomrod 8y ago
    I second Wooldridge. Greene I found to be much denser without providing much additional insight. It is a popular MS/PhD entry text though. I add any of Ken Train's work to this mix, especially on estimating discrete choice theory.
curiousgal 8y ago
Because saying "relevant username" is frowned upon I'll just point out that R A Fisher is "a genius who almost single-handedly created the foundations for modern statistical science"[0] 0.https://en.m.wikipedia.org/wiki/Ronald_Fisher https://en.m.wikipedia.org/wiki/Ronald_Fisher

laichzeit0 8y ago

I don't think it's entirely fair to say "Computer Scientists are basically rediscovering statistics". LSTMs are used beyond just time series prediction. It is also quite common in language modelling tasks, which is also a sequence modelling task, and where it works quite well. I'm not familiar at all with using GARCH/ARIMA for something like this. Also, with neural networks it's very easy and natural to build complex models where different "layers" perform different tasks. So an LSTM can very easily be extended to work bi-directionally (taking data from the beginning of the sequence, and the end of the sequence), adding things like attention, using word-vectors before the recurrent network or just using a character model. What are the statistical equivalents for this? Because most of the papers on this topic seem to come from Computer Science. Take a look at the epilogue of [1] for a thorough discussion on where statistical theory needs to catch up. [1] Computer Age Statistical Inference - Efron, Hastie.

srean 8y ago
> What are the statistical equivalents for this? That would be nonparametric statistics.
- sidr 8y ago
  > That would be nonparametric statistics. No, it wouldn't. Firstly, nonparametrics in general can be a little misleading. The most common instantiations place function ("process") priors on modeling decisions that are otherwise found through trial and error. Those process priors do have their own parameters though. But more importantly, LSTMs and neural networks are very much parametric - their success come from the advances in computing and optimization that have enabled estimating these parameters in very complicated model structures.
  - srean 8y ago
    Why are you talking about priors ? Nonparametric vs parametric is an axis completely orthogonal to Bayesian vs Frequentist. We weren't talking about the "success" though, I was responding to the question "where in the body of stats literature would a neural net model lie". I argue that would be non-parametric stats. In parametric stats the limit (#params/#data) goes to 0. For models where this is not the case, statisticians and probabilists call them non-parametric (and in certain cases semi-parametric models). Neural net, especially the deep kind (and certainly not the single layer kind) have the property that #params/#data is finite and large.

md2be 8y ago

I agree with your sentiments, but there is a contribution that the CS departments made that the statistics, math, Econ (as in econometrics departments) seemed to have overlooked. I remember going to each of these departments in 2002 and asking them why don’t we split the data sets to train and update the coefficients and automate the process. The answer was always the same “that’s trivial and adds nothing to the field”.

zwaps 8y ago
I think that ML is very useful, but remember that forecasting is really not the main objective of econometric models. Basically, forecasting implies you have a good handle on all properties of the relevant distributions, which in my opinion is a lost cause in social sciences (think external validity). Instead, econometrics is nowadays mainly concerned with the identification of causal effect using non-parametric or semi-parametric approaches. Basically, you can believably estimate the directionality of some mechanism, but you probably never have the data or model to make a good out of sample prediction. You can, but it's basically implied that approaches that consistently estimate some marginal of a conditional expectation will NOT be that useful to predict a whole stochastic process. Also, using training and test sets kind of predicates that your process is very stable. Otherwise the "test" set is not really a good test, is it? Again, in social sciences these things are hard to argue. You usually wanna generalize some mechanism from this industry to that industry, not find a good predictor in the same industry. Test datasets still run on the same data! ML is successful because in practice we DO care about prediction. This allows us to do all the cool things. Because econometrics/stats is so conservative and comes from a causal standpoint, people are just really shy to develop a model for prediction (not everywhere true, but that's the gist). For ML, the primary question is basically how good the thing predicts. When I first tried scikit learn way back, I was so confused it didn't offer standard errors or some other statistical measure. But then I saw how ingrained the in-sample, out-sample process is and I thought well - that's really useful. tl;dr: Stats and ML have different objectives, but there is a lot to learn in stats for ML
digitalzombie 8y ago
> why don’t we split the data sets to train and update the coefficients and automate the process. What you just stated is just a pipeline. You can just split the data and train it and automate with tree ensemble that aren't boosting that is if you're talking about doing in parallel. If you're just saying split and do as batch process in different time interval you can do that with nonparametric bayesian. CS contribution in creating Deep learning and having it be the best accurate algo for certain data domain is pretty nice. But again Stat care a lot more than prediction.

unhammer 8y ago

Apropos, here's a "Time series shootout: ARIMA vs. LSTM" : https://www.youtube.com/watch?v=h9QWefYBfJg&list=PL51yKFtVfMeCI5OVzqToxFoDfIsYfCS-r&index=15 https://www.youtube.com/watch?v=h9QWefYBfJg&list=PL51yKFtVfM...