4 ms·

"Any model, if given enough parameters, can be made to match historical data to an arbitrary degree." I've known this forever but for some reason haven't heard

by api 12y ago

"Any model, if given enough parameters, can be made to match historical data to an arbitrary degree." I've known this forever but for some reason haven't heard this precise statement of it. Thanks. Reductio ad absurdium: imagine a model where the number of parameters equals the number of data points. Obviously that model will have perfect fit. Predicting the future is hard. Predicting the future without a causal understanding of the system is epistemologically questionable.

arno_v 12y ago
This picture shows quite nicely what might happen when having too many parameters (or too little data): http://machinelearningac.files.wordpress.com/2011/10/polynomials.png http://machinelearningac.files.wordpress.com/2011/10/polynom...
- vmarsy 12y ago
  In red is your model whereas in green is the real one, M being the number of parameters. The technical term for the last one is "overfitting" if I remember correctly. But in the case you have an enormous amount of data, it is unlikely to happen. It reminds me of this awesome course: https://www.coursera.org/course/ml https://www.coursera.org/course/ml edit: The parent's parent's parent mention overfit for the MIT work, I don't think it'd be the case if you have that amount of data in hands
  - barisser 12y ago
    Here's another way to think of it. If the parameter space for my model includes, let's say 10 binary decisions (which is very conservative), that's 1024 possible states of my model. If I tested all 1024 states against historical data, it is likely that some of them might do very well (depending on the general architecture of the model of course). What if I then selected the successful minority and held them up as clever strategies? Their success would very likely have been arbitrary. By basically brute-forcing enough strategies, I will inevitably come across some that were historically successful. But these same historically successful strategies are unlikely to outperform another random strategy in the future. It's not impossible you'll find a nugget of wisdom hidden from everyone else, just much less likely than the more simple explanation I'm offering. So to your point, it's not just the size of the parameter space versus the data set that matters. Brute-forcing the former alone will likely produce a deceptive minority of winners.
    - Enzolangellotti 12y ago
      There is a fun chapter on this topic in Jordan Ellenberg's latest book "How not to be wrong". It's called the "Baltimore stockbroker fraud".
  - Houshalter 12y ago
    It's entirely possible to overfit with enormous amounts of data. As people are now creating models with enormous numbers of parameters.
- jessaustin 12y ago
  "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." -- John von Neumann The green "real" signal in your picture is amusing when juxtaposed with the red "zero-noise-assumption" signal in the last frame. TBH this accounts for most of my distrust of e.g. climate modeling.
dratman 12y ago
"Predicting the future without a causal understanding of the system is epistemologically questionable." That is an intriguing assertion, but it is circular. One can only demonstrate a causal understanding of a system by making usefully accurate predictions about the system's behavior. To attempt that with a system consisting of market prices of tradeable securities is an exercise in frustration, because such markets do not operate by consistent, unchanging causal rules. In fact, financial markets are not systems at all in the usual sense, because their parts and connections are continually changing.

by api 12y ago

arno_v 12y ago

This picture shows quite nicely what might happen when having too many parameters (or too little data): http://machinelearningac.files.wordpress.com/2011/10/polynomials.png http://machinelearningac.files.wordpress.com/2011/10/polynom...

vmarsy 12y ago
In red is your model whereas in green is the real one, M being the number of parameters. The technical term for the last one is "overfitting" if I remember correctly. But in the case you have an enormous amount of data, it is unlikely to happen. It reminds me of this awesome course: https://www.coursera.org/course/ml https://www.coursera.org/course/ml edit: The parent's parent's parent mention overfit for the MIT work, I don't think it'd be the case if you have that amount of data in hands
- barisser 12y ago
  Here's another way to think of it. If the parameter space for my model includes, let's say 10 binary decisions (which is very conservative), that's 1024 possible states of my model. If I tested all 1024 states against historical data, it is likely that some of them might do very well (depending on the general architecture of the model of course). What if I then selected the successful minority and held them up as clever strategies? Their success would very likely have been arbitrary. By basically brute-forcing enough strategies, I will inevitably come across some that were historically successful. But these same historically successful strategies are unlikely to outperform another random strategy in the future. It's not impossible you'll find a nugget of wisdom hidden from everyone else, just much less likely than the more simple explanation I'm offering. So to your point, it's not just the size of the parameter space versus the data set that matters. Brute-forcing the former alone will likely produce a deceptive minority of winners.
  - Enzolangellotti 12y ago
    There is a fun chapter on this topic in Jordan Ellenberg's latest book "How not to be wrong". It's called the "Baltimore stockbroker fraud".
- Houshalter 12y ago
  It's entirely possible to overfit with enormous amounts of data. As people are now creating models with enormous numbers of parameters.
jessaustin 12y ago
"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." -- John von Neumann The green "real" signal in your picture is amusing when juxtaposed with the red "zero-noise-assumption" signal in the last frame. TBH this accounts for most of my distrust of e.g. climate modeling.

dratman 12y ago

"Predicting the future without a causal understanding of the system is epistemologically questionable." That is an intriguing assertion, but it is circular. One can only demonstrate a causal understanding of a system by making usefully accurate predictions about the system's behavior. To attempt that with a system consisting of market prices of tradeable securities is an exercise in frustration, because such markets do not operate by consistent, unchanging causal rules. In fact, financial markets are not systems at all in the usual sense, because their parts and connections are continually changing.