3 ms·

I think what the author is describing is simple overfitting. http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting It is quite a n

I think what the author is describing is simple overfitting. http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting It is quite a newbie mistake for a scientist to be surprised by it. It affects every kind of modelling. I thought maybe this article would talk about why economic models are worst than other kinds of models. There are issues that arise when applying scientific models to the economy caused by the fact that when even good models are used to predict markets, the use of the models themselves to do trading, distorts the markets. When multiple parties use good models to compete in markets, they distort the markets in such a way that destroys the predictive power of the models. There is a great explanation by Glen Whitman of Agoraphilia, that uses grocery line wait time predictions as a metaphor for this: http://agoraphilia.blogspot.com/2005/03/doing-lines.html http://agoraphilia.blogspot.com/2005/03/doing-lines.html See also: http://lesswrong.com/lw/yv/markets_are_antiinductive/ http://lesswrong.com/lw/yv/markets_are_antiinductive/ http://en.wikipedia.org/wiki/Efficient-market_hypothesis http://en.wikipedia.org/wiki/Efficient-market_hypothesis

jerf 15y ago
Alternatively, it may be simple information theory: A model that takes in 100 bits of specification simply can not correctly describe a process that has 10,000 bit's worth of degrees of freedom. And that's before we talk about iteration over time, and before we get to the final killer you mention, which is when the models are ruined by their own application to the domain. I think radical underspecification is much more likely than overspecification, really. (Since I encounter this a lot, let me pre-answer one question in advance, which is "What if only 300 bits really matter and the rest don't matter as much?" and the answer is that the term bit in information theory encompasses that idea already. If you have ten "bits", but they tend to be highly correlated together such that they are usually all 0 or all 1, you in fact don't have ten bits in information theory. Ten bits are, by definition, ten fully-independent true or false values. Bits-in-memory are not the same as information-theory-bits. A real system with 10,000 bits can not, pretty much by definition, be modeled by 100 bits. If it could, it would be a system with only 100 bits in the first place. Information theory cares about the true degrees of freedom available, not about your particular representation of the system.)
- amalcon 15y ago
  Here's the thing: you're both right. It's both radically underspecified and overfitted. The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality. This article speaks of the separate problem that economic models are not evaluated in any sort of experiments, and thus are prone to overfitting. This makes them unlikely to even approximate well. Consider a basic multilayer perceptron-style neural network. Overfitting is a well-understood problem in training an MLP. We work around it by training on a part of the data, and then measuring its accuracy on another part -- much as Carter did in his analysis. If the accuracy is poor, something is adjusted: the size of the hidden layer can be increased, the training set expanded, the duration of the training increased or decreased, or the MLP model discarded entirely. If increase of the training set or reduction of the duration improves accuracy against the test set, this means we had an overfitting problem.
  - jeffdavis 15y ago
    "It's both radically underspecified and overfitted." He used a perfect model (of a hypothetical world) which had exactly the right parameters, and then he calibrated it using exactly correct data. So I don't see how this could be underspecified or overfitted. Can you please explain? "The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality." In this case he defined his model to be reality.
    - amalcon 15y ago
      Those particular statements referred to some representative economic model, not the experiment in question. In the experiment in question, the model is fully specified by definition. As far as overfitting goes, that applies when you have a parameterized general model and need to discover the correct parameters. You probably won't get the exact correct parameters; instead, you'll (hopefully) get parameters that approximate reality well. More closely matching the training data can actually make it a worse approximation in the general case.
  - 15y ago
jeffdavis 15y ago
"I think what the author is describing is simple overfitting." It doesn't look like overfitting to me. The input data is perfect, and the model is perfect, so it doesn't look like overfitting can occur.

jerf 15y ago

Alternatively, it may be simple information theory: A model that takes in 100 bits of specification simply can not correctly describe a process that has 10,000 bit's worth of degrees of freedom. And that's before we talk about iteration over time, and before we get to the final killer you mention, which is when the models are ruined by their own application to the domain. I think radical underspecification is much more likely than overspecification, really. (Since I encounter this a lot, let me pre-answer one question in advance, which is "What if only 300 bits really matter and the rest don't matter as much?" and the answer is that the term bit in information theory encompasses that idea already. If you have ten "bits", but they tend to be highly correlated together such that they are usually all 0 or all 1, you in fact don't have ten bits in information theory. Ten bits are, by definition, ten fully-independent true or false values. Bits-in-memory are not the same as information-theory-bits. A real system with 10,000 bits can not, pretty much by definition, be modeled by 100 bits. If it could, it would be a system with only 100 bits in the first place. Information theory cares about the true degrees of freedom available, not about your particular representation of the system.)

amalcon 15y ago
Here's the thing: you're both right. It's both radically underspecified and overfitted. The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality. This article speaks of the separate problem that economic models are not evaluated in any sort of experiments, and thus are prone to overfitting. This makes them unlikely to even approximate well. Consider a basic multilayer perceptron-style neural network. Overfitting is a well-understood problem in training an MLP. We work around it by training on a part of the data, and then measuring its accuracy on another part -- much as Carter did in his analysis. If the accuracy is poor, something is adjusted: the size of the hidden layer can be increased, the training set expanded, the duration of the training increased or decreased, or the MLP model discarded entirely. If increase of the training set or reduction of the duration improves accuracy against the test set, this means we had an overfitting problem.
- jeffdavis 15y ago
  "It's both radically underspecified and overfitted." He used a perfect model (of a hypothetical world) which had exactly the right parameters, and then he calibrated it using exactly correct data. So I don't see how this could be underspecified or overfitted. Can you please explain? "The information-theoretic argument demonstrate that a model cannot exactly match the reality unless it's as complex as the reality." In this case he defined his model to be reality.
  - amalcon 15y ago
    Those particular statements referred to some representative economic model, not the experiment in question. In the experiment in question, the model is fully specified by definition. As far as overfitting goes, that applies when you have a parameterized general model and need to discover the correct parameters. You probably won't get the exact correct parameters; instead, you'll (hopefully) get parameters that approximate reality well. More closely matching the training data can actually make it a worse approximation in the general case.
- 15y ago

jeffdavis 15y ago

"I think what the author is describing is simple overfitting." It doesn't look like overfitting to me. The input data is perfect, and the model is perfect, so it doesn't look like overfitting can occur.