3 ms·
That's a red herring, and here's why: simpler models take less data to train. Every orthogonal parameter that you add to a statistical model expands the proba
by timr 19y ago
That's a red herring, and here's why: simpler models take less data to train.
Every orthogonal parameter that you add to a statistical model expands the probability space of that model exponentially. If you keep the amount of training data fixed, but keep adding parameters, you'll very quickly arrive at a point where your model will fit your training data exceptionally well, but the result will be meaningless, because you've over-fit your data.
Now, I suppose you could make the case that new models are occasionally discovered that allow for more efficient uses of existing data. However, this is a decidedly rare event. Most of the time, gains are made in exactly the way the author suggests -- by adding new sources of data to well-established techniques.