3 ms·
A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal
by JoshuaDavid 8y ago
A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
- throwawaymath 8y agoRight. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.