8 ms·

In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of trai

by tedd4u 8y ago

In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of training data." Maybe for supply-chain and regression models there is a limit but it seems in deep neural nets maybe there's a different answer. Blog: https://ai.googleblog.com/2017/07/revisiting-unreasonable-effectiveness.html https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef... Paper: https://arxiv.org/abs/1707.02968 https://arxiv.org/abs/1707.02968

FPGAhacker 8y ago
Isn’t that the very definition of diminishing returns?
- mactrey 8y ago
  Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n
  - JoshuaDavid 8y ago
    A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
    - throwawaymath 8y ago
      Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.
- hopler 8y ago
  As a scientific term, "diminishing" means its derivative has limit/asymptote of 0 or negative approached fast enough that the integral (or series sum) is bounded. Logarithm, with derivative 1/x is the well-behaved function with smallest (ignoring constant factors and trivial higher order terms) non-diminishing return -- its returns flatten too slowly to bound it's value.
- throwawaymath 8y ago
  Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: https://jamesclear.com/wp-content/uploads/2015/04/logarithmic-growth-curve-1200x800.jpg https://jamesclear.com/wp-content/uploads/2015/04/logarithmi... So in point of fact, according to the cited paper Google is asserting there are diminishing returns to increasing the volume of data.
  - nostrademons 8y ago
    The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second. The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.
    - freyir 8y ago
      The headline isn't consistent with the article itself, which states (somewhat confusingly): * As the number of product data grows, the benefits were negligible * More observations per product was important * The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. In any case, this seems to be a case of "diminishing returns."
    - throwawaymath 8y ago
      I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.
    - YeGoblynQueenne 8y ago
      >> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".
burtonator 8y ago
With diminishing returns it's interesting to think about this as a purely economic function when it makes sense to index more data. If a 5% increase in accuracy worth more than the cost of data storage and computation then it's a pretty clear win.
simonbyrne 8y ago
I know it's an empirical result, but it would be interesting if it was truly logarithmic. Classic statistics gives various asymptotic results based on variations of the central limit theorem, which generally imply sqrt(n) scaling, but these assume various regularity conditions (typically smoothness and fixed dimensionality). It would be interesting if there was some corresponding law for more flexible models which did indeed give you logarithmic scaling.
- moultano 8y ago
  If the model is an ideal encoding, then the description length of every input grows logarithmically. Perhaps the performance is gated on the number of "features" in the ideal encoding.

by tedd4u 8y ago

FPGAhacker 8y ago

Isn’t that the very definition of diminishing returns?

mactrey 8y ago
Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n
- JoshuaDavid 8y ago
  A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
  - throwawaymath 8y ago
    Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.
hopler 8y ago
As a scientific term, "diminishing" means its derivative has limit/asymptote of 0 or negative approached fast enough that the integral (or series sum) is bounded. Logarithm, with derivative 1/x is the well-behaved function with smallest (ignoring constant factors and trivial higher order terms) non-diminishing return -- its returns flatten too slowly to bound it's value.
throwawaymath 8y ago
Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: https://jamesclear.com/wp-content/uploads/2015/04/logarithmic-growth-curve-1200x800.jpg https://jamesclear.com/wp-content/uploads/2015/04/logarithmi... So in point of fact, according to the cited paper Google is asserting there are diminishing returns to increasing the volume of data.
- nostrademons 8y ago
  The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second. The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.
  - freyir 8y ago
    The headline isn't consistent with the article itself, which states (somewhat confusingly): * As the number of product data grows, the benefits were negligible * More observations per product was important * The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. In any case, this seems to be a case of "diminishing returns."
  - throwawaymath 8y ago
    I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.
  - YeGoblynQueenne 8y ago
    >> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".

burtonator 8y ago

With diminishing returns it's interesting to think about this as a purely economic function when it makes sense to index more data. If a 5% increase in accuracy worth more than the cost of data storage and computation then it's a pretty clear win.

simonbyrne 8y ago

I know it's an empirical result, but it would be interesting if it was truly logarithmic. Classic statistics gives various asymptotic results based on variations of the central limit theorem, which generally imply sqrt(n) scaling, but these assume various regularity conditions (typically smoothness and fixed dimensionality). It would be interesting if there was some corresponding law for more flexible models which did indeed give you logarithmic scaling.

moultano 8y ago
If the model is an ideal encoding, then the description length of every input grows logarithmically. Perhaps the performance is gated on the number of "features" in the ideal encoding.