6 ms·

Isn’t that the very definition of diminishing returns?

• mactrey 8y ago
  Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n
  • JoshuaDavid 8y ago
    A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
    • throwawaymath 8y ago
      Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.
• hopler 8y ago
  As a scientific term, "diminishing" means its derivative has limit/asymptote of 0 or negative approached fast enough that the integral (or series sum) is bounded. Logarithm, with derivative 1/x is the well-behaved function with smallest (ignoring constant factors and trivial higher order terms) non-diminishing return -- its returns flatten too slowly to bound it's value.
• throwawaymath 8y ago
  Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: https://jamesclear.com/wp-content/uploads/2015/04/logarithmic-growth-curve-1200x800.jpg https://jamesclear.com/wp-content/uploads/2015/04/logarithmi... So in point of fact, according to the cited paper Google is asserting there are diminishing returns to increasing the volume of data.
  • nostrademons 8y ago
    The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second. The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.
    • freyir 8y ago
      The headline isn't consistent with the article itself, which states (somewhat confusingly): * As the number of product data grows, the benefits were negligible * More observations per product was important * The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. In any case, this seems to be a case of "diminishing returns."
    • throwawaymath 8y ago
      I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.
    • YeGoblynQueenne 8y ago
      >> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".