4 ms·

Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graph

Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: https://jamesclear.com/wp-content/uploads/2015/04/logarithmic-growth-curve-1200x800.jpg https://jamesclear.com/wp-content/uploads/2015/04/logarithmi... So in point of fact, according to the cited paper Google is asserting there are diminishing returns to increasing the volume of data.

nostrademons 8y ago
The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second. The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.
- freyir 8y ago
  The headline isn't consistent with the article itself, which states (somewhat confusingly): * As the number of product data grows, the benefits were negligible * More observations per product was important * The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. In any case, this seems to be a case of "diminishing returns."
- throwawaymath 8y ago
  I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.
- YeGoblynQueenne 8y ago
  >> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".
  - nostrademons 8y ago
    Few big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.
    - YeGoblynQueenne 8y ago
      If it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al. Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.
      - nostrademons 8y ago
        Only a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging. To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.
sampo 8y ago
> Logarithmic growth asymptotically decreases Logarithmic growth slows down, but not asymptotically. Think about it: what would the asymptote be? (There is none.)
- ThePhysicist 8y ago
  100 % accuracy is what the algorithm wants to achieve (in the most basic case), so that's your asymptote. If your precision increases logarithmically e.g. as 1-1/log(x) you get arbitrarily close to 1, which is the definition of an asymptote I think.
  - sampo 8y ago
    > 100 % accuracy is what the algorithm wants to achieve Well, in that context every kind of (monotonic) growth is asymptotic, so the word has no meaning.
- air7 8y ago
  It really slows down though. Log(n) indeed goes to infinity but if you numbered all the atoms in the universe and pass them through the function, the last one would be ~80.
  - siekmanj 8y ago
    That may be true, but that is a far cry from asymptotic
    - air7 8y ago
      Mathematically, yes. For all real-world intents and purposes, it's bounded. I sometimes use this very example to illustrate the profound difference between the two which can be easily under-estimated. (That or log(log(n)) which also goes to infinity but is real-world bounded at <3)
  - sampo 8y ago
    That is in log_10, in log_e it would be near 190.

nostrademons 8y ago

The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second. The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

freyir 8y ago
The headline isn't consistent with the article itself, which states (somewhat confusingly): * As the number of product data grows, the benefits were negligible * More observations per product was important * The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. In any case, this seems to be a case of "diminishing returns."
throwawaymath 8y ago
I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.
YeGoblynQueenne 8y ago
>> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".
- nostrademons 8y ago
  Few big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.
  - YeGoblynQueenne 8y ago
    If it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al. Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.
    - nostrademons 8y ago
      Only a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging. To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.

sampo 8y ago

> Logarithmic growth asymptotically decreases Logarithmic growth slows down, but not asymptotically. Think about it: what would the asymptote be? (There is none.)

ThePhysicist 8y ago
100 % accuracy is what the algorithm wants to achieve (in the most basic case), so that's your asymptote. If your precision increases logarithmically e.g. as 1-1/log(x) you get arbitrarily close to 1, which is the definition of an asymptote I think.
- sampo 8y ago
  > 100 % accuracy is what the algorithm wants to achieve Well, in that context every kind of (monotonic) growth is asymptotic, so the word has no meaning.
air7 8y ago
It really slows down though. Log(n) indeed goes to infinity but if you numbered all the atoms in the universe and pass them through the function, the last one would be ~80.
- siekmanj 8y ago
  That may be true, but that is a far cry from asymptotic
  - air7 8y ago
    Mathematically, yes. For all real-world intents and purposes, it's bounded. I sometimes use this very example to illustrate the profound difference between the two which can be easily under-estimated. (That or log(log(n)) which also goes to infinity but is real-world bounded at <3)
- sampo 8y ago
  That is in log_10, in log_e it would be near 190.