5 ms·

The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson

The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

version_five 4y ago
Yes I just read it too. It appears to have been discussed here multiple times: https://hn.algolia.com/?q=bitter+lesson https://hn.algolia.com/?q=bitter+lesson I'd say it has aged well and there are probably lots of new takes on it based on some of the achievements of the last 6 months. At the same time, I don't agree with it. To simplify, the opposite of "don't over-optimize" isn't "it's too complicated to understand so never try". It is thought provoking though
sva_ 4y ago
I think there is no doubt that there must be more efficient model architectures out there, take for example the sample efficiency of GPT-3: > If you think about what a human, a human probably in a human’s lifetime, 70 years, processes probably about a half a billion words, maybe a billion, let’s say a billion. So when you think about it, GPT-3 has been trained on 57 billion times the number of words that a human in his or her lifetime will ever perceive.[0] 0. https://hai.stanford.edu/news/gpt-3-intelligent-directors-conversation-oren-etzioni https://hai.stanford.edu/news/gpt-3-intelligent-directors-co...
- lostmsu 4y ago
  Humans are not trianed on words.
  - lgas 4y ago
    What do you mean? That we are not trained ONLY on words? Or that the training we receive on words is not the same as the training of a NN? Or something else?
    - lostmsu 4y ago
      We are not trained only on words. In fact, we are not trained on words at all. We are trained on images, sounds, tactile, and a few more types of sources.
      - lgas 4y ago
        Seems odd to say we're trained on sounds but not words, especially when many of the sounds we hear while we are developing are words being spoken.
- IanCal 4y ago
  I cannot wrap my head around that. I listened to the audio to check it wasn't a transcription error and don't think it is. He is claiming GPT3 was trained on 57 billion billion words. The training dataset is something like 500B tokens and not all of that is used (common crawl is processed less than once), and I'm damn near certain that it wasn't trained for a hundred million epochs. Their original paper says the largest model was trained on 300B tokens [0] Assuming a token is a word, as we're going for orders of magnitude, you're actually looking at about a few hundred times more text. The point kind of stands, it's more, but not billions of times. I wouldn't be surprised if I'm wrong here because they seem to be an expert but this didn't pass the sniff test and looking into it doesn't support what they're saying to me. [0] https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf appendix D
  - sva_ 4y ago
    I didn't even catch that. Surely they meant 57 billion words. Some words are broken up into several tokens, which might explain the 300B tokens.
    - IanCal 4y ago
      It's at about 7 minutes in the video, they really do say it several times in a few ways. He starts by saying it's trained on 570 billion megabytes, which is probably where this confusion starts. Looking again at the paper, Common Crawl after filtering is 570GB or 570 billion bytes. So he makes two main mistakes - one is straight up multiplying by another million, then by assuming one byte is equivalent to one word. Then a bit more because less than half of it is used. That's probably taking it out by a factor of about ten million or more. 300B is then the "training budget" in a sense, not every dataset is used in its entirety, some are processed more than once, but each of the GPT3 sizes were trained on 300B tokens.
sweezyjeezy 4y ago
In the years since this came out I have shifted my opinion from 100% agreement to kind of the opposite in a lot of cases, a bitter lesson from using AI to solve complex end-to-end tasks. If you want to build a system that will get the best score on a test set - he's right, get all the data you possibly can, try to make your model as e2e as possible. This often has the advantage of being the easiest approach. The problem is: your input dataset definitely has biases that you don't know about, and the model is going to learn spurious things that you don't want it to. It can make some indefensible decisions with high confidence and you may not know why. Some domains your model may be basically useless, and you won't know this until it happens. This often isn't good enough in industry. To stop batshit things coming out of your system, you may have to do the opposite - use domain knowledge to break the problem the model is trying to solve down into steps you can reason about. Use this to improve your data. Use this to stop the system from doing anything you know makes no sense. This is really hard and time consuming, but IMO complex e2e models are something to be wary of.
- abernard1 4y ago
  Your understanding that problem solving has to do with "steps" is the correct one in my opinion. Also, with data driven approaches, the model isn't necessarily learned in any meaningful way. If you train with certain inputs to get certain realistic looking outputs, you can build sophisticated parrots or chameleons. That's what GPT and stable diffusion are: compressed knowledge bases to get an output without a knowledge model. (No, language models are not knowledge models). Thinking, rational or otherwise, requires causal steps. Since none of these data driven approaches have even fuzzy causal models, they require memorizing infinite universes to see if search can find a particular universe that's seen this before. That's why they're not intelligent and never will be. An intelligent animal only needs one universe and limited experiences to solve a problem because it knows the latent structure of the problem and can generate approaches. Intelligent entities, unlike these autistic Rain Man automatons, do not need to memorize every book in the library to multiply two large numbers together.
gnfargbl 4y ago
One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a huge power budget to get anywhere close to a brain; this estimate [2] puts training GPT-3 at about a TWh, which is about six million years' of power for a single brain. [1] https://www.pnas.org/doi/10.1073/pnas.2107022118 https://www.pnas.org/doi/10.1073/pnas.2107022118 [2] https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-planet/#:~:text=One%20popular%20model%2C%20GPT%2D3,GPT%2D3%20consumed%20936%20MWh https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-pl....
- dsign 4y ago
  Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or playing Go. Have you ever talked to a System's Biologist? Ask them about how anything works, and they'll start with 'oh, it's so complicated. You have X, and then Y, and then Z, and nobody knows about W. And then how they work together? Madness!"
  - deleted 4y ago
    [deleted]
  - TeMPOraL 4y ago
    > Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally). I also imagine it wouldn't be larger than six thousand people specifically raised and taught in coordinated fashion to fulfill this role. On the other other hand, a human brain does a lot more than learning language, ideas and conversion between one and the other. It also, simultaneously, learns a lot of video and audio processing, not to mention smell, proprioception, touch (including pain and temperature), and... a bunch of other stuff (I was actually surprised by the size of the "human" part of the Wikipedia infobox here: https://en.wikipedia.org/wiki/Template:Sensation_and_perception https://en.wikipedia.org/wiki/Template:Sensation_and_percept...). So it's probably hard to compare to specialized NN models until we learn to better classify and isolate how biological brains learn and encode all the various things they do.

The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

version_five 4y ago

Yes I just read it too. It appears to have been discussed here multiple times: https://hn.algolia.com/?q=bitter+lesson https://hn.algolia.com/?q=bitter+lesson I'd say it has aged well and there are probably lots of new takes on it based on some of the achievements of the last 6 months. At the same time, I don't agree with it. To simplify, the opposite of "don't over-optimize" isn't "it's too complicated to understand so never try". It is thought provoking though

sva_ 4y ago

I think there is no doubt that there must be more efficient model architectures out there, take for example the sample efficiency of GPT-3: > If you think about what a human, a human probably in a human’s lifetime, 70 years, processes probably about a half a billion words, maybe a billion, let’s say a billion. So when you think about it, GPT-3 has been trained on 57 billion times the number of words that a human in his or her lifetime will ever perceive.[0] 0. https://hai.stanford.edu/news/gpt-3-intelligent-directors-conversation-oren-etzioni https://hai.stanford.edu/news/gpt-3-intelligent-directors-co...

lostmsu 4y ago
Humans are not trianed on words.
- lgas 4y ago
  What do you mean? That we are not trained ONLY on words? Or that the training we receive on words is not the same as the training of a NN? Or something else?
  - lostmsu 4y ago
    We are not trained only on words. In fact, we are not trained on words at all. We are trained on images, sounds, tactile, and a few more types of sources.
    - lgas 4y ago
      Seems odd to say we're trained on sounds but not words, especially when many of the sounds we hear while we are developing are words being spoken.
IanCal 4y ago
I cannot wrap my head around that. I listened to the audio to check it wasn't a transcription error and don't think it is. He is claiming GPT3 was trained on 57 billion billion words. The training dataset is something like 500B tokens and not all of that is used (common crawl is processed less than once), and I'm damn near certain that it wasn't trained for a hundred million epochs. Their original paper says the largest model was trained on 300B tokens [0] Assuming a token is a word, as we're going for orders of magnitude, you're actually looking at about a few hundred times more text. The point kind of stands, it's more, but not billions of times. I wouldn't be surprised if I'm wrong here because they seem to be an expert but this didn't pass the sniff test and looking into it doesn't support what they're saying to me. [0] https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf appendix D
- sva_ 4y ago
  I didn't even catch that. Surely they meant 57 billion words. Some words are broken up into several tokens, which might explain the 300B tokens.
  - IanCal 4y ago
    It's at about 7 minutes in the video, they really do say it several times in a few ways. He starts by saying it's trained on 570 billion megabytes, which is probably where this confusion starts. Looking again at the paper, Common Crawl after filtering is 570GB or 570 billion bytes. So he makes two main mistakes - one is straight up multiplying by another million, then by assuming one byte is equivalent to one word. Then a bit more because less than half of it is used. That's probably taking it out by a factor of about ten million or more. 300B is then the "training budget" in a sense, not every dataset is used in its entirety, some are processed more than once, but each of the GPT3 sizes were trained on 300B tokens.

sweezyjeezy 4y ago

In the years since this came out I have shifted my opinion from 100% agreement to kind of the opposite in a lot of cases, a bitter lesson from using AI to solve complex end-to-end tasks. If you want to build a system that will get the best score on a test set - he's right, get all the data you possibly can, try to make your model as e2e as possible. This often has the advantage of being the easiest approach. The problem is: your input dataset definitely has biases that you don't know about, and the model is going to learn spurious things that you don't want it to. It can make some indefensible decisions with high confidence and you may not know why. Some domains your model may be basically useless, and you won't know this until it happens. This often isn't good enough in industry. To stop batshit things coming out of your system, you may have to do the opposite - use domain knowledge to break the problem the model is trying to solve down into steps you can reason about. Use this to improve your data. Use this to stop the system from doing anything you know makes no sense. This is really hard and time consuming, but IMO complex e2e models are something to be wary of.

abernard1 4y ago
Your understanding that problem solving has to do with "steps" is the correct one in my opinion. Also, with data driven approaches, the model isn't necessarily learned in any meaningful way. If you train with certain inputs to get certain realistic looking outputs, you can build sophisticated parrots or chameleons. That's what GPT and stable diffusion are: compressed knowledge bases to get an output without a knowledge model. (No, language models are not knowledge models). Thinking, rational or otherwise, requires causal steps. Since none of these data driven approaches have even fuzzy causal models, they require memorizing infinite universes to see if search can find a particular universe that's seen this before. That's why they're not intelligent and never will be. An intelligent animal only needs one universe and limited experiences to solve a problem because it knows the latent structure of the problem and can generate approaches. Intelligent entities, unlike these autistic Rain Man automatons, do not need to memorize every book in the library to multiply two large numbers together.

gnfargbl 4y ago

One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a huge power budget to get anywhere close to a brain; this estimate [2] puts training GPT-3 at about a TWh, which is about six million years' of power for a single brain. [1] https://www.pnas.org/doi/10.1073/pnas.2107022118 https://www.pnas.org/doi/10.1073/pnas.2107022118 [2] https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-planet/#:~:text=One%20popular%20model%2C%20GPT%2D3,GPT%2D3%20consumed%20936%20MWh https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-pl....

dsign 4y ago
Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or playing Go. Have you ever talked to a System's Biologist? Ask them about how anything works, and they'll start with 'oh, it's so complicated. You have X, and then Y, and then Z, and nobody knows about W. And then how they work together? Madness!"
- deleted 4y ago
  [deleted]
- TeMPOraL 4y ago
  > Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally). I also imagine it wouldn't be larger than six thousand people specifically raised and taught in coordinated fashion to fulfill this role. On the other other hand, a human brain does a lot more than learning language, ideas and conversion between one and the other. It also, simultaneously, learns a lot of video and audio processing, not to mention smell, proprioception, touch (including pain and temperature), and... a bunch of other stuff (I was actually surprised by the size of the "human" part of the Wikipedia infobox here: https://en.wikipedia.org/wiki/Template:Sensation_and_perception https://en.wikipedia.org/wiki/Template:Sensation_and_percept...). So it's probably hard to compare to specialized NN models until we learn to better classify and isolate how biological brains learn and encode all the various things they do.