3 ms·

I'm fascinated by the parts of the article suggesting LLMs will face diminishing returns in terms of getting larger. Once you've memorized the entire Internet,

I'm fascinated by the parts of the article suggesting LLMs will face diminishing returns in terms of getting larger. Once you've memorized the entire Internet, you can't really continue to increase the model size and get any benefits, right?

throwaway4aday 4y ago
They haven't memorized the entire internet let alone the entire world, a few new avenues: 1. Use recorded movies and TV, buy the rights to transcribe (or get access to the scripts) and pair that with still frames from the video 2. Do the same for YouTube videos, harder to get the rights since you're dealing with Google and all the individual creators but bigger catalogue by far 3. If they haven't already, use the contents of Library Genesis or Google Books or just set up your own scanning project 4. Scoop up any scientific papers that they haven't ingested yet 5. Transcribe podcasts, there are millions of them now and many that publish daily content 6. Pay people to wear recording devices, this one's a bit far fetched but I'm sure it's doable with the right incentives and by choosing regions where laws allow for this 7. Non-English sources, basically applicable to everything above and maybe to the sources they've already used if they had excluded them before.
- jxy 4y ago
  Many car insurances in the US are giving people incentives to install a tracking app in people's phones. I won't be surprised when they start handing out dash cams with audio recordings.
  - ecommerceguy 4y ago
    Not sure what made me agree to it, off day i suppose, but i had a dongle attached to the diagnostic port for a year for car insurance. didnt decrease premium so i sent it back. but with a phone app, how would they know what car you are in?
    - thenipper 4y ago
      Worked in this industry for a bit. Lots of ways to tell. Like where did the drive start? Did it start at the same address as your residence? Also most likely they care more about you as a driver then you as a driver of your car. At least in terms of rating your ability.
      - Jarwain 4y ago
        For this example they'd need info from the car itself. Otherwise how would they distinguish between me leaving in my car vs being picked up by a friend vs catching an uber
- toomuchtodo 4y ago
  Internet Archive would be a useful corpus to ingest continuously.
- jayd16 4y ago
  Is this sarcasm because Google does all of these already?
- tetris11 4y ago
  I'm waiting for an AI to use https://frinkiac.com/ https://frinkiac.com/ Full frame-by-frame Simpsons transcripts with accompanying images. This is a treasure trove of training material for writing Simpsons episodes.
FrustratedMonky 4y ago
The way I took it, was maybe LLM's don't need to get much larger, but that doesn't mean they aren't going keep improving it and maybe expanding with adding on other architectures. -- Maybe he was just trying to tamp down the hype since it is doubtful they could make something as instantly ground braking as GPT very soon.
comfypotato 4y ago
The internet is just the training data. And there’s plenty to be done to improve the quality of the data. The size of the model determines how well it performs. A bigger model would perform better. We won’t be seeing much bigger models in the immediate future because the 1T models are a good balance of cost and performance.
autokad 4y ago
its not really memorizing the internet. its passing the internet through the network. imagine you are in class for an hour, and the professor was boring - speaking in a monotone voice. did you really memorize everything they said? Now imagine the professor is extremely engaging, maybe you learned more, but you still haven't memorized what everything they said. so neural network architechers are going to evolve and get more complicated or even just more efficient at learning. a standard NN was supposed to be able to memorize even time series data, but a memory component was added and LSTMs did a lot better at that task. so for now, we can keep adding model size and topology to these NN and still get huge value add. gpt4 is a proof if that. lastly, its not just about throwing data at it. openai spent a lot of time giving the model feedback.
kypro 4y ago
Don't think in terms of memorisation, but compression. These models are trained to compress knowledge and then to leverage it to produce useful text. They're not remembering it, or storing it in a traditional sense that computers might. It's similar to the loose way humans remember little facts and things that we've seen or heard a lot. For example, you can ask ChatGPT about obscure Wikipedia content – obscure things that it's almost certainly read. You'll find it doesn't have any recollection of the information. Larger models should enable LLMs to draw on more the information they're trained on, and in more depth.

throwaway4aday 4y ago

They haven't memorized the entire internet let alone the entire world, a few new avenues: 1. Use recorded movies and TV, buy the rights to transcribe (or get access to the scripts) and pair that with still frames from the video 2. Do the same for YouTube videos, harder to get the rights since you're dealing with Google and all the individual creators but bigger catalogue by far 3. If they haven't already, use the contents of Library Genesis or Google Books or just set up your own scanning project 4. Scoop up any scientific papers that they haven't ingested yet 5. Transcribe podcasts, there are millions of them now and many that publish daily content 6. Pay people to wear recording devices, this one's a bit far fetched but I'm sure it's doable with the right incentives and by choosing regions where laws allow for this 7. Non-English sources, basically applicable to everything above and maybe to the sources they've already used if they had excluded them before.

jxy 4y ago
Many car insurances in the US are giving people incentives to install a tracking app in people's phones. I won't be surprised when they start handing out dash cams with audio recordings.
- ecommerceguy 4y ago
  Not sure what made me agree to it, off day i suppose, but i had a dongle attached to the diagnostic port for a year for car insurance. didnt decrease premium so i sent it back. but with a phone app, how would they know what car you are in?
  - thenipper 4y ago
    Worked in this industry for a bit. Lots of ways to tell. Like where did the drive start? Did it start at the same address as your residence? Also most likely they care more about you as a driver then you as a driver of your car. At least in terms of rating your ability.
    - Jarwain 4y ago
      For this example they'd need info from the car itself. Otherwise how would they distinguish between me leaving in my car vs being picked up by a friend vs catching an uber
toomuchtodo 4y ago
Internet Archive would be a useful corpus to ingest continuously.
jayd16 4y ago
Is this sarcasm because Google does all of these already?
tetris11 4y ago
I'm waiting for an AI to use https://frinkiac.com/ https://frinkiac.com/ Full frame-by-frame Simpsons transcripts with accompanying images. This is a treasure trove of training material for writing Simpsons episodes.

FrustratedMonky 4y ago

The way I took it, was maybe LLM's don't need to get much larger, but that doesn't mean they aren't going keep improving it and maybe expanding with adding on other architectures. -- Maybe he was just trying to tamp down the hype since it is doubtful they could make something as instantly ground braking as GPT very soon.

comfypotato 4y ago

The internet is just the training data. And there’s plenty to be done to improve the quality of the data. The size of the model determines how well it performs. A bigger model would perform better. We won’t be seeing much bigger models in the immediate future because the 1T models are a good balance of cost and performance.

autokad 4y ago

its not really memorizing the internet. its passing the internet through the network. imagine you are in class for an hour, and the professor was boring - speaking in a monotone voice. did you really memorize everything they said? Now imagine the professor is extremely engaging, maybe you learned more, but you still haven't memorized what everything they said. so neural network architechers are going to evolve and get more complicated or even just more efficient at learning. a standard NN was supposed to be able to memorize even time series data, but a memory component was added and LSTMs did a lot better at that task. so for now, we can keep adding model size and topology to these NN and still get huge value add. gpt4 is a proof if that. lastly, its not just about throwing data at it. openai spent a lot of time giving the model feedback.

kypro 4y ago

Don't think in terms of memorisation, but compression. These models are trained to compress knowledge and then to leverage it to produce useful text. They're not remembering it, or storing it in a traditional sense that computers might. It's similar to the loose way humans remember little facts and things that we've seen or heard a lot. For example, you can ask ChatGPT about obscure Wikipedia content – obscure things that it's almost certainly read. You'll find it doesn't have any recollection of the information. Larger models should enable LLMs to draw on more the information they're trained on, and in more depth.