3 ms·

It seems to me you could generate a lot of fresh information from running every youtube video, every hour of TV on archive.org, every movie on the pirate bay --

It seems to me you could generate a lot of fresh information from running every youtube video, every hour of TV on archive.org, every movie on the pirate bay -- do scene by scene image captioning + high quality whisper transcriptions (not whatever junk auto-transcription YouTube has applied), and use that to produce screenplays of everything anyone has ever seen. I'm not sure why I've never heard of this being done, it would be a good use of GPUs in between training runs.

airstrike 2y ago
Don't forget every hour of news broadcasting, of which we likely won't run out any time soon. Plus high quality radio
miltonlost 2y ago
> a lot of fresh information from running every youtube video EVERY youtube video?? Even the 9/11 truther videos? Sandy Hook conspiracy videos? Flat earth? Even the blatantly racist? This would be some bad training data without some pruning.
- lanstin 2y ago
  The best videos would be those where you accidentally start recording and you get 2 hours of naturalistic conversation between real people in reality. Not sure how often they are uploaded to YouTube. Part of the reason that kids need less material is that the aren't just listening, they are also able to do experiments to see what works and what doesn't.
jensvdh 2y ago
The fact that OpenAI can just scrape all of Youtube and Google isn't even taking legal action or attempting to stop it is wild to me. Is Google just asleep?
- bdangubic 2y ago
  what are they going to use to sue - DMCA? OpenAI (and others) are scraping everything imaginable (MS is scraping private Github repos…) - don’t think anyone in the current government will be regulating any of this anytime soon
  - lanstin 2y ago
    Such a biased source of data-that gets them all the LaTeX source for my homeworks, but not my professor's grading of the homework, and not the invaluable words I get from my professor at office hours. No wonder the LLMs have bizarre blindnesses in different directions.
    - bdangubic 2y ago
      Such a biased source of data-that gets them all the LaTeX source for my homeworks but also myriad of hardcore private repositories of many high-tech US enterprises hacking amazing shit (mine included) :)
      - lanstin 2y ago
        One reason LLMs are better at commercial coding than maths.
ilaksh 2y ago
I think that this is the obvious path to more robust models -- grounding language on video.

airstrike 2y ago

Don't forget every hour of news broadcasting, of which we likely won't run out any time soon. Plus high quality radio

miltonlost 2y ago

> a lot of fresh information from running every youtube video EVERY youtube video?? Even the 9/11 truther videos? Sandy Hook conspiracy videos? Flat earth? Even the blatantly racist? This would be some bad training data without some pruning.

lanstin 2y ago
The best videos would be those where you accidentally start recording and you get 2 hours of naturalistic conversation between real people in reality. Not sure how often they are uploaded to YouTube. Part of the reason that kids need less material is that the aren't just listening, they are also able to do experiments to see what works and what doesn't.

jensvdh 2y ago

The fact that OpenAI can just scrape all of Youtube and Google isn't even taking legal action or attempting to stop it is wild to me. Is Google just asleep?

bdangubic 2y ago
what are they going to use to sue - DMCA? OpenAI (and others) are scraping everything imaginable (MS is scraping private Github repos…) - don’t think anyone in the current government will be regulating any of this anytime soon
- lanstin 2y ago
  Such a biased source of data-that gets them all the LaTeX source for my homeworks, but not my professor's grading of the homework, and not the invaluable words I get from my professor at office hours. No wonder the LLMs have bizarre blindnesses in different directions.
  - bdangubic 2y ago
    Such a biased source of data-that gets them all the LaTeX source for my homeworks but also myriad of hardcore private repositories of many high-tech US enterprises hacking amazing shit (mine included) :)
    - lanstin 2y ago
      One reason LLMs are better at commercial coding than maths.

ilaksh 2y ago

I think that this is the obvious path to more robust models -- grounding language on video.