3 ms·
I think we're not close to running out of training data. It's just that we would like knowledge, but not necessary behavior of said texts. LLMs are very bad at
by RicoElectrico 2y ago
I think we're not close to running out of training data. It's just that we would like knowledge, but not necessary behavior of said texts. LLMs are very bad at recalling popular memes (known by any seasoned netizen) if they had no press coverage. Maybe training with 4chan isn't as pointless if you could make it memorize it, but not imitate it.
Also, what about movie scripts and song lyrics? Transcripts of well known YouTube videos? Hell, television programs even.
- stavros 2y agoWe've run out of training data that definitely did not contain LLM outputs.
- DAGdug 2y agoWhat about non-text modalities - image and video, specifically?
- riffraff 2y agovideo is probably still fine, but images sourced from the internet now contain a massive amount of AI slop. It seems, for example, that many newsletters, blogs etc resort to using AI-generated images to give some color to their writings (which is something I too intended to do, before realizing how annoyed I am by it)
- gcollard- 2y agoAll the publicly accessible sources you mentioned have already been scraped or licensed to avoid legal issues. This is why it’s often said, “there’s no public data left to train on.” For evidence of this, consider observing non-English-speaking young children (ages 2–6) using ChatGPT’s voice mode. The multimodal model frequently interprets a significant portion of their speech as “thank you for watching my video,” reflecting child-like patterns learned from YouTube videos.