3 ms·
There is no training data left. Every refinement in AI from here on will come from architectural changes. All models basically have reached a local maximum on n
by rpmisms 1y ago
There is no training data left. Every refinement in AI from here on will come from architectural changes. All models basically have reached a local maximum on new information.
- blibble 1y agoyeah, I said this on this site two years ago there's no second internet of high quality content to plagarise and the valuable information on the existing one is starting to be locked down pretty hard
- vajrabum 1y agoAnd even if it's not locked down hard how do you separate the signal from the noise with all the ai generated blah blah blah.
- p1esk 1y agoAre you saying they have already trained gpt-5 on the entirety of world’s video data?
- blackqueeriroh 1y agoStudies show relatively conclusively that using primarily synthetic data woven intentionally with seeded real-world data is an effective strategy for training frontier LLMs: https://consensus.app/search/synthetic-data-effectiveness-frontier/hNdPhq9FSEy9lvpsG82oyw https://consensus.app/search/synthetic-data-effectiveness-fr...
- BriggyDwiggs42 1y agoYou can only take that so far before model collapse. Even if it were equivalent to multiplying currently available data by four, would that be enough? Might still look like an incremental improvement.
- booleandilemma 1y agoIsn't new training data being created every day? YouTube, Facebook, Tiktok, etc.? Humans are content creation machines.
- BlueTemplar 1y agoNo training data for LLMs -- in the text form. Meanwhile the fraction of text that the real world consists of is microscopic.