3 ms·
I think LLM technology, not necessarily all of CNN, has plateaued. We've used up all the human discourse, so there's nothing to train it on. It's like fossil f
by BearOso 2y ago
I think LLM technology, not necessarily all of CNN, has plateaued. We've used up all the human discourse, so there's nothing to train it on.
It's like fossil fuels. They took billions of years to create and centuries to consume. We can't just create more.
Another problem is that the data sets are becoming contaminated, creating a reinforcement cycle that makes LLMs trained on more recent data worse.
My thoughts are that it won't get any better with this method of just brute-forcing data into a model like everyone's been doing. There needs to be some significant scientific innovations. But all anybody is doing is throwing money at copying the major players and applying some distinguishing flavor.
- senko 2y agoActually, the sources we had (everything scraped from the internet) turns out to be pretty bad. Imagine not going to school and instead learning everything from random blog posts or reddit comments. You could do it if you read a lot, but it's clearly suboptimal. That's why OpenAI, and probably every other serious AI company, is investing huge amounts in generating (proprietary) datasets.
- chuckledog 2y agoGitHub, especially filtered by starred repos, is a pretty high quality dataset.
- theptip 2y agoWhat data are you using to back up this belief? Progress on benchmarks continues to improve (see GPT-o1). The claim that there is nothing left to train on is objectively false. The big guys are building synthetic training sets, moving to multimodal, and are not worried about running out of data. o1 shows that you can also throw more inference compute at problems to improve performance, so it gives another dimension to scale models on.
- KaiserPro 2y ago> Progress on benchmarks continues to improve (see GPT-o1). thats not evidence of a step change. > The big guys are building synthetic training sets Yes, that helps to pre-train models, but its not a replacement for real data. > not worried about running out of data. they totally are. The more data, the more expensive it is to train. Exponentially more expensive. > o1 shows that you can also throw more inference compute I suspect that its not actually just compute, its changes to training and model design.
- askafriend 2y agoAny thoughts on synthetic data?