4 ms·
Alternative argument, there is no need for more training data, just better algorithms. Throwing more tokens at the problem doesn't solve the fact that training
by ottah 1y ago
Alternative argument, there is no need for more training data, just better algorithms. Throwing more tokens at the problem doesn't solve the fact that training llms using supervised learning is a poor way to integrate knowledge. We have however seen promising results coming out of reinforcement learning and self play. Which means that anthropic and openais' bet on scale is likely a dead end, but we may yet see capability improvements coming from other labs, without the need for greater data collection.
- rmunn 1y agoBetter algorithms is one of the things I meant by "better ways to tweak the models to make better use of the available training data". But that produces slower growth than the jaw-droppingly rapid growth you can get by slurping pretty much the whole Internet. That produced the sharp part of the S curve, but that part is behind us now, which is why I assert we're approaching the slower-growth part at the top of the curve.