3 ms·
> Could this "data collection" code not potentially be put into a few thousand lines also? This would mean hardcoding a model of the world. Which maybe, with t
by GTP 2y ago
> Could this "data collection" code not potentially be put into a few thousand lines also?
This would mean hardcoding a model of the world. Which maybe, with the help of some breakthrough from physics, would be possibe (but I think the kind of breakthrough needed to get down to such a small size would be a theory of everything). But this means eliminating the self-learning part of current neural networks, which is what makes them attractive: you don't have to hardcode the rules, as the model will be able to learn them.
- n4r9 2y agoWhat I mean is, we could program an entity which gathers its own data and updates its own weights. With the lines of code being in the tens of thousands.
- GTP 2y agoBut op's point was about people talking about tiny LLMs failing to consider the amount of parameters used. This has nothing to do with the ability of autonomously collecting the training data.