3 ms·
I'm curious how the predictive aspect of LLMs can generate / "solve" equations. Is it purely "these inputs are most likely followed by this output", and thus ne
by tqi 4y ago
I'm curious how the predictive aspect of LLMs can generate / "solve" equations. Is it purely "these inputs are most likely followed by this output", and thus needs to have actually seen the problem to get it right, or is it able to infer some of the rules underlying the operations?
- famouswaffles 4y agoPrediction is only the goal. The whole point of training is to learn what it takes to predict well and accurately. and that necessitates understanding and whatever else an LLM learns from the data.
- sebzim4500 4y agoClearly it doesn't need to have seen the exact input before because you can ask GPT-4 to add together two 6 digit numbers and it will get it right. More impressively, if you generate a random small neural network and then put some samples in the prompt it can do a surprisingly good job of predicting the result of putting other inputs into the network. Presumably doing something that looks a bit like gradient decent, see https://arxiv.org/pdf/2212.07677.pdf https://arxiv.org/pdf/2212.07677.pdf Solely in order to predict the next token, LLMs are learning an incredibly sophisticed model of both computation and the outside world.