2 ms·
> Statistically predicting the next item means I am limited to the training set Not at all. Extrapolation is perfectly possible in a purely predictive model. I
by jameshart 3y ago
> Statistically predicting the next item means I am limited to the training set
Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at.
In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.
- usrbinbash 3y ago> with LLMs where it is able to declare a novel function and then use it. Novel as in "implements a new algorithm that has never seen in any form and is actually an improvement over existing methodology"? Here is a little thought experiment: If all the training data in the set says that manned, powered flight is impossible, is a statistical prediction engine trained on that data capable of developing an airplane? In other words, could a very powerful stochastic parrot do what the Wright Brothers did? My best guess is that no, it cannot.