3 ms·
Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability. The extent to which they are able to
by jiggawatts 2mo ago
Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability.
The extent to which they are able to do this is the more interesting question!
- 5555watch 2mo agoThe extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z. By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).