Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting t

Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting to pinpointing the exact locations of the knowledge and updating it freely. It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning. 1. https://arxiv.org/abs/2210.07229 https://arxiv.org/abs/2210.07229