2 ms·
What Algorithms Can Transformers Learn? A Study in Length Generalization
- westurner 3y agoDoesn't this also generalize to LLMs (which are mostly all doing just one-next-word prediction): https://news.ycombinator.com/item?id=38830186 https://news.ycombinator.com/item?id=38830186 : >>> Making the prefix shorter tends to produce less coherent prose; making it longer tends to reproduce the input text verbatim. For English text, using two words to select a third is a good compromise; it seems to recreate the flavor of the input while adding its own whimsical touch. But could classical LLMs approximate quantum relations? https://news.ycombinator.com/item?id=39255848 https://news.ycombinator.com/item?id=39255848 : > If there is some sort of e.g. geometric correspondence, it could be possible for a Church-Turing classical computer to compute quantum functions (that return wave functions) that a Church-Turing-Deutsch quantum computer can compute; but otherwise Lean [and all non-quantum LLMs] can't compute most quantum circuits either.