3 ms·
For all of these "how can we batch predicting the next n tokens?" the intuition is basically that it takes a buttload of math to predict some of the tokens, but
by TomatoCo 2y ago
For all of these "how can we batch predicting the next n tokens?" the intuition is basically that it takes a buttload of math to predict some of the tokens, but that most tokens are actually easy to guess. For example, if I asked "What was that phone number from that 80's song?" as soon as a model generates 867- it shouldn't take that much math at all to finish predicting 5309.
- snyhlxde 2y agoA bit more intuition on how training works: in natural language processing, some phrases/collocations, for example "remind ... of ...", "make a decision", "learn a skill" etc. are used together. We can ask LLMs to learn such collections & frequently appearing n-grams. After learning, the model can use parallel decoding to predict many tokens that are frequently appear together in one forward pass.