4 ms·
So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?
by m12k 5mo ago
So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?
- fragmede 5mo agoWell, the TPUs they're running on don't have branch prediction, so that had to end up somewhere in the stack.
- c7b 5mo agoThe concept of predicting future elements in a series is not specific to CS. It's older than computers.
- mike_hearn 5mo agoMaybe at very high level of abstraction, but there's no branching involved.
- lossolo 5mo agoWell, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.
- monster_truck 5mo agoWell, not exactly, but that was the dream we were sold (here be dragons)
- kpw94 5mo agoSpeculative execution techniques in software & hardware exist everywhere, - Speculative multi threading - Data Value Speculation - Speculative Memory Disambiguation - Runahead Execution - Speculative Prefetching - Multi-path (Dual-path) Execution (goes beyond branch prediction by computing both paths) - Optimistic Concurrency Control (for database transactions etc)