4 ms·
Yeah but it gets the whole prompt every time, right? So the whatever state it could compute for the first forward pass, it could also compute on the subsequent
by heisenzombie 3y ago
Yeah but it gets the whole prompt every time, right? So the whatever state it could compute for the first forward pass, it could also compute on the subsequent one. Hence, does it matter that it can’t store the state, if it can recompute it?
- ukuina 3y agoExactly. It's inefficient for LLMs to recompute state based on previous output, but that's what we have so far, and it works, surprisingly.