4 ms·
You need all of the model in RAM to perform the matmult that gets you the next token from it. There's no shortcut.
by cjbprime 3y ago
You need all of the model in RAM to perform the matmult that gets you the next token from it. There's no shortcut.