4 ms·
I think he's talking about computational efficiency. If you're loading in 29k tokens and you're expecting to use those again, you wouldn't need to do the whole
by delusional 4y ago
I think he's talking about computational efficiency. If you're loading in 29k tokens and you're expecting to use those again, you wouldn't need to do the whole matrix multiplication song and dance again if you just kept the old buffers around for the next prompt.
- weird-eye-issue 4y agoI don't think this can necessarily be optimized at least with how the models work right now