4 ms·
Handle the state on the application side... It is like complaining that HTTP is limiting because it is stateless. Build state on top of it.
by weird-eye-issue 3y ago
Handle the state on the application side...
It is like complaining that HTTP is limiting because it is stateless. Build state on top of it.
- delusional 3y agoI think he's talking about computational efficiency. If you're loading in 29k tokens and you're expecting to use those again, you wouldn't need to do the whole matrix multiplication song and dance again if you just kept the old buffers around for the next prompt.
- weird-eye-issue 3y agoI don't think this can necessarily be optimized at least with how the models work right now