3 ms·

> Every token generated would take as long as it would if you were loading the model from scratch. Wouldn't make sense to keep most of the model in GPU (as muc

> Every token generated would take as long as it would if you were loading the model from scratch. Wouldn't make sense to keep most of the model in GPU (as much as it fits) and only load the remaining layers on each pass?

cjbprime 4y ago
(I think the project we're talking about does CPU runtime inference, not GPU.)
- geoah 4y ago
  Yap, only does CPU.