3 ms·
Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. For reference I have a 3
by janalsncm 3y ago
Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.
For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.
- bilekas 3y ago> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. I can understand that but where time is not a factor and solely a question of data, can a model be streamed?
- pests 3y agoLLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit. I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.