3 ms·
> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. I can understand that
by bilekas 3y ago
> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.
I can understand that but where time is not a factor and solely a question of data, can a model be streamed?
- pests 3y agoLLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit. I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.