3 ms·
This is probably a really stupid question, but can the models be streamed as they're being ran? So that you as the browser wouldn't need to wait for the entire
by bilekas 3y ago
This is probably a really stupid question, but can the models be streamed as they're being ran? So that you as the browser wouldn't need to wait for the entire download first? Or there is even the concept of ordered model transformers?
As I ask it seems wrong to me but just to confirm?
- janalsncm 3y agoUsually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.
- bilekas 3y ago> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time. I can understand that but where time is not a factor and solely a question of data, can a model be streamed?
- pests 3y agoLLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit. I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.