LLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limi

LLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit. I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.