3 ms·
There is another problem. If you have to use a second model to check the safety of the first model’s output, then you have to wait for the first model to finish
by ttul 3y ago
There is another problem. If you have to use a second model to check the safety of the first model’s output, then you have to wait for the first model to finish generating before the second model can do its work. Uber-scale LLMs are slow at generation, so most UIs stream the output a token at a time to give the user a sense of progress and the experience of being “spoken to”.
Halting output while the entire thing generates will not be usable for real-time chat-like interfaces.
- quickthrower2 3y agoYou need some kind of verbal spinner.