3 ms·
I am guessing it might not be the main transformer that is occasionally slower, but rather a secondary LLM that is used to censor stuffs which occasionally runs
by NhanH 3y ago
I am guessing it might not be the main transformer that is occasionally slower, but rather a secondary LLM that is used to censor stuffs which occasionally runs ?
So basically the unusual answer needs more additional processing to make sure it is "safe" to be streamed to the client.