3 ms·
I think it's processing the tokenized prompt at 378 tokens per second, but emitting predicted tokens by the model at 11 tokens per second.
by PeterStuer 3y ago
I think it's processing the tokenized prompt at 378 tokens per second, but emitting predicted tokens by the model at 11 tokens per second.