3 ms·
Decoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.
by moffkalast 2y ago
Decoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.
- colorant 2y agoWith ~1000 input, the TTFT is ~10 seconds