3 ms·
Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)
by colorant 2y ago
Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)
- moffkalast 2y agoDecoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.
- colorant 2y agoWith ~1000 input, the TTFT is ~10 seconds