3 ms·
Could someone explain how token generation speed relates to latency for the first token to be outputted? And if anyone have any metrics on latency on a 4090 fo
by jensb1 3y ago
Could someone explain how token generation speed relates to latency for the first token to be outputted?
And if anyone have any metrics on latency on a 4090 for the 70B model, that would be very helpful.
- MezzoDelCammin 3y ago4090Ti? AFAIK that has never been released
- jensb1 3y agotypo
- redox99 3y agoUnrelated. What matters for that is prompt processing time (which is in the high hundreds of tokens per second).