3 ms·
How do you count tokens per second?
by atharv_jaju 3y ago
How do you count tokens per second?
- csjh 3y agoYou measure how many tokens are output in a period of time and take the average?
- atharv_jaju 3y agoOh my bad, I meant how to code it?
- csjh 3y agoPython pseudocode: ```py output = prompt("My favourite animal is ") # returns a generator start = time.time() tokens = 0 for token in output: tokens += 1 print(token) print(f"Outputted {tokens / (time.time() - start)} tokens/second") ```