3 ms·

"up to 750 tokens per second" Emphasis on "up to". Imagine whatever limited situation (e.g. pre-cached query) and that will probably be the only time it hits 7

by xnx 4mo ago

"up to 750 tokens per second" Emphasis on "up to". Imagine whatever limited situation (e.g. pre-cached query) and that will probably be the only time it hits 750 tokens/sec.