3 ms·
Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?
by Tepix 6mo ago
Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?
- sacrelege 6mo agoWe typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s. While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel. Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.