4 ms·
1kW under load for 30 tokens/sec and each request is about 100-1000 tokens and each response is about 100-1000 tokens (so a response takes about 3-33s)
by MuffinFlavored 3y ago
1kW under load for 30 tokens/sec and each request is about 100-1000 tokens and each response is about 100-1000 tokens (so a response takes about 3-33s)