4 ms·

~1kW I guess.

• MuffinFlavored 3y ago
  1kW under load for 30 tokens/sec and each request is about 100-1000 tokens and each response is about 100-1000 tokens (so a response takes about 3-33s)