3 ms·
Most of the computational cost of producing an output token is spent on consuming input tokens (including previous output tokens that are fed back in); only the
by yorwba 4y ago
Most of the computational cost of producing an output token is spent on consuming input tokens (including previous output tokens that are fed back in); only the final unembedding matrix could be eliminated if you don't care about the output logits for the context.
So it's not correct to only modify OpenAI's prices to account for the ratio of context tokens to output tokens. Both of them get multiplied by 20 (if that's what your ratio is).
- ImprobableTruth 4y agoNo, because they're already taking that into account. >Metric: generation throughput (token/s) = number of the generated tokens / (time for processing prompts + time for generation). (Though they're doing batching, so this is an unfair comparison. Would be interesting to get single batch speed.)