4 ms·
Number of tokens is still a useful metric, as their endpoints have Tokens Per Minute quotas. Decreasing number of tokens used means increasing throughput, up un
by throwaway74432 3y ago
Number of tokens is still a useful metric, as their endpoints have Tokens Per Minute quotas. Decreasing number of tokens used means increasing throughput, up until the Request Per Minute quota.
- afro88 3y agoBut the sentence was "Indeed was able to improve cost..."
- mechagodzilla 3y agoFrom OpenAI’s perspective the cost improved!
- throwaway74432 3y ago"...and latency" Higher throughput = lower latency* *Under certain conditions