4 ms·
The pay-per-use API sucks. If you end up on the $50/mo plan, it's better, with caveats: 1 million tokens per minute, 24 million tokens per day. BUT: cached t
by mlyle 9mo ago
The pay-per-use API sucks. If you end up on the $50/mo plan, it's better, with caveats:
1 million tokens per minute, 24 million tokens per day. BUT: cached tokens count full, so if you have 100,000 tokens of context you can burn a minute of tokens in a few requests.
- solarkraft 9mo agoIt’s wild that cached tokens count full - what’s in it for you to care about caching at all then? Is the processing speed gain significant?
- mlyle 9mo agoNot really worth it, in general. It does reduce latency a little. In practice, you do have a continuing context, though, so you end up using it whether you care or not.
- indigodaddy 9mo agoTry a nano-gpt subscription. Not going to be as fast as cerebras obviously but it's $8/mo for 60,000 requests