2 ms·
They have cache, but it costs the same indeed, no idea what the point of the cache is
by olivermuty 29d ago
They have cache, but it costs the same indeed, no idea what the point of the cache is
- lostmsu 29d agoThey don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!
- orbifold 28d agoMore precisely they can't cache it.
- lostmsu 28d agoYes. Their architecture recomputes every time so at 150k context every request will have to spend 1.5 min waiting for the model to reread the context. Say avg model response length is 1024 tok. At 50 tok/s normal providers do your turn will only take 20s (vs Cerebras 101s) and will cost 20x less. That time and cost is per single tool call.