7 ms·
This effect is likely even larger when you consider that the raw cost per inferred token grows linearly with context, rather than being constant. So longer tas
by zozbot234 7mo ago
This effect is likely even larger when you consider that the raw cost per inferred token grows linearly with context, rather than being constant. So longer tasks performed with higher-context models will cost quadratically more. The computational cost also grows super-linearly with model parameter size: a 20B-active model is more than four times the cost of a 5B-active model.
- tibbar 7mo agoDoesn't context cacheing mostly eliminate this problem? (I suppose for enough context the 90% discount is eventually a lot anyway)
- zozbot234 7mo agoContext caching is really storing the KV-cache for reuse. It saves running prefill for that part of the context, but tokens referencing that KV-cache will still cost more.