4 ms·
You think they are caching? Even though one of the parameters is temperature? Can of worms, and should be reflected in the pricing if true, don't get me started
by PUSH_AX 3y ago
You think they are caching? Even though one of the parameters is temperature? Can of worms, and should be reflected in the pricing if true, don't get me started if they are charging per token for cached responses.
I just don't see it.
- why_only_15 3y agoYou can keep around the KV cache from previous generations which lowers the cost of prompts significantly.