2 ms·
Cloud models will always have massive benefits of scale. Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hostin
by kristofferR 21d ago
Cloud models will always have massive benefits of scale.
Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.
- nixon_why69 21d agoKV cache is per conversation, I'm getting 100% hit rate on my single tenant local set up. The benefits of scale are on the token generation side, you can batch rounds and generate tokens for multiple conversations per pass instead of just one token per pass.