4 ms·
Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating
by gpugreg 29d ago
Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.
- perching_aix 29d agothen it's basically useless lol, wtf, this has to be a defect
- selcuka 29d agoIt's PR: https://news.ycombinator.com/item?id=49556302 https://news.ycombinator.com/item?id=49556302
- nullbio 29d agoCached tokens counting toward the limit is ridiculous.