3 ms·
> …but, when you’ve got a massive context windows like the GPT 35k, who cares? AIUI, prompt size still impacts the inference cost (the compute resources, even
by dragonwriter 4y ago
> …but, when you’ve got a massive context windows like the GPT 35k, who cares?
AIUI, prompt size still impacts the inference cost (the compute resources, even if you are the first party so you aren’t paying retail API pricing), and while the “you won’t have room left for work in your context window” problem is not as bad with the bigger long-window models, the inference cost per token is higher for those models, so one way or another its a factor.