I'm surprised most of that isn't cached token usage. It's true that increasing length is a problem on its own because the model needs to attend to it all, but w

I'm surprised most of that isn't cached token usage. It's true that increasing length is a problem on its own because the model needs to attend to it all, but with caching it should be pretty fast anyway. My system prompt is quite large and I haven't noticed much of a generation penalty in the range from 5k to 10k.