3 ms·
I assume they use a conversation, so if you compress the prompt immediately you should only break cache once, and still hit cache on subsequent prompts? So ins
by irthomasthomas 1y ago
I assume they use a conversation, so if you compress the prompt immediately you should only break cache once, and still hit cache on subsequent prompts?
So instead of
Write
Hit
Hit
Hit
It's
Write
Write
Hit
Hit
Hit