3 ms·
And a "prune here" button. It often happens that the interesting information is in the first paragraph or so, and the remainder is all just the LLM not knowing
by amelius 7mo ago
And a "prune here" button.
It often happens that the interesting information is in the first paragraph or so, and the remainder is all just the LLM not knowing when to stop. This is super annoying as a conversation then ends up being 90% noise.
- postalcoder 7mo agoPruning an assistant's response like that would break prompt caching. Prompt caching is probably the single most important thing that people building harnesses think about and yet it's mind share in end users is virtually zero. If you had to think of all the weirdest, most seemingly baffling design decisions in an AI product, the answer to "why" is probably "to not break prompt caching".
- amelius 7mo agoMaybe so, but pruning is still a useful feature. If it hurts performance that much, maybe pruning could just hide the text leaving the cache intact?
- zozbot234 7mo agoGrug says prompt caching just store KV-cache which is sequenced by token. Easy cut it back to just before edit. Then regenerate after is just like prefill but tiny.