I agree that fine-tuning isn't good at imparting knowledge, though in my experience, K-shot prompting is very close to as good as fine-tuning at getting output

I agree that fine-tuning isn't good at imparting knowledge, though in my experience, K-shot prompting is very close to as good as fine-tuning at getting output formats (for recent models, at least) So except for cases where you are really trying to perfectly match a particular writing style and want to fine-tune on loads of some author's text, I think prompt caching mostly dominates fine-tuning for most real-world workloads. I'd still fine-tune if I wanted my model to have a really particular "voice" though.