With fine-tuning you are training on a couple hundred thousand tokens, compared to the billions of tokens the original dataset the LLM was trained on. It won't

With fine-tuning you are training on a couple hundred thousand tokens, compared to the billions of tokens the original dataset the LLM was trained on. It won't have as much of an effect. Maybe the original dataset includes a company policy from 5 years ago and now you fine-tune on the new policy. It's hard to guarantee what the model will spit out. By giving the in-context prompt, it's more likely the model will choose the information you just gave it. Of course, there are use cases where fine-tuning is worth it. For example, someone recently finetuned a model on their Messenger group chat that was going on for years, about 500k messages, and now they have a model that can imitate all the members of the chat.