I'm not sure less data should require full fine-tuning. If I had 5 pages of text, I don't see why I need to train billions of parameters that are already traine

I'm not sure less data should require full fine-tuning. If I had 5 pages of text, I don't see why I need to train billions of parameters that are already trained pretty well on general internet knowledge, and already know how to chat.. From a practical perspective, unless cost is really immaterial, I think most will end up starting with Lora, especially for 13b or 70b models.. you could do 10 fine-tuning runs for the cost of a few full fine-tunings. But it's still all witchcraft to me to some degree, and I'd probably try full and Lora.