4 ms·

I'll bite. How do i train/make and/or use LoRA, or, separately, how do i fine-tune? I've been asking this for months, and no one has a decent answer. websearch

I'll bite. How do i train/make and/or use LoRA, or, separately, how do i fine-tune? I've been asking this for months, and no one has a decent answer. websearch on my end is seo/geo-spam, with no real instructions. I know how to make an SD LoRA, and use it. I've known how to do that for 2 years. So what's the big secret about LLM LoRA?

minimaxir 1y ago
If you're using Hugging Face transformers, the library you want to use is peft: https://huggingface.co/docs/peft/en/quicktour https://huggingface.co/docs/peft/en/quicktour There are Colab Notebook tutorials around training models with it as well.
notpublic 1y ago
https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth I'm not sure if it contains exactly what you're looking for, but it includes several resources and notebooks related to fine-tuning LLMs (including LoRA) that I found useful.
techwizrd 1y ago
We have been fine-tuning models using Axolotl and Unsloth, with a slight preference for Axolotl. Check out the docs [0] and fine-tune or quantize your first model. There is a lot to be learned in this space, but it's exciting. 0: https://axolotl.ai/ https://axolotl.ai/ and https://docs.axolotl.ai/ https://docs.axolotl.ai/
- syntaxing 1y ago
  What hardware do you train on using axolotl? I use unsloth with Google colab pro
- arkmm 1y ago
  When do you think fine tuning is worth it over prompt engineering a base model? I imagine with the finetunes you have to worry about self-hosting, model utilization, and then also retraining the model as new base models come out. I'm curious under what circumstances you've found that the benefits outweigh the downsides.
  - whimsicalism 1y ago
    finetuning rarely makes sense unless you are an enterprise and even generally doesn't in most cases there either.
  - tough 1y ago
    only for narrow applications where your fine tune can let you use a smaller model locally , specialised and trained for your specific use-case mostly
  - reissbaker 1y ago
    For self-hosting, there are a few companies that offer per-token pricing for LoRA finetunes (LoRAs are basically efficient-to-train, efficient-to-host finetunes) of certain base models: - (shameless plug) My company, Synthetic, supports LoRAs for Llama 3.1 8b and 70b: https://synthetic.new https://synthetic.new All you need to do is give us the Hugging Face repo and we take care of the rest. If you want other people to try your model, we charge usage to them rather than to you. (We can also host full finetunes of anything vLLM supports, although we charge by GPU-minute for full finetunes rather than the cheaper per-token pricing for supported base model LoRAs.) - Together.ai supports a slightly wider number of base models than we do, with a bit more config required, and any usage is charged to you. - Fireworks does the same as Together, although they quantize the models more heavily (FP4 for the higher-end models). However, they support Llama 4, which is pretty nice although fairly resource-intensive to train. If you have reasonably good data for your task, and your task is relatively "narrow" (i.e. find a specific kind of bug, rather than general-purpose coding; extract a specific kind of data from legal documents rather than general-purpose reasoning about social and legal matters; etc), finetunes of even a very small model like an 8b will typically outperform — by a pretty wide margin — even very large SOTA models while being a lot cheaper to run. For example, if you find yourself hand-coding heuristics to fix some problem you're seeing with an LLM's responses, it's probably more robust to just train a small model finetune on the data and have the finetuned model fix the issues rather than writing hardcoded heuristics. On the other hand, no amount of finetuning will make an 8b model a better general-purpose coding agent than Claude 4 Sonnet.
    - 1y ago
svachalek 1y ago
For completeness, for Apple hardware MLX is the way to go.
- w10-1 1y ago
  MLX github: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx get started: https://developer.apple.com/videos/play/wwdc2025/315/ https://developer.apple.com/videos/play/wwdc2025/315/ details: https://developer.apple.com/videos/play/wwdc2025/298/ https://developer.apple.com/videos/play/wwdc2025/298/
qcnguy 1y ago
LLM fine tuning tends to destroy the model's capabilities if you aren't very careful. It's not as easy or effective as with image generation.
- israrkhan 1y ago
  do you have a suggestion or a way to measure if model capabilities are getting destroyed? how do one measure it objectively?
  - RALaBarge 1y ago
    Ask it a series of the same questions after you train that you posed before training started. Is the quality lower?
    - israrkhan 1y ago
      That series of questions will measure only a particular area. I am concerned about destorying model capabilities in some other area that that I do not pay attention to, and have no way of knowing.
      - simonh 1y ago
        Isn’t that a general problem with LLMs? The only way to know how good it is at something is to test it.
  - mensetmanusman 1y ago
    These are now questions at the cutting edge of academic research. It might be computationally unknowable until checked.
- nxobject 1y ago
  My very cursory understanding -- at least from Unsloth's recommendations -- is that you have to work very hard to preserve reasoning/instruct capabilities [1]: for example to "preserve" Qwen3's reasoning capabilities (however that's operationalized), they suggest a fine-tuning corpus that's 75% chain of thought to 25% non-reasoning. Is that a significant issue for orgs/projects that currently rely on fine-tuning? [1] https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#notebook-guide https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tun...
electroglyph 1y ago
unsloth is the easiest way to finetune due to the lower memory requirements
pdntspa 1y ago
Have you tried asking an LLM?
jasonjmcghee 1y ago
brev.dev made an easy to follow guide a while ago but apparently Nvidia took it down or something when they bought them? So here's the original https://web.archive.org/web/20231127123701/https://brev.dev/blog/fine-tuning-mistral https://web.archive.org/web/20231127123701/https://brev.dev/...
otabdeveloper4 1y ago
> So what's the big secret about LLM LoRA? No clear use case for LLMs yet. ("Spicy" aka pornography finetunes are the only ones with broad adoption, but we don't talk about that in polite society here.)
- AlecSchueler 1y ago
  Where do we speak about it? It feels like the biggest use for these models right now is for deep fakes and other harassment but few people in the industry want to talk about it while continuing to enable it.

minimaxir 1y ago

If you're using Hugging Face transformers, the library you want to use is peft: https://huggingface.co/docs/peft/en/quicktour https://huggingface.co/docs/peft/en/quicktour There are Colab Notebook tutorials around training models with it as well.

notpublic 1y ago

https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth I'm not sure if it contains exactly what you're looking for, but it includes several resources and notebooks related to fine-tuning LLMs (including LoRA) that I found useful.

techwizrd 1y ago

We have been fine-tuning models using Axolotl and Unsloth, with a slight preference for Axolotl. Check out the docs [0] and fine-tune or quantize your first model. There is a lot to be learned in this space, but it's exciting. 0: https://axolotl.ai/ https://axolotl.ai/ and https://docs.axolotl.ai/ https://docs.axolotl.ai/

syntaxing 1y ago
What hardware do you train on using axolotl? I use unsloth with Google colab pro
arkmm 1y ago
When do you think fine tuning is worth it over prompt engineering a base model? I imagine with the finetunes you have to worry about self-hosting, model utilization, and then also retraining the model as new base models come out. I'm curious under what circumstances you've found that the benefits outweigh the downsides.
- whimsicalism 1y ago
  finetuning rarely makes sense unless you are an enterprise and even generally doesn't in most cases there either.
- tough 1y ago
  only for narrow applications where your fine tune can let you use a smaller model locally , specialised and trained for your specific use-case mostly
- reissbaker 1y ago
  For self-hosting, there are a few companies that offer per-token pricing for LoRA finetunes (LoRAs are basically efficient-to-train, efficient-to-host finetunes) of certain base models: - (shameless plug) My company, Synthetic, supports LoRAs for Llama 3.1 8b and 70b: https://synthetic.new https://synthetic.new All you need to do is give us the Hugging Face repo and we take care of the rest. If you want other people to try your model, we charge usage to them rather than to you. (We can also host full finetunes of anything vLLM supports, although we charge by GPU-minute for full finetunes rather than the cheaper per-token pricing for supported base model LoRAs.) - Together.ai supports a slightly wider number of base models than we do, with a bit more config required, and any usage is charged to you. - Fireworks does the same as Together, although they quantize the models more heavily (FP4 for the higher-end models). However, they support Llama 4, which is pretty nice although fairly resource-intensive to train. If you have reasonably good data for your task, and your task is relatively "narrow" (i.e. find a specific kind of bug, rather than general-purpose coding; extract a specific kind of data from legal documents rather than general-purpose reasoning about social and legal matters; etc), finetunes of even a very small model like an 8b will typically outperform — by a pretty wide margin — even very large SOTA models while being a lot cheaper to run. For example, if you find yourself hand-coding heuristics to fix some problem you're seeing with an LLM's responses, it's probably more robust to just train a small model finetune on the data and have the finetuned model fix the issues rather than writing hardcoded heuristics. On the other hand, no amount of finetuning will make an 8b model a better general-purpose coding agent than Claude 4 Sonnet.
  - 1y ago

svachalek 1y ago

For completeness, for Apple hardware MLX is the way to go.

w10-1 1y ago
MLX github: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx get started: https://developer.apple.com/videos/play/wwdc2025/315/ https://developer.apple.com/videos/play/wwdc2025/315/ details: https://developer.apple.com/videos/play/wwdc2025/298/ https://developer.apple.com/videos/play/wwdc2025/298/

qcnguy 1y ago

LLM fine tuning tends to destroy the model's capabilities if you aren't very careful. It's not as easy or effective as with image generation.

israrkhan 1y ago
do you have a suggestion or a way to measure if model capabilities are getting destroyed? how do one measure it objectively?
- RALaBarge 1y ago
  Ask it a series of the same questions after you train that you posed before training started. Is the quality lower?
  - israrkhan 1y ago
    That series of questions will measure only a particular area. I am concerned about destorying model capabilities in some other area that that I do not pay attention to, and have no way of knowing.
    - simonh 1y ago
      Isn’t that a general problem with LLMs? The only way to know how good it is at something is to test it.
- mensetmanusman 1y ago
  These are now questions at the cutting edge of academic research. It might be computationally unknowable until checked.
nxobject 1y ago
My very cursory understanding -- at least from Unsloth's recommendations -- is that you have to work very hard to preserve reasoning/instruct capabilities [1]: for example to "preserve" Qwen3's reasoning capabilities (however that's operationalized), they suggest a fine-tuning corpus that's 75% chain of thought to 25% non-reasoning. Is that a significant issue for orgs/projects that currently rely on fine-tuning? [1] https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#notebook-guide https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tun...

electroglyph 1y ago

unsloth is the easiest way to finetune due to the lower memory requirements

pdntspa 1y ago

Have you tried asking an LLM?

jasonjmcghee 1y ago

brev.dev made an easy to follow guide a while ago but apparently Nvidia took it down or something when they bought them? So here's the original https://web.archive.org/web/20231127123701/https://brev.dev/blog/fine-tuning-mistral https://web.archive.org/web/20231127123701/https://brev.dev/...

otabdeveloper4 1y ago

> So what's the big secret about LLM LoRA? No clear use case for LLMs yet. ("Spicy" aka pornography finetunes are the only ones with broad adoption, but we don't talk about that in polite society here.)

AlecSchueler 1y ago
Where do we speak about it? It feels like the biggest use for these models right now is for deep fakes and other harassment but few people in the industry want to talk about it while continuing to enable it.