3 ms·

I think the biggest case for fine tuning is probably that you can take small models, fine tune them for applications that require structured output, and then ru

I think the biggest case for fine tuning is probably that you can take small models, fine tune them for applications that require structured output, and then run cheap inference at scale. "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run.

throwaway6977 7mo ago
I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. It needs to be as tiny as possible so it doesn't take resources away from the running game.
- yw3410 7mo ago
  How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory?
  - lelanthran 7mo ago
    > How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory? I dunno, for game prose I expect that a tiny highly quantized model would be sufficient (generating no more than a paragraph), so 300MB - 500MB maybe? Running on CPU not GPU is feasible too, I think.
- lelanthran 7mo ago
  > I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. Unless your game states have combinatoral exlosion, would it not be better to generate all of that pre-build? If templated you can generate a few hundreds of thousands of templates to use for any circumstance, then instantiate and stitch together those templates during the game runtime.
- hedgehog 7mo ago
  There are a bunch of tutorials on how to use GRPO to fine tune a small Qwen. Depending what you're doing LoRA or even just prefix tuning can give pretty good results with no special hardware.
derwiki 7mo ago
Exactly, inference cost is a very good reason to fine tune with something like Qwen
butILoveLife 7mo ago
This is literally what I'm waiting for. I want a ~8B model that works well with OpenClaw.
- prettyblocks 7mo ago
  I don't think you will get that anytime soon because for a model to work well with something like openclaw it needs a massive context window.
  - butILoveLife 7mo ago
    but but but but unified memory! (jk, I don't actually believe in Apple marketing words) There might be future optimizations. Like, have your small model do COT to find where to look for memory that is relevant.
- piyh 7mo ago
  Qwen 9B doesn't?
  - butILoveLife 7mo ago
    Nothing is really usable outside Opus. I've tried too. Wasted a few days trying out even high end paid models.
Me1000 7mo ago
Wouldn’t it be better to use a grammar in the token sampler? Tuning is fine, but doesn’t guarantee a syntactical correct structured output. But if the sampler is grammar aware it could.
- MillionOClock 7mo ago
  I think both should be done, they don't really serve the same purpose.
faxmeyourcode 7mo ago
Especially for super constrained applications. I don't care if the language model that I use for my extremely specific business domain can solve PhD math or remember the works of Shakespeare. I'd trade all of that for pure task specific accuracy.
- arkmm 7mo ago
  Can you share more details about your use case? The good applications of fine tuning are usually pretty niche, which tends to make people feel like others might not be interested in hearing the details. As a result it's really hard to read about real-world use cases online. I think a lot of people would love to hear more details - at least I know I would!
  - faxmeyourcode 7mo ago
    If you treat LLMs as generic transformers, you can fine tune with a ton of examples of input output pairs. For messy input data with lots of examples already built, this is ideal. At my day job we have experimented with fine tuned transformers for our receipt processing workflow. We take images of receipts, run them through OCR (this step might not even be necessary, but we do it at scale already anyways), and then take the OCR output text blobs and "transform" them into structured receipts with retailer, details like zip code, transaction timestamps, line items, sales taxes, sales, etc. I trained a small LLM (mistral-7b) via SFT with 1000 (maybe 10,000? I don't remember) examples from receipts in our database from 2019. When I tested the model on receipts from 2020 it hit something like 98% accuracy. The key that made this work so well is that we had a ton of data (potentially billions of example input/output pairs) and we could easily evaluate the correctness by unpacking the json output and comparing with our source tables. Note that this isn't running in production, it was an experiment. There are edge cases I didn't consider, and there's a lot more to it in terms of accurately evaling, when to re-train, dealing with net new receipt types, retailers, new languages (we're doing global expansion RN so it's top of mind), general diversity of edge cases in your training data, etc.
_the_inflator 7mo ago
I agree. Also for certain use cases there are constraints like embedded hardware systems with no internet access. These LLMs have to be trained to specialize for clearly defined use cases under hardware constraints. Frontier LLMs also are rarely function in isolation instead are orchestrating a system of special units aka subsystems and agents. While costs and effort are one thing, being able to downsize these monster LLMs through finetuning itself in the first place is extremly valuable.
andriy_koval 7mo ago
> "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run. I am not expert in this topic, but I am wondering if large cached context is actually cheap to run and frontier models would be cost efficient too in such setting?
- prettyblocks 7mo ago
  I'd like to read more about that if anyone has any suggestions.
  - andriy_koval 7mo ago
    I am not expert in this topic, but its easy to observe that price for cached tokens is usually 10x cheaper on major providers.

throwaway6977 7mo ago

I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. It needs to be as tiny as possible so it doesn't take resources away from the running game.

yw3410 7mo ago
How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory?
- lelanthran 7mo ago
  > How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory? I dunno, for game prose I expect that a tiny highly quantized model would be sufficient (generating no more than a paragraph), so 300MB - 500MB maybe? Running on CPU not GPU is feasible too, I think.
lelanthran 7mo ago
> I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. Unless your game states have combinatoral exlosion, would it not be better to generate all of that pre-build? If templated you can generate a few hundreds of thousands of templates to use for any circumstance, then instantiate and stitch together those templates during the game runtime.
hedgehog 7mo ago
There are a bunch of tutorials on how to use GRPO to fine tune a small Qwen. Depending what you're doing LoRA or even just prefix tuning can give pretty good results with no special hardware.

derwiki 7mo ago

Exactly, inference cost is a very good reason to fine tune with something like Qwen

butILoveLife 7mo ago

This is literally what I'm waiting for. I want a ~8B model that works well with OpenClaw.

prettyblocks 7mo ago
I don't think you will get that anytime soon because for a model to work well with something like openclaw it needs a massive context window.
- butILoveLife 7mo ago
  but but but but unified memory! (jk, I don't actually believe in Apple marketing words) There might be future optimizations. Like, have your small model do COT to find where to look for memory that is relevant.
piyh 7mo ago
Qwen 9B doesn't?
- butILoveLife 7mo ago
  Nothing is really usable outside Opus. I've tried too. Wasted a few days trying out even high end paid models.

Me1000 7mo ago

Wouldn’t it be better to use a grammar in the token sampler? Tuning is fine, but doesn’t guarantee a syntactical correct structured output. But if the sampler is grammar aware it could.

MillionOClock 7mo ago
I think both should be done, they don't really serve the same purpose.

faxmeyourcode 7mo ago

Especially for super constrained applications. I don't care if the language model that I use for my extremely specific business domain can solve PhD math or remember the works of Shakespeare. I'd trade all of that for pure task specific accuracy.

arkmm 7mo ago
Can you share more details about your use case? The good applications of fine tuning are usually pretty niche, which tends to make people feel like others might not be interested in hearing the details. As a result it's really hard to read about real-world use cases online. I think a lot of people would love to hear more details - at least I know I would!
- faxmeyourcode 7mo ago
  If you treat LLMs as generic transformers, you can fine tune with a ton of examples of input output pairs. For messy input data with lots of examples already built, this is ideal. At my day job we have experimented with fine tuned transformers for our receipt processing workflow. We take images of receipts, run them through OCR (this step might not even be necessary, but we do it at scale already anyways), and then take the OCR output text blobs and "transform" them into structured receipts with retailer, details like zip code, transaction timestamps, line items, sales taxes, sales, etc. I trained a small LLM (mistral-7b) via SFT with 1000 (maybe 10,000? I don't remember) examples from receipts in our database from 2019. When I tested the model on receipts from 2020 it hit something like 98% accuracy. The key that made this work so well is that we had a ton of data (potentially billions of example input/output pairs) and we could easily evaluate the correctness by unpacking the json output and comparing with our source tables. Note that this isn't running in production, it was an experiment. There are edge cases I didn't consider, and there's a lot more to it in terms of accurately evaling, when to re-train, dealing with net new receipt types, retailers, new languages (we're doing global expansion RN so it's top of mind), general diversity of edge cases in your training data, etc.

_the_inflator 7mo ago

I agree. Also for certain use cases there are constraints like embedded hardware systems with no internet access. These LLMs have to be trained to specialize for clearly defined use cases under hardware constraints. Frontier LLMs also are rarely function in isolation instead are orchestrating a system of special units aka subsystems and agents. While costs and effort are one thing, being able to downsize these monster LLMs through finetuning itself in the first place is extremly valuable.

andriy_koval 7mo ago

> "Frontier LLMs can do it with enough context" is not really a strong argument against fine-tuning, because they're expensive to run. I am not expert in this topic, but I am wondering if large cached context is actually cheap to run and frontier models would be cost efficient too in such setting?

prettyblocks 7mo ago
I'd like to read more about that if anyone has any suggestions.
- andriy_koval 7mo ago
  I am not expert in this topic, but its easy to observe that price for cached tokens is usually 10x cheaper on major providers.