8 ms·

Fine-tune Google's Gemma 3

• rockwotj 2y ago
  is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months
  • deepsquirrelnet 2y ago
    I’m trying right now. The combination of small models, qlora and grpo has made it accessible to experimenters. I’m not using unsloth yet, but I will probably start checking it out pretty soon so that I can train larger models or increase the number of generations for grpo.
  • simonw 2y ago
    I've had trouble getting a great answer to this question - I ask it in various places every month or so, most recently here: https://nitter.net/simonw/status/1895301139819860202 https://nitter.net/simonw/status/1895301139819860202 On paper fine tuning smaller models can greatly reduce the cost for a specific task, but I've not heard many real-world success stories around that. I think vision LLMs are one of the most interesting applications here - things like fine-tuning for better results extracting data from a specific paper form or report structure. Again, not many public examples of that.
    • kiratp 2y ago
      We use multiple post-trained models in production, at scale at https://osmos.io https://osmos.io
      • simonw 2y ago
        Have you published details of how you're doing that anywhere? Could be a useful marketing strategy for you, given how starved we all are of information about successful fine tuning stories.
        • kiratp 2y ago
          Things have been moving so fast that it’s honestly hard for a small team to do that in parallel. I got to present at GCP Next about a part of this last year: https://www.youtube.com/watch?v=5QsM1K9ahtw https://www.youtube.com/watch?v=5QsM1K9ahtw I’m presenting in one (and maybe two) sessions with more info on the training side this year.
    • deet 2y ago
      Vision LLMs are definitely an interesting application. At Avy.ai we're running small (2B-7B, quantized) vision models as part of a Mac desktop application for understanding what someone is working on in the moment, to offer them related information and actions. We found that the raw results in understanding the images with a light LORA fine tune are not substantially different -- but the ease of getting a small model to follow instructions in outputting structured data in response to the image and at the level of verbosity and detail we need is greatly enhanced with fine tuning. Without fine tuning the models on the smaller end of that scale would be much more difficult to use, not reliably producing output that matched what the consuming application expects
      • msp26 2y ago
        Was constrained decoding not enough to force the output to be in a specific format?
        • deet 2y ago
          Using a grammar to force decoding say valid JSON would work, but that hasn't always been available in the implementations we've been using (like MLX). Solvable by software engineering and adding that to the decoders in those frameworks, but fine tuning has been effective without that work. The bigger thing though was getting the models to have the appropriate levels of verbosity and detail in their ouput which fine tuning made more consistent.
    • danielhanchen 2y ago
      Oh there's a lot! Some cool examples I see: 1. Codebases, docs, large corpses of internal datasets - fill in the middle, auto completion etc. 2. I know a tonne of financial institutions use fine-tuning for trading, real time data parsing headline analysis, signal creation etc 3. Distillation is also relatively common - taking outputs of a large model and distilling it to a small model 4. Accuracy increasing is the most important - not cost or latency - we find if you solve the finetuning life cycle ie continuous auto fine-tuning, data filtering, reinforcement learning via DPO, that works well! 5. Lots of organizations use DPO and preference fine-tuning to align models since they have tonnes of feedback data! 6. Yep vision fine-tuning! For eg medical diagnosis, docs, qa on pics etc 7. And obviously large model labs finetune all base models ie chatgpt4.5 is a finetune of a base model 8. Finally reasoning finetuning via GRPO is very cool! If you have inputs and outputs but no labelled cot in between, GRPO is the way to go! Custom reward functions by companies!
      • simonw 2y ago
        "Codebases, docs, large corpses of internal datasets" I still haven't seen a convincing demo of using fine-tuning to "teach" a model new information from additional documents. I'd love to see one. (Closest I've come to that is I heard a rumor that Jane Street have fine-tuned an LLM for OCaml)
        • adefa 2y ago
          Here is a small LLM I trained to output dollars and cents from a verbal numeric amount: https://huggingface.co/TrevorJS/check-amount-deverbalizer-smollm2 https://huggingface.co/TrevorJS/check-amount-deverbalizer-sm...
  • jbentley1 2y ago
    I am. I have some use cases related to data extraction where using a fine tuned small model outperforms the best-in-class closed source models and at a fraction of the cost.
  • icelancer 2y ago
    We were but with the models becoming so good, so large, and so cheap, we've largely abandoned it in our long-term roadmap.
  • minimaxir 2y ago
    Finetuning is easy and worthwhile, especially with LoRAs as these Unsloth demos do. The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable. In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.
    • slopeloaf 2y ago
      Yeah this big time. I haven’t found a solution that makes sense. Larger models are already good enough and so convenient. When it’s more feasible to do inference on the client (browser or desktop) I can see SLMs popping up more common in production.
    • naveen99 2y ago
      If you have the resources to fine tune, you have the resources to run inference on fine tuned model. If you want to scale up and down on demand, you can just fine tune on openai and google cloud as well.
      • simonw 2y ago
        > If you have the resources to fine tune, you have the resources to run inference on fine tuned model. I don't think that's true. I can fine tune a model by renting a few A100s for a few hours, total cost in the double digit dollars. It's a one-time cost. Running inference with the resulting model for a production application could cost single digit dollars per hour, which adds up to hundreds or even thousands of dollars a month on an ongoing basis.
        • curious_cat_163 2y ago
          This assumes that inference is needed 24/7. That may or may not be true for use-cases that require asynchronous, bulk inference _and_ require some task-specific post-training. FWIW, my approach towards tasks like the above is to 1. start with using an off-the-shelf LM API until 2. one figures out (using evals that capture product intent) what the failure modes are (there always are some) and then 3. post-train against those (using the evals)
  • refulgentis 2y ago
    IMHO the biggest factor holding that back is how rushed and distanced these model releases are, still. Both Phi-4-mini and Gemma 3 were released recently. Phi-4's damn close to a good, real, model release. Microsoft's done a great job of iterating. Gemma 3's an excellent, intelligent, model, but it's got a gaping blind spot: tool-calling / JSON output. There was a vague quick handwave about it in some PR, a PM/eng on the Gemma team commented here in response to someone else that TL;DR "it's supported in Ollama!", which is Not Even Wrong, i.e. in the Pauli sense of the phrase. - Ollama uses a weak, out of date llama.cpp thing where the output tokens are constrained to match a JSON schema. This falls apart almost immediately, i.e. as soon as there is more than one tool. - The thing that matters isn't whether we can constrain output tokens, any model can do that, I've had Llama 3 1B making tool calls that way. The thing that matters is A) did you train that in and B) if you did, tell us the format All that to say, IMHO we're still 6 months to a year out from BigCo understanding enough about their own stuff to even have a good base for it. Sure, tool calling and fine-tuning are orthogonal, in a sense, but in practice, if I'm interested in getting a specific type of output, odds are I wanted that formatted a specific way.
    • tough 2y ago
      could one train now a gemma 3 fine tune for tool use? found this on HF https://huggingface.co/ZySec-AI/gemma-3-27b-tools https://huggingface.co/ZySec-AI/gemma-3-27b-tools
    • eternityforest 2y ago
      Gemma3 1B seems to be able to choose which tool to use for very simple cases, if you constrain using anyOf, and narrow it down to just a few with RAG first. It can't understand numbers very well though, "one thousand five" might become "1500". JSON constraints seem to make them unable to figure it out even if they'd normally get it every time. Maybe it's different with models above 4B though.
  • 317070 2y ago
    I've been finetuning these models since before chatGPT, and the one lesson I've learned is that by the time you have set up everything to fine-tune a model, you can expect a newer model to do as well with prompt-tuning. So, unless you hope to stay at the fore front (e.g. to be ahead of competitors), there has been no real reason to finetune for the last 4 years, at best you could hope to stay about 1-3 months ahead, depending on how fast you were at setting up your training. And if that is what you did hope to achieve, you needed to automate on a higher level, i.e. automate data collection and the collection of eval cases.
    • nwienert 2y ago
      It feels like there should be a service where I just drag drop a folder of examples and it fine tunes the latest DeepSeek or whatever for me and even can host it for me at some cost. I'd pay for that immediately, but last I checked there was nothing that really did that well (would love to be wrong).
      • arkmm 2y ago
        There are some options out there, depending on what type of task you're trying to fine tune. I think RL finetuning for DeepSeek e.g. isn't well developed yet, but you can finetune a small LLama model (~3B params) for classification or extraction tasks and it works really well. What sort of tasks were you looking at finetuning for?
        • nwienert 2y ago
          Code generation or question answering. But ideally 70+B
    • m101 2y ago
      I feel like this is true but would be great if you could provide examples so we could get a better idea of why you think/know this.
      • 317070 2y ago
  • netdur 2y ago
    I have documents from the last 50 years that I need to digitalize, millions of them written in old Arabic. The OCR is not accurate due to handwritten documents, so I need to fine-tune a model on around 300k pairs of texts (OCR output and manually corrected versions)
    • Diederich 2y ago
      This sounds very interesting; can you share more? Thanks!
      • netdur 2y ago
        I followed this guide for fine-tuning: https://ai.google.dev/gemini-api/docs/model-tuning https://ai.google.dev/gemini-api/docs/model-tuning Arabic OCR is a mess with historical texts. Take the word الف (alf/thousand) in dates like 1950 - in old documents, the ف (fa) had a dot below it, but modern OCR doesn't get this and outputs الد (alad), which is just gibberish in Arabic Same problem with ق (qaf) written as ف (fa) in old Arabic And don't get me started on merged letters! In محمد (Muhammad), sometimes the م (meem) sits right on top of the ح (haa), or appears as a little circle below the line. Modern OCR has no clue what to do with these My solution? Run OCR first, then use LLMs to fix the mess based on context. The surprising part? In my tinkering, smaller fine-tuned models actually do BETTER at this specific task than the big general-purpose ones. They seem to learn the patterns of historical Arabic quirks more effectively. Pretty neat tradeoff of specialized knowledge vs. general intelligence
• siliconc0w 2y ago
  It likely makes sense to use more expensive frontier models as teachers or architects for smaller fine-tuned ones that generate the majority of tokens (though possibly against the ToS).
• yieldcrv 2y ago
  Instead of versions, these things should be labeled by their release date, since this kind of training is based on started at a dataset snapshot in time, colloquially called knowledge-cutoff date which isnt really accurate we are optimizing these on different dimensions at once, and multiple branches of evolution from each model so a successor version name doesn't really convey that
• bryan0 2y ago
  Are people fine-tuning LLMs on their local machines with a single GPU? What are people using to scale their training to multiple nodes / gpus? I've been playing around with Hugging Face Estimators in sagemaker.huggingface but not sure if there are better options for this?
  • samspenc 2y ago
    It takes a significant amount of time (few hours) on a single consumer GPU, even 4090 / 5090, on personal machines. I think most people use online services like runpod, vast ai, etc to rent out high-powered H100 and similar GPUs for a few cents per hour, run the fine-tuning / training there, and just use local GPUs for inference on those fine-tuned models generated on cloud-rented instances.
    • danielhanchen 2y ago
      It used to be that way! Interestingly I find people in large orgs and the general enthusiast don't mind waiting - memory usage and quality are more important factors!
  • _ea1k 2y ago
    For experimentation? Absolutely. It can often be done overnight for smaller models and reasonably sized GPUs (24GB+). It'd become a lot less practical with huge datasets, but I'd guess that a lot of fine tuning tasks aren't really that large.
  • michaelt 2y ago
    Take a look at the hardware requirements at https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#hardware-requirement https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#... A 'LoRA' is a memory-efficient type of fine tuning that only tunes a small fraction of the LLM's parameters. And 'quantisation' reduces an LLM to, say, 4 bits per parameter. So it's feasible to fine-tune a 7B parameter model at home. Anything bigger than 7B parameters and you'll want to look at renting GPUs on a platform like Runpod. In the current market, there are used 4090s selling on ebay right now for $2100 while runpod will rent you a 4090 for $0.34/hr - you do the math. It's certainly possible to scale model training to span multiple nodes, but generally scaling through bigger GPUs and more GPUs per machine is easier.
• smokel 2y ago
  I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?
  • t1amat 2y ago
    I would like to see more knowledgeable people with experience talk about this. Is it just a matter of assembling Q/A pairs like: “What’s class X?”, “class X { … }” Do you really need to do this training on the base model instead, which means you have to fine tune chat on it afterward? How does this work?
    • Tostino 2y ago
      I've not done fine tuning on code bases but I have done other fine tuning. You will generally get better results when you fine-tune the base model on your data. Since you still want to use it with the chat template in the end, you fine-tune the base model with the chat template with your specific data. From there you'll have a lora that knows your data alright, but still doesn't really work for chatting. You take that lora, merge it with the base model. Let's call this the stage model. Then you use mergekit to merge the base model with both the stage model and the chat model. I used the TIES merge method in the past. Now you have your final model. I use vLLM for inference, and needed access to multiple fine tunes on only a single set of hardware. So from that point I go and take the base model and my final model and extract a new lora. I also take the base model and chat model and extract another lora for that. Then I load up vLLM with the base model and as many of the fine tune loras I need + the chat lora. The only time this hasn't worked is if the chat model adds a bunch of new tokens on top of the base model. If I remember right there was an issue with that This has worked well for me in the past.
      • danielhanchen 2y ago
        Yes!! The trick is the merging of models weights!!
• admiralrohan 2y ago
  Have anyone used those small models in any production environment? If yes, what they are good and bad at?
• huqedato 2y ago
  Great article, but I didn't see anything about the costs. I'm particularly interested in this aspect because we're considering fine-tuning Gemma 3, but our budget is tight. We're looking into (real-world) cost estimates for this approach.
  • flakiness 2y ago
    > This also means Colab Notebooks with free Tesla T4 GPUs also work! My understanding is that they don't charge these by themselves although you might have to pay Colab fee to Google. They charge higher end models it seems https://unsloth.ai/pricing https://unsloth.ai/pricing
    • danielhanchen 2y ago
      Yep via Colab for now! If its popular I can spin up a deployment fine-tuning system!
  • danielhanchen 2y ago
    Oh hey! For now we don't have a platform, so we generally tell folks to use Colab free gpus! Kaggle also has 30 hours for free per week! I put links for kaggle here: https://docs.unsloth.ai/get-started/unsloth-notebooks https://docs.unsloth.ai/get-started/unsloth-notebooks
• zk 2y ago
  Is there a version of Gemma 3 that has tool calling? Google's blog claimed it supports tools but it doesn't seem like it actually does.
  • weird-eye-issue 2y ago
    Not sure but you can just implement tool calling with some custom prompting yourself, it's really not too hard and is what we do
  • xnx 2y ago
    Function calling docs: https://ai.google.dev/gemma/docs/capabilities/function-calling https://ai.google.dev/gemma/docs/capabilities/function-calli...
• yash2401 2y ago
  [dead]
• dhooper 2y ago
  Please try to enjoy each Gemma tuning equally, and not show preference for any over the others