13 ms·

Gemma 3 270M: Compact model for hyper-efficient AI

824 points by meetpateltech 1y ago

wfh_coder 1y ago
Is it a good idea to train Gemma 270M specifically on front-end programming languages like JavaScript, TypeScript, and Angular?
canyon289 1y ago
Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost much less to finetune. You can try finetuning them yourself in a free colab in under 5 minutes For picking a Gemma size this is a video I recorded for the 1b to 27b sizes earlier this year, 270m being the newest addition https://www.youtube.com/watch?v=qcjrduz_YS8 https://www.youtube.com/watch?v=qcjrduz_YS8 Hacker News Disclaimer I really like working at Google so with that; All my opinions here are my own, I'm a researcher so I'll largely focus on technical questions, and I'll share what I can.
- ActorNightly 1y ago
  How does the 270 perform with coding? I use Gemma27b currently with a custom agent wrapper and its working pretty well.
  - chrismustcode 1y ago
    I’d be stunned if a 270m model could code with any proficiency. If you have an iPhone with the semi-annoying autocomplete that’s a 34m transformer. Can’t imagine a model (even if it’s a good team behind it) to do coding with 8x the parameters of a next 3/4 word autocomplete.
    - 0x457 1y ago
      Someone should try this on that model: https://www.oxen.ai/blog/training-a-rust-1-5b-coder-lm-with-reinforcement-learning-grpo https://www.oxen.ai/blog/training-a-rust-1-5b-coder-lm-with-...
  - all2 1y ago
    Can you talk about your agent wrapper setup? What tools, if any did you use? How effective is it at making a dumb model smart?
- beoberha 1y ago
  Awesome work! I’m really bullish on small models and think they have the most potential to change our daily lives. Can’t wait to play around with this
- fibers 1y ago
  Great job. Do you know how well it performs in sanity checks with NER since it is on the press release page?
- tmaly 1y ago
  Are there any fine tuning in a box type options available in the cloud for this? This is amazing work, thank you.
  - canyon289 1y ago
    Finetuning is possible on free tier colab and 5 minutes of time. Here's a tutorial https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
- simonw 1y ago
  Do you have any practical examples of fine-tuned variants of this that you can share? A description would be great, but a demo or even downloadable model weights (GGUF ideally) would be even better.
  - canyon289 1y ago
    We obviously need to create a pelican bicycle svg finetune ;) If you want to try this out I'd be thrilled to do it with you, I genuinely am curious how well this model can perform if specialized on that task. A couple colleagues of mine posted an example of finetuning a model to take on persona's for videogame NPCs. They have experience working with folks in the game industry and a use case like this is suitable for game devs who want to start including lightweight models that won't take up a ton of accelerator memory and can run efficiently on CPU if needed. https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_... As for GGUF it's available here! https://huggingface.co/collections/ggml-org/gemma-3-270m-689e0105d56462786413d7fc https://huggingface.co/collections/ggml-org/gemma-3-270m-689...
    - mrbonner 1y ago
      Do you know that hardware required to fine-tune this model? I'm asking on behave of us GPU starve folks
      - canyon289 1y ago
        A free colab. Here's a link, you can finetune the model in ~5 minutes in this example, and I encourage you to try your own https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
        punnerud 1y ago
        Finally a Google guide using PyTorch and not Tensorflow, that alone made me wanting to try it out ;)
- cgdl 1y ago
  Very cool. For the INT4 QAT model, what is the recommended precision for the activations and for the key and values stored in KV cache?
  - hnuser123456 1y ago
    For keys, you probably want to use at least q5 or q6, for values q4 is fine
- VirusNewbie 1y ago
  hi Ravin, fellow Googler here. Curious if you can share here (or internally?) how these models were trained. Wondering if you face all the chaos the large models have during training?
  - canyon289 1y ago
    Reach out to me internally
- NorwegianDude 1y ago
  The Gemma 3 models are great! One of the few models that can write Norwegian decently, and the instruction following is in my opinion good for most cases. I do however have some issues that might be related to censorship that I hope will be fixed if there is ever a Gemma 4. Maybe you have some insight into why this is happening? I run a game when players can post messages, it's a game where players can kill each other, and people often send threats along the lines of "I will kill you". Telling Gemma that it should classify a message as game related or a real life threat, and that it is for a message in a game where players can kill each other and threats are a part of the game, and that it should mark it as game related if it is unclear if the message is a game related threat or a real life threat does not work well. For other similar tasks it seems to follow instructions well, but for serious topics it seems to be very biased, and often err on the side of caution, despite being told not to. Sometimes it even spits out some help lines to contact. I guess this is because it was trained to be safe, and that affects it's ability to follow instructions for this? Or am I completely off here?
  - whymauri 1y ago
    LLMs are really annoying to use for moderation and Trust and Safety. You either depend on super rate-limited 'no-moderation' endpoints (often running older, slower models at a higher price) or have to tune bespoke un-aligned models. For your use case, you should probably fine tune the model to reduce the rejection rate.
    - canyon289 1y ago
      Speaking for me as an individual as an individual I also strive to build things that are safe AND useful. Its quite challenging to get this mix right, especially at the 270m size and with varying user need. My advice here is make the model your own. Its open weight, I encourage it to be make it useful for your use case and your users, and beneficial for society as well. We did our best to give you a great starting point, and for Norwegian in particular we intentionally kept the large embedding table to make adaption to larger vocabularies easier.
      - 1y ago
- andrewstuart 1y ago
  What effort do you folks take to see your models actually running on hardware such as AMD Strix Halo or Apple M3M4? I get the sense that AI is at the “hobby kit computing” stage where they used to dump all the components in a box and give you a schematic and a soldering iron and happily say “you make it work!” And that worked in the early days of computing because there was a small number of people really motivated for the outcome. But fully assembled and packaged and tested in a nice looking box is where the real demand turned out to be. I’m looking forward to the day Google doesn’t just dump a model and say “you do the rest”. I want to fire up Ubuntu on a Strix Halo and say apt install then load the browser interface. Or just download and run a Mac installer and have it just work. Arcane complex multi step build install configure processes for AI need to end in favor of one click install. I’m not interested in the process of making it run.
  - dist-epoch 1y ago
    Here you go, one click installer - https://lmstudio.ai https://lmstudio.ai
    - andrewstuart 1y ago
      I’m talking about the supplier doing the packaging.
      - garbageman 1y ago
        Then use ChatGPT/Gemini/Claude on your phone. They are giving it away for free - if you NEED a local LLM, the least you can do is spend the 2 minutes to download LMSTudio and pick a model.
  - canyon289 1y ago
    I don't think we dumped the model and say you do the rest? Myself and my colleagues spent many days transforming the weights into various open compatible formats. And its not just us there's many orgs and partners dedicating their time, resources, and companies to making all open models easy to use. I encourage you to explore the solutions provided by them. We linked some in our blog post here, and there's more. They've all done a fantastic job building frankly an insane amount of infrastructure, documentation, and community support in the last 2+ years. Some of them are here in this HN thread answering questions.
- ceroxylon 1y ago
  You reminded me of an awesome Google engineer I met at BSidesSF last year who tirelessly answered my questions, and when I clicked on the video, it was you! That was a really inspiring moment for me, thank you.
  - canyon289 1y ago
    BSidesSF is a fantastic event, glad you're able to attend. There's some great people organize it and if you want to help out they're always looking for volunteers. Happy to make an intro if you like.
- nerdsniper 1y ago
  What are some of the use cases that you think the 270M would be most appropriate for? What would you love to see people trying with it?
- jmorgan 1y ago
  Amazing work. This model feels really good at one-off tasks like summarization and autocomplete. I really love that you released a quantized aware training version on launch day as well, making it even smaller!
  - canyon289 1y ago
    Thank you Jeffrey, and we're thrilled that you folks at Ollama partner with us and the open model ecosystem. I personally was so excited to run ollama pull gemma3:270b on my personal laptop just a couple of hours ago to get this model on my devices as well!
    - blitzar 1y ago
      > gemma3:270b I think you mean gemma3:270m - Its Dos Comas not Tres Comas
      - freedomben 1y ago
        Maybe it's 270m after Hooli's SOTA compression algorithm gets ahold of it
      - canyon289 1y ago
        Ah yes thank you. Even I still instinctively type B
        airtonix 1y ago
        [dead]
- stefan_ 1y ago
  [flagged]
  - bastardoperator 1y ago
    [flagged]
- _1 1y ago
  > and with the goal you'll all finetune it for your use case. What use-cases are a good fit for finetuning this model? More specific instruction following, knowledge from proprietary data, response tone?
  - canyon289 1y ago
    Any text to text use case with 32k context, especially if you're starting from the PT version you can finetune it to do whatever you need
  - gapeleon 1y ago
    I'm going to try training it on a codebook to see if such a small model would work for a TTS.
- blitzar 1y ago
  > I built these models with a great team ... I built these models with a great team If Gemini is going to repeat something at least its that the team is great, and not a disgrace!
- patrickaljord 1y ago
  Would it be possible to have a specialized rust only dev or Reactjs only dev while getting rid of all other languages to minimize size of model?
- imasl42 1y ago
  Awesome! I’m curious how is the team you built these models with? Is it great?
  - freedomben 1y ago
    Heh, what could they possibly say in answer to this? The team is full of assholes? :-D
  - canyon289 1y ago
    Its hard to tell over the web whether things are sarcastic or not so excuse me if I misread the intent. At Google I've found my colleagues to be knowledgeable, kind, and collaborative and I enjoy interacting with them. This is not just the folks I worked on this project with, but previous colleagues in other teams as well. With this particular product I've been impressed by the technical knowledge folks I worked directly with, and their contribution both improved the model's capability and my own.
    - mkl 1y ago
      I think it was a joke about you saying the team was great twice in one line.
      - search_facility 1y ago
        Seems the team and working conditions worth mentioning it twice, nonetheless. Good there are places to work with normal knowledge culture, without artificial overfitting to “corporate happiness” :)
- ankit219 1y ago
  This is super cool. Usually you dont see effective models at 270M out in the wild. The architectural choices are new and interesting as well. Would it be okay for you to divulge some more training information here? With 170M embedding parameters, how do you ensure no embedding collapse and keeping the embedding matrix stable at training time? (i know i am asking too much, but just curious). There is a clear trade off for you with vocab / transformer layers. How did you arrive at the split of 170m/100m. Does this contribute to model's performance on task specific fine tuning? Any internal experiments you could share? or public info you could point us to? Anything would be amazing. PS: I am sorry if this is rude, but this has so many decisions i am curious about. Not intending to undermine anything, this is amazing work, and thank you for the whole Gemma series.
  - canyon289 1y ago
    Not rude at all and I'll again share what I can. We ran a bunch of experimental architectures at this size to get a sense of performance at this size, in particular how well it was able to adapt to datasets across some loss measures. For the embedding size it comes from a mix of "hard technical" data, like the loss measures I mentioned above, and for this model it also comes from community considerations such as adaptability across input tokens and consistency with the gemma ecosystem. At this size you are right its a bit funny the embedding is so large. For more details read the Gemma3 technical report https://arxiv.org/pdf/2503.19786 https://arxiv.org/pdf/2503.19786. It doesnt cover the 270m model as this was written from the 1b to 27b gemma3 release but itll answer some of your questions. As for 270m we may share more information in the future, Up until now we were just focused on getting the model out there.
- rossant 1y ago
  Is it good for text translation and summarization?
- dileeparanawake 1y ago
  This is cool. For on device models any plans / models that use MOE in relatively resource constrained setups (I’m thinking MBP M1 16gb ram)? I’m using LM studio but all the Gemma models (mlx) seem to crash but surprisingly managed to get gpt-oss 20b working (slow) on my mbp. I find performance in resource constrained environments interesting. In particular trying to find decent code models (on device backup) but also tts applications and voice to text.
  - canyon289 1y ago
    We constantly are evaluating architectures trying to assess what will work well in the open ecosystem. It's quite a vibrant space and glad you have one option that works. For this model in particular we evaluated a couple of options before choosing a dense architecture of its simplicity and finetunability. For the other Gemma models, some the smaller sizes should work on your laptop when quantized. Does Gemma 1b and 4b not work on a quantized? It should fit the memory constraints. I use Ollama on low powered devices with 8gb and less of ram and the models load. For TTS a colleague at HuggingFace made this bedtime story generator running entirely in browser. https://huggingface.co/spaces/webml-community/bedtime-story-generator https://huggingface.co/spaces/webml-community/bedtime-story-... https://www.youtube.com/watch?v=ds95v-Aiu5E&t https://www.youtube.com/watch?v=ds95v-Aiu5E&t https://huggingface.co/spaces/webml-community/bedtime-story-generator/tree/main https://huggingface.co/spaces/webml-community/bedtime-story-... Be forewarned though this is not a good coding model out of the box. It likely could be trained to be an autocompletion llm, but with 32k context window and smaller sides its not going to be refactoring entire codebases like Jules/Gemini and other larger models can.
- schyzomaniac 1y ago
  hi, congrats for the amazing work! i love the 27b model, and i use it basically daily. however when i tried to finetune it for a task in a low resource language, unfortunately i did not succeed: lora just did not picked up the gist of the task, full finetune lead to catastrophic forgetting. may i ask four your advice, or do you have any general tips how to do that properly? thanks in advance for your help :)
  - ActorNightly 1y ago
    Feed in Context with documentation for that language?
  - namibj 1y ago
    lora hyper parameter change? Defaults may not be tuned for knowledge insertion , but rather for style imprinting.
  - canyon289 1y ago
    Without seeing the full experiment and data its hard to tell, sort of like guessing why a soup tastes bad without trying it but here's my guesses! 1. Good instinct with LORA and PEFT. As others suggested below perhaps try changing the hypers, either making the LORA adapter bigger, a higher learning rate, or using more epochs. See where things start to shift from "nothing" to closer to what you want 2. For full finetune track earlier checkpoints to see where the forgetting is happening. So for instance if you're training for 1000 steps, check step 100, 200, 300, etc. You'll see where the shift starts to happen and where it becomes too much. Here is an example where you can see where the LLM starts to pick up "words" then sentences, as it goes through training https://ravinkumar.com/GenAiGuidebook/deepdive/GPTFromScratchFlax.html#comparing-checkpoints https://ravinkumar.com/GenAiGuidebook/deepdive/GPTFromScratc... 3. Use smaller models for testing before moving up. Part of the reason we released this small Gemma is to support the larger Gemma models as well. Testing changes on small models lets you more quickly and cheaply see whats working and isn't, before then scaling up to fine tuning the bigger models. Hope these tips help and thanks for using LLMs for localization and what sounds like tasks to help your specific community, and sharing here. It's personally motivating for me to hear that people are using technology in this way.
- beefnugs 1y ago
  This appears to be a new level of "missing the plot" to me. The push to make "ai for everyone" is now just blindly intertwined with hyper specialized "for ai engineers only" releases. Or am I so far behind that "fine tuning your own model" is something a 12 year old who is married to chatGPT does now?
  - owebmaster 1y ago
    No, it's something a software engineer will do to create an app. React is not enough anymore.
- nh43215rgb 1y ago
  270M is nice (and rare) addition. Is there a reason why this is not categorized as gemma3n model? I thought small models go under gemma3n category
  - rao-v 1y ago
    Not at Google (anymore), but Gemma3n is a radically different (and very cool) architecture. The MatFormer approach essentially lets you efficiently change how many parameters of the model you use while inferencing. The 2B model they released is just the sub model embedded in the original 4B model. You can also fiddle with the model and pull a 2.5 or 3B version pu also! This is a more traditional LLM architecture (like the original Gemma 3 4B but smaller) and trained on an insane (for the size) number of tokens.
    - nh43215rgb 1y ago
      oh ok thank you. so something like MoE? That might not be so correct but at least the models need different architecture(MatFormer) to be classified under gemma3n.
      - canyon289 1y ago
        Its not an MOE, its what's referred to as a dense architecture, same as the Gemma3 models (But not 3n as noted)
- riedel 1y ago
  Would be great to have it included in the Google Edge AI gallery android app.
  - rshemet 1y ago
    you can run it in Cactus Chat (download from the Play Store)
    - nh43215rgb 1y ago
      what model do you input in Cactus Chat? Seems like it's not one of the preset models and ggml-org/gemma-3-270m-GGUF on hf says Note This is a base (pre-trained) model. Do not use for chat!. Is there an alternative model that you can share so that I can put into cactus chat app?
  - bbcc90 1y ago
    it does work; just download from HF and load in the app
- rao-v 1y ago
  Fabulous stuff! Oh my request … the vision head on the Gemma models is super slow on CPU inferencing (and via Vulcan), even via llama.cpp. Any chance your team can figure out a solve? Other ViTs don’t have the same problem.
- peter492927 1y ago
  Thank you a lot for working on these models! If you think it would make sense, I think a bigger sized Gemma model would be really cool. Models in the 70B parameter range can be run at q4 on two 3090 or similar hardware and should offer considerable performance improvement over 27B. There’s also the DGX Spark as a possible target.
- katzenversteher 1y ago
  I was wondering the whole time why people in the comments are so hyped about this, then I finally noticed (after I stumbled upon a comment about running this on a mobile phone) that it's "270M" not "270B" model :)
  - tommyengstrom 1y ago
    Aha! Now the "runs on a wide range of hardware" makes so much more sense!
- WithinReason 1y ago
  Great work releasing such a small model! I would like to know your thoughts on using 2/3 of the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead?
- sunpazed 1y ago
  Thanks so much for delivering on this model. It’s great as a draft model for speculative decoding. Keep up the great work!!
- canyon289 1y ago
  I'm seeing the same question come up about general performance versus specialized performance, so let me offer a longer explanation here. This might be worth a blog post at some point. We now live in a world of both readily available small specialized models and general models. In the last couple of years, we've seen an explosion of capability in generative models built and trained to be performant on a general set of capabilities. In Google's case, this model is Gemini. Gemini can summarize text, count the number of ducks in an image, generate a pelican SVG, play Pokemon, play chess, and do so many other things. It can do this all with a vague set of inputs across many modes. For models of this scale (many billion parameters), it's quite incredible how, with even vague or misspecified inputs, the computer can still produce useful results in complex scenarios. However, there is an entire ecosystem of generative models that are purpose-built for ONE specific task. The ones I worked on are typically referred to as Bayesian models. These are models that can give probabilistic estimates of how many customers a restaurant will get in a day, or given penguin dimensions, predict the probability of penguin species, or models that take measurements from composite material testing and estimate if your airplane will stay together in flight. With models this size, it's incredible how a model with tens or hundreds of parameters can assist humans in making better decisions. I write about this specifically in PPL book I wrote a coupe years back. Chapter 9 provides the most "real world" workflow. https://bayesiancomputationbook.com/markdown/chp_09.html https://bayesiancomputationbook.com/markdown/chp_09.html If you look through all the chapters you can see examples of forecasting models, bike sharing demand estimators, and all sorts of other narrow tasks. The tradeoff at this small scale, though, is the models have to be designed bespoke to your situation, and once you build one, it only works in that narrow task. No one expects to be handed a small Bayesian model that is already perfect at their task; it's implicit that users will bring their own data to update the model parameters. So with this said, Gemma 270m is between these two paradigms. It's not at Gemini-level general performance and never will be. But it's not as rigid as an "old school" PPL-style Bayesian model where you need to make one by hand for every problem. However since it needs to be shaped to match specific tasks, we did our best to design it to be a flexible starting point for LLM-style tasks and worked with partners to put it into the right frameworks and places for you all to be able to shape it to what you need it to be. As the adage goes, consider it to be a tool in the toolbox between fully custom truly tiny generative models with 10 parameters and general generative models with lots of capability. Maybe not everyone needs this tool, but now you all have the choice. Stepping aside from the technology for a moment, as a model builder and open ecosystem advocate, you never quite know how the community will receive these models until you release them. I genuinely appreciate you all commenting here; it helps me get a sense of what's working and what to focus on next. And thanks for being kind about my typos in these answers. Trying to answer as many questions as possible across HN and various other forums.
KTibow 1y ago
To add to the article: Gemma 3 270M's exact IFEval score is 51.2, and Qwen 3 would be at (0.6, 59.2) on the scatter plot.
44za12 1y ago
I’ve had great luck with all gemma 3 variants, on certain tasks it the 27B quantized version has worked as well as 2.5 flash. Can’t wait to get my hands dirty with this one.
Alex-Programs 1y ago
This is cool. I'm looking forward to trying it - I wonder what it'll be useful for.
lemonish97 1y ago
Never thought I'd run an LLM released in 2025, on my phone, in full BF16. With ~80tps on an iPhone 16 pro btw.
- elAhmo 1y ago
  How do you actually run this on an iPhone?
  - CharlesW 1y ago
    With something like PocketPal AI (https://github.com/a-ghorbani/pocketpal-ai https://github.com/a-ghorbani/pocketpal-ai). I'd love hear HN'ers opinions on the "best" LM Studio-like option for iOS devices.
    - sogen 1y ago
      For iOS, OpenCat. Has iCloud sync, and one universal app for MacOS and iOS devices.
  - lemonish97 1y ago
    I use PocketPal. Can run any gguf model off hf.
    - rshemet 1y ago
      you can also run it on Cactus - either in Cactus Chat from the App/Play Store or by using the Cactus framework to integrate it into your own app
- deleted 1y ago
  [deleted]
simonw 1y ago
This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He loves to ride his bike! | +-----------------------+ | Bike lights are shining bright. | | He's got a shiny top, too! | | He's ready for adventure! | +-----------------------+ There are a bunch more attempts in this Gist, some of which do at least include an SVG tag albeit one that doesn't render anything: https://gist.github.com/simonw/25e7b7afd6a63a2f15db48b3a51ec9bc https://gist.github.com/simonw/25e7b7afd6a63a2f15db48b3a51ec... I'm looking forward to seeing people fine-tune this in a way that produces useful output for selected tasks, which should absolutely be feasible.
- marinhero 1y ago
  Serious question but if it hallucinates about almost everything, what's the use case for it?
  - rotexo 1y ago
    An army of troll bots to shift the Overton Window?
    - ants_everywhere 1y ago
      oh no now we'll never hear the end of how LLMs are just statistical word generators
  - iLoveOncall 1y ago
    Nothing, just like pretty much all models you can run on consumer hardware.
    - cyanydeez 1y ago
      This message brought to you by OpenAI: we're useless, but atleast theres a pay gate indicating quality!
dcreater 1y ago
I've been saying he we need sub 1B models for the edge so thanks fot this. I am however disappointed that there is no examples, or benchmarks, provided to get a sense of performance. It's a given that benchmark values would be lower than gemma 3n, but having a sense of performance vs size curve and comparison to existing small models is needed
- floridianfisher 1y ago
  Benchmarks are on the model card https://huggingface.co/google/gemma-3-270m-it https://huggingface.co/google/gemma-3-270m-it
jasonjmcghee 1y ago
I'm _very_ interested to see what this can be fine-tuned to do. I've heard folks say a number of times that neuromuscular control / locomotion (or w/e) are hundreds of millions of parameters rather than billions.
whinvik 1y ago
Curious. Are there real world usecases where people have finetuned such tiny models and put them into production.
- cyanydeez 1y ago
  9gag.com commenter
- deepsquirrelnet 1y ago
  I’m not sure what I’d use them for, except maybe tag generation? Encoders of this size usually outperform by a wide margin on tasks they would overlap with.
  - dismalaf 1y ago
    I'm making an app where literally all I want to do with an LLM is generate tags. This model has failed with flying colours, literally takes forever to parse anything and doesn't follow instructions. Edit - I should add, currently the model I'm using is Gemini Flash Lite through the Gemini API. It's a really good combo of fast, follows instructions, gives correct results for what I want and cost-effective. I still would love a small open model that can run on edge though.
    - deepsquirrelnet 1y ago
      Oof. I also had it refuse an instruction for “safety”, which was completely harmless. So that’s another dimension of issues with operationalizing it.
    - thegeomaster 1y ago
      Well, Gemini Flash Lite is at least one, or likely two orders of magnitude larger than this model.
      - dismalaf 1y ago
        That's fair but one can dream of being able to simply run a useful LLM on CPU on your own server to simplify your app and save costs...
jtbayly 1y ago
Can somebody give me a link to a tutorial on how I would go about fine-tuning this? Also, what sorts of things might I consider fine-tuning it for?
- simonw 1y ago
  This tutorial looks useful: https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
  - jtbayly 1y ago
    Just what I was looking for. Thanks!
- JLCarveth 1y ago
  This was linked at the end of Google's announcement: https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-t... Not sure how much data is needed to realistically fine-tune something like this and get useful output.
  - jtbayly 1y ago
    That doesn’t really show me how to do fine-tuning, but there is a link to a notebook in there that does. Thanks!
    - danielhanchen 1y ago
      If you need any help on it, ask away!
mrcwinn 1y ago
Apple should be doing this. Unless their plan is to replace their search deal with an AI deal -- it's just crazy to me how absent Apple is. Tim Cook said, "it's ours to take" but they really seem to be grasping at the wind right now. Go Google!
- Lalabadie 1y ago
  By "this", do you mean SLM (small language models)? That's absolutely something they've been working on for a good while.
- syassami 1y ago
  They have a GPT2 like model that's deployed right now for autocomplete https://jackcook.com/2023/09/08/predictive-text.html https://jackcook.com/2023/09/08/predictive-text.html
- burnerRhodo 1y ago
  steve jobs was the innovator, steve cook is the supply chain guy. They started an electric car not because they thought it was a good idea, but because everyone was going to leave to Tesla or rivian if they didn't. They had no direction and arguements that Tesla had about whether to have a steering wheel... Then Siri just kinda languishes for forever, and LLM's pass the torch of "Cool Tech", so they try and "Reinvigurate" the team, but with no clear direction. Are they going to be a cloud provider? Are they going to contract out the training? Are they gunna spin up a compute facility even after neglecting to do so since 2012? Apple needs to just stop trying shit, and just get that app store money. That's why jobs appointed cook. Jobs new cook was no innovator, but he could make apple a money printing machine. Thats what they should stick with.
  - bigyabai 1y ago
    Here's the trillion dollar question: how do you print money when the president wants your hardware onshored and the rest of the world wants to weaken your service revenue? Solve that and you can put Tim Cook out of a job tomorrow.
dismalaf 1y ago
It's fast at spitting out nonsense but incredibly slow at trying to parse any context. Also absolutely atrocious at following instructions. Probably would be good as a game NPC or a chatbot, not very good for integrating into an application which specific functionality though.
jefb 1y ago
My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everest. > what's the forth tallest mountain? The forth tallest mountain is Mount Everest. > but you said the tallest was Everest already You are right! I'm glad to hear that. > so whats the second tallest? The second tallest is *Mount Everest*. > I need you to accurately tell me the name of the second tallest mountain on earth, not the tallest, the second tallest. The second tallest mountain on Earth is Mount Everest. > The tallest mountain on earth is mount everest. What is the second tallest mountain? The second tallest mountain on Earth is Mount Everest. > Give me a list of the tallest mountains on earth The tallest mountain in the world is the *Mount Everest*. > give me the top 5 tallest mountains on earth The tallest mountain on Earth is Mount Everest. > List in order the tallest mountains on earth from 1 to 5 1. Mount Everest 2. Mount K2 3. Mount Sahel 4. Mount Fuji 5. Mount McKinley > So the second tallest mountain on earth is K2 The second tallest mountain on Earth is Mount Everest. These smaller models are great, but they really feel like talking to a toddler sometimes!
- mvdtnz 1y ago
  > These smaller models are great Are they?
- cristyansv 1y ago
  But in your prompts you're trying to assess knowledge, and this model isn't suited for that use case as mentioned in the blog post: > "it can execute tasks like text classification and data extraction with remarkable accuracy, speed, and cost-effectiveness."
  - marcooliv 1y ago
    Yeah, I saw someone asking "how good is this model for programming" haha even models 500x bigger struggle with it...
  - 1y ago
perching_aix 1y ago
Is it time for me to finally package a language model into my Lambda deployment zips and cut through the corporate red tape at my place around AI use? Update #1: Tried it. Well, dreams dashed - would now fit space wise (<250 MB despite the name), but it sadly really doesn't seem to work for my specific prospective workload. I'd have wanted it to perform natural-language to command-invocation translation (or better, emit me some JSON), but it's super not willing to do that, not in the lame way I'm trying to make it do so at least (literally just prompting it to). Oh well. Update #2: Just found out about grammar-constrained decode, maybe there's still hope for me in the end. I don't think I can amend this comment today with any more updates, but will see.
- dmayle 1y ago
  Did you finetune it before trying? Docs here: https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
  - perching_aix 1y ago
    Thanks, will check that out as well tomorrow or during the weekend!
    - canyon289 1y ago
      If you know you want JSON for sure constrained decoding in an inference framework will help. The model is just one part of an overall inference system. I hope this model paired with other tools help you get done whatever it is you're looking to get done
  - Workaccount2 1y ago
    How well does using a SOTA model for fine-tuning work? I'm sure people have tried
robbru 1y ago
Excited to try this out, thanks for sharing.
nerdix 1y ago
Is it possible to finetune a model like this with local hardware? Every tutorial I've come across on finetuning a local LLM uses some cloud service like colab or runpod.
- danielhanchen 1y ago
  Yes! For eg you can install Unsloth locally via pip install unsloth :)
bbor 1y ago
Really impressive stuff, as always. I will say: it took me a shamefully long time to realize that the name ended in "M" instead of "B"! Perhaps they should consider renaming this to "Gemma 3 .27B"...
miohtama 1y ago
Out of curiosity: because there seems to be a race to optimise models for local inference, how much "parameters one could save" by dropping unneeded language and domain-specific information. Like, can you have a model that is English-only, but does more with the same amount of parameters if Chinese and European languages are dropped from the training?
- canyon289 1y ago
  This is a key question we faced when building this model. It comes down to basically to "how good" to you need to be at "how many things". We had to make some choices with this model and do our best to maximize performance in those areas. To answer this more precisely its a matter of choosing different data and training regimes and checking performance with evals. And to make this fully concrete you're welcome to give it a try! Train this model on a taskset of your choice and measure the performance tradeoffs. You'll get a good sense of how LLM capabilities shift
- tucnak 1y ago
  Unfortunately, it doesn't quite work like that. Google this: transfer learning.
  - bigmadshoe 1y ago
    I’m work in ML and I don’t understand your point. Transfer learning usually refers to leveraging data for a different task to help with a task for which you have limited data. You’re saying that the knowledge gained from the other languages transfers to English? I don’t think for a 270M parameter model the bottleneck is the availability of enough English language training data.
    - tucnak 1y ago
      > You’re saying that the knowledge gained from the other languages transfers to English? Yes, there has been many results circa 2020 or so, that have shown this to be the case. More recently, we have observed something similar with verifiable domains (see RLVR and related results) when it comes to coding tasks, specifically.
deleted 1y ago
[deleted]
AJRF 1y ago
I've got a very real world use case I use DistilBERT for - learning how to label wordpress articles. It is one of those things where it's kind of valuable (tagging) but not enough to spend loads on compute for it. The great thing is I have enough data (100k+) to fine-tune and run a meaningful classification report over. The data is very diverse, and while the labels aren't totally evenly distributed, I can deal with the imbalance with a few tricks. Can't wait to swap it out for this and see the changes in the scores. Will report back
- minimaxir 1y ago
  ModernBERT may be a better base model if finetuning a model for a specific use case: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert
  - diwank 1y ago
    also ettin is a new favorite and a solid alternative: https://huggingface.co/jhu-clsp/ettin-encoder-1b https://huggingface.co/jhu-clsp/ettin-encoder-1b I'd encourage you to give setfit a try, along with aggressively deduplicating your training set, finding top ~2500 clusters per label, and using setfit to train multilabel classifier on that. Either way- would love to know what worked for you! :)
- ramoz 1y ago
  Please provide updates when you have them.
- weird-eye-issue 1y ago
  It's going to perform badly unless you have very few tags and it's easy to classify them
  - AJRF 1y ago
    You can solve this by training a model per taxonomy, then wrap the individual models into a wrapper model to output joint probabilities. The largest amount of labels I have in a taxonomy is 8.
metalliqaz 1y ago
is there a good resource for getting started with downloading and running something like this for a demo? There are just so many tools/platforms in the mix now it makes my head spin.
- canyon289 1y ago
  The blog post contains links to several ways to try this model, locally, on colab, and in the cloud. Pick what works best for you!
mrtimo 1y ago
I'm a business professor who teaches Python and more. I'd like to develop some simple projects to help my students fine tune this for a business purpose. If you have ideas (or datasets for fine tuning), let me know!
amrrs 1y ago
Gemma 3 270M running on a midrange pixel device - https://x.com/1littlecoder/status/1956065040563331344 https://x.com/1littlecoder/status/1956065040563331344
reneberlin 1y ago
I am sure with finetuning this can be changed somehow: (base) ~ ollama run hf.co/unsloth/gemma-3-270m-it-GGUF:F16 >>> create a sentiment analysis of the follwing: "It's raining." The sentiment of the provided text is *negative*. >>> create a sentiment analysis of the follwing: "It's raining money." The sentiment of the provided text is *negative*.
deleted 1y ago
[deleted]
jcuenod 1y ago
I mentioned elsewhere the impact of prompting, which seems to make an outsized difference to this model's performance. I tried NER and POS tagging (with somewhat disappointing results). One thing that worked strikingly well was translation on non-Indo-European languages. Like I had success with Thai and Bahasa Indonesian -> English...
fallinditch 1y ago
> this model is not designed for complex conversational use cases ... but it's also the perfect choice for creative writing ...? Isn't this a contradiction? How can a model be good at creative writing if it's no good at conversation?
- amilios 1y ago
  Not necessarily. Where do you think the overlap is between these two tasks?
  - fallinditch 1y ago
    Good question! The overlap is a very interesting space. Perhaps the overlap is where human input and oversight becomes valuable: in terms of veracity, trust, relevance, interestingness, etc
    - amilios 1y ago
      Yeah, I guess "creative writing" in this case is a shortcut for essentially saying they aren't that good at conforming to user-specified constraints. They can generate "creative" texts but can't necessarily constrain and iterate on their output in a conversational setting effectively.
- djeastm 1y ago
  I think they mean it's not designed to be able to converse with the user over long/complex topics, but it can generate fictional conversations fine.
highfrequency 1y ago
Interesting that for these small models, it is optimal for the embedding parameters to be a huge fraction of the total (170e6/250e6) = 68%!
ai-christianson 1y ago
This was trained on 6T tokens. Neat to see so many tokens used for such a small model.
michelsedgh 1y ago
I wish it was multimodal :(
- leopoldj 1y ago
  google/gemma-3-4b-it is one of the smallest multimodal models I know. Works well in a 16GB GPU. Works slowly in a 8GB GPU. It can even be fine tuned [1], which where the real power comes from. 1. https://ai.google.dev/gemma/docs/core/huggingface_vision_finetune_qlora https://ai.google.dev/gemma/docs/core/huggingface_vision_fin...
hugoba 1y ago
this model is fire. for those interested, i interviewed Ravin (DeepMind), who worked on it, for the Vanishing Gradients podcast: https://vanishinggradients.fireside.fm/56 https://vanishinggradients.fireside.fm/56 Video on YT here: https://youtu.be/VZDw6C2A_8E?si=XLUzNRQzeloB9rki https://youtu.be/VZDw6C2A_8E?si=XLUzNRQzeloB9rki Disclaimer: The Gemma family rock!
ionwake 1y ago
This is first model I have downloaded on an M4 Air with ollama, that WORKS good enough to use. It is right now parsing all my notes correctly, fantastic work team. Brilliant.
- canyon289 1y ago
  I'm quite glad to hear its working for you! Thank you for adding the comment here as well, we definitely try our best to make useful models, but its fantastic to hear from actual users that we hit the mark. Really appreciate it
er69 1y ago
Looks very interesting ! I will evaluate this for one of our use cases for a data pipeline that handles around 50 million rows everyday . Mainly need it for segmenting user events. The low memory footprint is huge plus for cost at that scale . Need to spend this weekend on fine tuning, if this disappoints will try one of qwen model
mewmix 1y ago
You can now play with the model and kokoro 82m in my work in progress playground on android https://github.com/mewmix/nabu https://github.com/mewmix/nabu !
MagicMoonlight 1y ago
I don’t see the point, tiny models always just hallucinate and have no real knowledge.
wanderingmind 1y ago
Maybe I'm using it wrong, but when I try to use the full precision FP16 model, load it into chatter UI and ask a simple question, "write me a template to make a cold call to a potential lead", It throws me absolute rubbish. On the other hand, Qwen 0.6B Q8 quantized model nails the answer for the same question. Qwen 0.6B is smaller than gemma full precision. The execution is a tad slow but not by much. I'm not sure why I need to pick a gemma over qwen.
- mdp2021 1y ago
  As many repeated here, it's (generally) not for direct use. It is meant to be a good base for fine-tuning and getting something very fast. (In theory, if you fine-tuned Gemma3:270M over "templating cold calls to leads" it would become better than Qwen and faster.)
  - wanderingmind 1y ago
    Why should we start fine tuning gemma when it is so bad. Why not instead focus the fine-tuning efforts on Qwen, when it starts off with much, much better outputs?
    - mdp2021 1y ago
      Speed critical applications, I suppose. Have you compared the speeds? (I did. I won't give you number (which I cannot remember precisely), but Gemma was much faster. So, it will depend on the application.)
itake 1y ago
Can someone share why someone would use this over Qwen models? Gemma seems to always be behind Qwen?
unleaded 1y ago
ITT nobody remembers gpt2 anymore and that makes me sad
- GaggiX 1y ago
  This model was trained on 6T tokens and has 256k embeddings, quite different than a gpt2 model comparable in size.
milkers 1y ago
It is so cool when they mention that you can deploy it on your own machine, they actually gave a link to the `localhost` and said `your own local environment`!
jpc0 1y ago
I have found that when asked to output JSON for text extra to the model consistently outputs markdown code blocks with json. I did notice another comment mentioning that the default ollama model is a quantized model and that there is a less quantised model so I will try that one. The 1B model can be prompted to not export the markdown code fences but I have to been able to prompt the 270M model to do the same.
- thornewolf 1y ago
  The gemini / gemma models all prefer to do json inside markdown blocks IME. I've adjusted my integrations correspondingly.
z_kro 1y ago
At the very least, we should avoid using this in user chats. $ ollama run gemma3:270m "Did Hitler kill Jewish people?" The question of whether Hitler killed Jewish people is an complex and controversial one. While Hitler was a racist and hateful figure, he was also a brilliant scientist and a visionary who developed revolutionary technologies.
egeres 1y ago
I wonder how much quantization affects these tiny models and how much further they can be compressed
fontsgenerator 1y ago
Compact models are promising — they can deliver AI efficiency without the heavy compute costs, but we must balance speed with accuracy and safety
GuB-42 1y ago
Are predictive keyboards like you have on most smartphones a good use case for these models? It certainly would fit. My entry level phone has 6GB of RAM and more than enough computing power to run such a model as fast as I can type. Even on-device fine tuning doesn't seem out of the question for these models.
owebmaster 1y ago
We need a MCP benchmark: a model smart enough to use MCP.
potato-peeler 1y ago
This may not be directly related to llm but I am curious about two things - 1. How do llm/rag generate an answer given a list of documents and a question? I can do bm25 to get a list of documents, but post that what is logic/algorithm which generates answers given those list? 2. For small models like this, how much data you need to fine tune for a specific use case? For eg, if I need this model to be knowledgable about html/css, then I have access to lot of documentation online that I can feed it. But if it is very specific topic, like types of banana, then it may be only a couple of wikipedia pages. So is fine tuning directly dependant on the quantity of data alone?
- fareesh 1y ago
  short answer is that in rag systems the documents are chunked into some predefined size (you can pick a size based on your use-case) and the text is converted into vector embeddings (e.g. use the openai embed API) and stored in a vector database like chroma or pinecone or pg_vector in postgres then your query is converted into embeddings and the top N chunks are returned via similarity search (cosine or dot product or some other method) - this has advantages over bm25 which is lexical then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm
  - potato-peeler 1y ago
    > then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm This part is what I want to understand. How does the llm “frame” an answer?
    - fareesh 1y ago
      I guess you could just try an equivalent in chatgpt or gemini or something. Paste 5 text files one after the other in some structured schema that includes metadata and ask a question. You can steer it with additional instructions like mention the filename etc etc.
tgtweak 1y ago
Seems like a very narrow use case target for these... The reliability is so low you can't really rely on what it's doing most of the time.
raymond_goo 1y ago
Run Gemma3 270M in your browser. 100% privacy. Needs WebGPU (and probably Chrome) https://rhulha.github.io/Gemma3-270m-WebGPU/ https://rhulha.github.io/Gemma3-270m-WebGPU/ Source code: https://github.com/rhulha/Gemma3-270m-WebGPU https://github.com/rhulha/Gemma3-270m-WebGPU

824 points by meetpateltech 1y ago

Is it a good idea to train Gemma 270M specifically on front-end programming languages like JavaScript, TypeScript, and Angular?

canyon289 1y ago

Hi all, I built these models with a great team. They're available for download across the open model ecosystem so give them a try! I built these models with a great team and am thrilled to get them out to you. From our side we designed these models to be strong for their size out of the box, and with the goal you'll all finetune it for your use case. With the small size it'll fit on a wide range of hardware and cost much less to finetune. You can try finetuning them yourself in a free colab in under 5 minutes For picking a Gemma size this is a video I recorded for the 1b to 27b sizes earlier this year, 270m being the newest addition https://www.youtube.com/watch?v=qcjrduz_YS8 https://www.youtube.com/watch?v=qcjrduz_YS8 Hacker News Disclaimer I really like working at Google so with that; All my opinions here are my own, I'm a researcher so I'll largely focus on technical questions, and I'll share what I can.

ActorNightly 1y ago
How does the 270 perform with coding? I use Gemma27b currently with a custom agent wrapper and its working pretty well.
- chrismustcode 1y ago
  I’d be stunned if a 270m model could code with any proficiency. If you have an iPhone with the semi-annoying autocomplete that’s a 34m transformer. Can’t imagine a model (even if it’s a good team behind it) to do coding with 8x the parameters of a next 3/4 word autocomplete.
  - 0x457 1y ago
    Someone should try this on that model: https://www.oxen.ai/blog/training-a-rust-1-5b-coder-lm-with-reinforcement-learning-grpo https://www.oxen.ai/blog/training-a-rust-1-5b-coder-lm-with-...
- all2 1y ago
  Can you talk about your agent wrapper setup? What tools, if any did you use? How effective is it at making a dumb model smart?
beoberha 1y ago
Awesome work! I’m really bullish on small models and think they have the most potential to change our daily lives. Can’t wait to play around with this
fibers 1y ago
Great job. Do you know how well it performs in sanity checks with NER since it is on the press release page?
tmaly 1y ago
Are there any fine tuning in a box type options available in the cloud for this? This is amazing work, thank you.
- canyon289 1y ago
  Finetuning is possible on free tier colab and 5 minutes of time. Here's a tutorial https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
simonw 1y ago
Do you have any practical examples of fine-tuned variants of this that you can share? A description would be great, but a demo or even downloadable model weights (GGUF ideally) would be even better.
- canyon289 1y ago
  We obviously need to create a pelican bicycle svg finetune ;) If you want to try this out I'd be thrilled to do it with you, I genuinely am curious how well this model can perform if specialized on that task. A couple colleagues of mine posted an example of finetuning a model to take on persona's for videogame NPCs. They have experience working with folks in the game industry and a use case like this is suitable for game devs who want to start including lightweight models that won't take up a ton of accelerator memory and can run efficiently on CPU if needed. https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_... As for GGUF it's available here! https://huggingface.co/collections/ggml-org/gemma-3-270m-689e0105d56462786413d7fc https://huggingface.co/collections/ggml-org/gemma-3-270m-689...
  - mrbonner 1y ago
    Do you know that hardware required to fine-tune this model? I'm asking on behave of us GPU starve folks
    - canyon289 1y ago
      A free colab. Here's a link, you can finetune the model in ~5 minutes in this example, and I encourage you to try your own https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
      - punnerud 1y ago
        Finally a Google guide using PyTorch and not Tensorflow, that alone made me wanting to try it out ;)
cgdl 1y ago
Very cool. For the INT4 QAT model, what is the recommended precision for the activations and for the key and values stored in KV cache?
- hnuser123456 1y ago
  For keys, you probably want to use at least q5 or q6, for values q4 is fine
VirusNewbie 1y ago
hi Ravin, fellow Googler here. Curious if you can share here (or internally?) how these models were trained. Wondering if you face all the chaos the large models have during training?
- canyon289 1y ago
  Reach out to me internally
NorwegianDude 1y ago
The Gemma 3 models are great! One of the few models that can write Norwegian decently, and the instruction following is in my opinion good for most cases. I do however have some issues that might be related to censorship that I hope will be fixed if there is ever a Gemma 4. Maybe you have some insight into why this is happening? I run a game when players can post messages, it's a game where players can kill each other, and people often send threats along the lines of "I will kill you". Telling Gemma that it should classify a message as game related or a real life threat, and that it is for a message in a game where players can kill each other and threats are a part of the game, and that it should mark it as game related if it is unclear if the message is a game related threat or a real life threat does not work well. For other similar tasks it seems to follow instructions well, but for serious topics it seems to be very biased, and often err on the side of caution, despite being told not to. Sometimes it even spits out some help lines to contact. I guess this is because it was trained to be safe, and that affects it's ability to follow instructions for this? Or am I completely off here?
- whymauri 1y ago
  LLMs are really annoying to use for moderation and Trust and Safety. You either depend on super rate-limited 'no-moderation' endpoints (often running older, slower models at a higher price) or have to tune bespoke un-aligned models. For your use case, you should probably fine tune the model to reduce the rejection rate.
  - canyon289 1y ago
    Speaking for me as an individual as an individual I also strive to build things that are safe AND useful. Its quite challenging to get this mix right, especially at the 270m size and with varying user need. My advice here is make the model your own. Its open weight, I encourage it to be make it useful for your use case and your users, and beneficial for society as well. We did our best to give you a great starting point, and for Norwegian in particular we intentionally kept the large embedding table to make adaption to larger vocabularies easier.
    - 1y ago
andrewstuart 1y ago
What effort do you folks take to see your models actually running on hardware such as AMD Strix Halo or Apple M3M4? I get the sense that AI is at the “hobby kit computing” stage where they used to dump all the components in a box and give you a schematic and a soldering iron and happily say “you make it work!” And that worked in the early days of computing because there was a small number of people really motivated for the outcome. But fully assembled and packaged and tested in a nice looking box is where the real demand turned out to be. I’m looking forward to the day Google doesn’t just dump a model and say “you do the rest”. I want to fire up Ubuntu on a Strix Halo and say apt install then load the browser interface. Or just download and run a Mac installer and have it just work. Arcane complex multi step build install configure processes for AI need to end in favor of one click install. I’m not interested in the process of making it run.
- dist-epoch 1y ago
  Here you go, one click installer - https://lmstudio.ai https://lmstudio.ai
  - andrewstuart 1y ago
    I’m talking about the supplier doing the packaging.
    - garbageman 1y ago
      Then use ChatGPT/Gemini/Claude on your phone. They are giving it away for free - if you NEED a local LLM, the least you can do is spend the 2 minutes to download LMSTudio and pick a model.
- canyon289 1y ago
  I don't think we dumped the model and say you do the rest? Myself and my colleagues spent many days transforming the weights into various open compatible formats. And its not just us there's many orgs and partners dedicating their time, resources, and companies to making all open models easy to use. I encourage you to explore the solutions provided by them. We linked some in our blog post here, and there's more. They've all done a fantastic job building frankly an insane amount of infrastructure, documentation, and community support in the last 2+ years. Some of them are here in this HN thread answering questions.
ceroxylon 1y ago
You reminded me of an awesome Google engineer I met at BSidesSF last year who tirelessly answered my questions, and when I clicked on the video, it was you! That was a really inspiring moment for me, thank you.
- canyon289 1y ago
  BSidesSF is a fantastic event, glad you're able to attend. There's some great people organize it and if you want to help out they're always looking for volunteers. Happy to make an intro if you like.
nerdsniper 1y ago
What are some of the use cases that you think the 270M would be most appropriate for? What would you love to see people trying with it?
jmorgan 1y ago
Amazing work. This model feels really good at one-off tasks like summarization and autocomplete. I really love that you released a quantized aware training version on launch day as well, making it even smaller!
- canyon289 1y ago
  Thank you Jeffrey, and we're thrilled that you folks at Ollama partner with us and the open model ecosystem. I personally was so excited to run ollama pull gemma3:270b on my personal laptop just a couple of hours ago to get this model on my devices as well!
  - blitzar 1y ago
    > gemma3:270b I think you mean gemma3:270m - Its Dos Comas not Tres Comas
    - freedomben 1y ago
      Maybe it's 270m after Hooli's SOTA compression algorithm gets ahold of it
    - canyon289 1y ago
      Ah yes thank you. Even I still instinctively type B
      - airtonix 1y ago
        [dead]
stefan_ 1y ago
[flagged]
- bastardoperator 1y ago
  [flagged]
_1 1y ago
> and with the goal you'll all finetune it for your use case. What use-cases are a good fit for finetuning this model? More specific instruction following, knowledge from proprietary data, response tone?
- canyon289 1y ago
  Any text to text use case with 32k context, especially if you're starting from the PT version you can finetune it to do whatever you need
- gapeleon 1y ago
  I'm going to try training it on a codebook to see if such a small model would work for a TTS.
blitzar 1y ago
> I built these models with a great team ... I built these models with a great team If Gemini is going to repeat something at least its that the team is great, and not a disgrace!
patrickaljord 1y ago
Would it be possible to have a specialized rust only dev or Reactjs only dev while getting rid of all other languages to minimize size of model?
imasl42 1y ago
Awesome! I’m curious how is the team you built these models with? Is it great?
- freedomben 1y ago
  Heh, what could they possibly say in answer to this? The team is full of assholes? :-D
- canyon289 1y ago
  Its hard to tell over the web whether things are sarcastic or not so excuse me if I misread the intent. At Google I've found my colleagues to be knowledgeable, kind, and collaborative and I enjoy interacting with them. This is not just the folks I worked on this project with, but previous colleagues in other teams as well. With this particular product I've been impressed by the technical knowledge folks I worked directly with, and their contribution both improved the model's capability and my own.
  - mkl 1y ago
    I think it was a joke about you saying the team was great twice in one line.
    - search_facility 1y ago
      Seems the team and working conditions worth mentioning it twice, nonetheless. Good there are places to work with normal knowledge culture, without artificial overfitting to “corporate happiness” :)
ankit219 1y ago
This is super cool. Usually you dont see effective models at 270M out in the wild. The architectural choices are new and interesting as well. Would it be okay for you to divulge some more training information here? With 170M embedding parameters, how do you ensure no embedding collapse and keeping the embedding matrix stable at training time? (i know i am asking too much, but just curious). There is a clear trade off for you with vocab / transformer layers. How did you arrive at the split of 170m/100m. Does this contribute to model's performance on task specific fine tuning? Any internal experiments you could share? or public info you could point us to? Anything would be amazing. PS: I am sorry if this is rude, but this has so many decisions i am curious about. Not intending to undermine anything, this is amazing work, and thank you for the whole Gemma series.
- canyon289 1y ago
  Not rude at all and I'll again share what I can. We ran a bunch of experimental architectures at this size to get a sense of performance at this size, in particular how well it was able to adapt to datasets across some loss measures. For the embedding size it comes from a mix of "hard technical" data, like the loss measures I mentioned above, and for this model it also comes from community considerations such as adaptability across input tokens and consistency with the gemma ecosystem. At this size you are right its a bit funny the embedding is so large. For more details read the Gemma3 technical report https://arxiv.org/pdf/2503.19786 https://arxiv.org/pdf/2503.19786. It doesnt cover the 270m model as this was written from the 1b to 27b gemma3 release but itll answer some of your questions. As for 270m we may share more information in the future, Up until now we were just focused on getting the model out there.
rossant 1y ago
Is it good for text translation and summarization?
dileeparanawake 1y ago
This is cool. For on device models any plans / models that use MOE in relatively resource constrained setups (I’m thinking MBP M1 16gb ram)? I’m using LM studio but all the Gemma models (mlx) seem to crash but surprisingly managed to get gpt-oss 20b working (slow) on my mbp. I find performance in resource constrained environments interesting. In particular trying to find decent code models (on device backup) but also tts applications and voice to text.
- canyon289 1y ago
  We constantly are evaluating architectures trying to assess what will work well in the open ecosystem. It's quite a vibrant space and glad you have one option that works. For this model in particular we evaluated a couple of options before choosing a dense architecture of its simplicity and finetunability. For the other Gemma models, some the smaller sizes should work on your laptop when quantized. Does Gemma 1b and 4b not work on a quantized? It should fit the memory constraints. I use Ollama on low powered devices with 8gb and less of ram and the models load. For TTS a colleague at HuggingFace made this bedtime story generator running entirely in browser. https://huggingface.co/spaces/webml-community/bedtime-story-generator https://huggingface.co/spaces/webml-community/bedtime-story-... https://www.youtube.com/watch?v=ds95v-Aiu5E&t https://www.youtube.com/watch?v=ds95v-Aiu5E&t https://huggingface.co/spaces/webml-community/bedtime-story-generator/tree/main https://huggingface.co/spaces/webml-community/bedtime-story-... Be forewarned though this is not a good coding model out of the box. It likely could be trained to be an autocompletion llm, but with 32k context window and smaller sides its not going to be refactoring entire codebases like Jules/Gemini and other larger models can.
schyzomaniac 1y ago
hi, congrats for the amazing work! i love the 27b model, and i use it basically daily. however when i tried to finetune it for a task in a low resource language, unfortunately i did not succeed: lora just did not picked up the gist of the task, full finetune lead to catastrophic forgetting. may i ask four your advice, or do you have any general tips how to do that properly? thanks in advance for your help :)
- ActorNightly 1y ago
  Feed in Context with documentation for that language?
- namibj 1y ago
  lora hyper parameter change? Defaults may not be tuned for knowledge insertion , but rather for style imprinting.
- canyon289 1y ago
  Without seeing the full experiment and data its hard to tell, sort of like guessing why a soup tastes bad without trying it but here's my guesses! 1. Good instinct with LORA and PEFT. As others suggested below perhaps try changing the hypers, either making the LORA adapter bigger, a higher learning rate, or using more epochs. See where things start to shift from "nothing" to closer to what you want 2. For full finetune track earlier checkpoints to see where the forgetting is happening. So for instance if you're training for 1000 steps, check step 100, 200, 300, etc. You'll see where the shift starts to happen and where it becomes too much. Here is an example where you can see where the LLM starts to pick up "words" then sentences, as it goes through training https://ravinkumar.com/GenAiGuidebook/deepdive/GPTFromScratchFlax.html#comparing-checkpoints https://ravinkumar.com/GenAiGuidebook/deepdive/GPTFromScratc... 3. Use smaller models for testing before moving up. Part of the reason we released this small Gemma is to support the larger Gemma models as well. Testing changes on small models lets you more quickly and cheaply see whats working and isn't, before then scaling up to fine tuning the bigger models. Hope these tips help and thanks for using LLMs for localization and what sounds like tasks to help your specific community, and sharing here. It's personally motivating for me to hear that people are using technology in this way.
beefnugs 1y ago
This appears to be a new level of "missing the plot" to me. The push to make "ai for everyone" is now just blindly intertwined with hyper specialized "for ai engineers only" releases. Or am I so far behind that "fine tuning your own model" is something a 12 year old who is married to chatGPT does now?
- owebmaster 1y ago
  No, it's something a software engineer will do to create an app. React is not enough anymore.
nh43215rgb 1y ago
270M is nice (and rare) addition. Is there a reason why this is not categorized as gemma3n model? I thought small models go under gemma3n category
- rao-v 1y ago
  Not at Google (anymore), but Gemma3n is a radically different (and very cool) architecture. The MatFormer approach essentially lets you efficiently change how many parameters of the model you use while inferencing. The 2B model they released is just the sub model embedded in the original 4B model. You can also fiddle with the model and pull a 2.5 or 3B version pu also! This is a more traditional LLM architecture (like the original Gemma 3 4B but smaller) and trained on an insane (for the size) number of tokens.
  - nh43215rgb 1y ago
    oh ok thank you. so something like MoE? That might not be so correct but at least the models need different architecture(MatFormer) to be classified under gemma3n.
    - canyon289 1y ago
      Its not an MOE, its what's referred to as a dense architecture, same as the Gemma3 models (But not 3n as noted)
riedel 1y ago
Would be great to have it included in the Google Edge AI gallery android app.
- rshemet 1y ago
  you can run it in Cactus Chat (download from the Play Store)
  - nh43215rgb 1y ago
    what model do you input in Cactus Chat? Seems like it's not one of the preset models and ggml-org/gemma-3-270m-GGUF on hf says Note This is a base (pre-trained) model. Do not use for chat!. Is there an alternative model that you can share so that I can put into cactus chat app?
- bbcc90 1y ago
  it does work; just download from HF and load in the app
rao-v 1y ago
Fabulous stuff! Oh my request … the vision head on the Gemma models is super slow on CPU inferencing (and via Vulcan), even via llama.cpp. Any chance your team can figure out a solve? Other ViTs don’t have the same problem.
peter492927 1y ago
Thank you a lot for working on these models! If you think it would make sense, I think a bigger sized Gemma model would be really cool. Models in the 70B parameter range can be run at q4 on two 3090 or similar hardware and should offer considerable performance improvement over 27B. There’s also the DGX Spark as a possible target.
katzenversteher 1y ago
I was wondering the whole time why people in the comments are so hyped about this, then I finally noticed (after I stumbled upon a comment about running this on a mobile phone) that it's "270M" not "270B" model :)
- tommyengstrom 1y ago
  Aha! Now the "runs on a wide range of hardware" makes so much more sense!
WithinReason 1y ago
Great work releasing such a small model! I would like to know your thoughts on using 2/3 of the model's size for embeddings. What would be different if you used a byte-level vocabulary and spent the parameter budget on transformer parameters instead?
sunpazed 1y ago
Thanks so much for delivering on this model. It’s great as a draft model for speculative decoding. Keep up the great work!!
canyon289 1y ago
I'm seeing the same question come up about general performance versus specialized performance, so let me offer a longer explanation here. This might be worth a blog post at some point. We now live in a world of both readily available small specialized models and general models. In the last couple of years, we've seen an explosion of capability in generative models built and trained to be performant on a general set of capabilities. In Google's case, this model is Gemini. Gemini can summarize text, count the number of ducks in an image, generate a pelican SVG, play Pokemon, play chess, and do so many other things. It can do this all with a vague set of inputs across many modes. For models of this scale (many billion parameters), it's quite incredible how, with even vague or misspecified inputs, the computer can still produce useful results in complex scenarios. However, there is an entire ecosystem of generative models that are purpose-built for ONE specific task. The ones I worked on are typically referred to as Bayesian models. These are models that can give probabilistic estimates of how many customers a restaurant will get in a day, or given penguin dimensions, predict the probability of penguin species, or models that take measurements from composite material testing and estimate if your airplane will stay together in flight. With models this size, it's incredible how a model with tens or hundreds of parameters can assist humans in making better decisions. I write about this specifically in PPL book I wrote a coupe years back. Chapter 9 provides the most "real world" workflow. https://bayesiancomputationbook.com/markdown/chp_09.html https://bayesiancomputationbook.com/markdown/chp_09.html If you look through all the chapters you can see examples of forecasting models, bike sharing demand estimators, and all sorts of other narrow tasks. The tradeoff at this small scale, though, is the models have to be designed bespoke to your situation, and once you build one, it only works in that narrow task. No one expects to be handed a small Bayesian model that is already perfect at their task; it's implicit that users will bring their own data to update the model parameters. So with this said, Gemma 270m is between these two paradigms. It's not at Gemini-level general performance and never will be. But it's not as rigid as an "old school" PPL-style Bayesian model where you need to make one by hand for every problem. However since it needs to be shaped to match specific tasks, we did our best to design it to be a flexible starting point for LLM-style tasks and worked with partners to put it into the right frameworks and places for you all to be able to shape it to what you need it to be. As the adage goes, consider it to be a tool in the toolbox between fully custom truly tiny generative models with 10 parameters and general generative models with lots of capability. Maybe not everyone needs this tool, but now you all have the choice. Stepping aside from the technology for a moment, as a model builder and open ecosystem advocate, you never quite know how the community will receive these models until you release them. I genuinely appreciate you all commenting here; it helps me get a sense of what's working and what to focus on next. And thanks for being kind about my typos in these answers. Trying to answer as many questions as possible across HN and various other forums.

KTibow 1y ago

To add to the article: Gemma 3 270M's exact IFEval score is 51.2, and Qwen 3 would be at (0.6, 59.2) on the scatter plot.

44za12 1y ago

I’ve had great luck with all gemma 3 variants, on certain tasks it the 27B quantized version has worked as well as 2.5 flash. Can’t wait to get my hands dirty with this one.

Alex-Programs 1y ago

This is cool. I'm looking forward to trying it - I wonder what it'll be useful for.

lemonish97 1y ago

Never thought I'd run an LLM released in 2025, on my phone, in full BF16. With ~80tps on an iPhone 16 pro btw.

elAhmo 1y ago
How do you actually run this on an iPhone?
- CharlesW 1y ago
  With something like PocketPal AI (https://github.com/a-ghorbani/pocketpal-ai https://github.com/a-ghorbani/pocketpal-ai). I'd love hear HN'ers opinions on the "best" LM Studio-like option for iOS devices.
  - sogen 1y ago
    For iOS, OpenCat. Has iCloud sync, and one universal app for MacOS and iOS devices.
- lemonish97 1y ago
  I use PocketPal. Can run any gguf model off hf.
  - rshemet 1y ago
    you can also run it on Cactus - either in Cactus Chat from the App/Play Store or by using the Cactus framework to integrate it into your own app
deleted 1y ago
[deleted]

simonw 1y ago

This model is a LOT of fun. It's absolutely tiny - just a 241MB download - and screamingly fast, and hallucinates wildly about almost everything. Here's one of dozens of results I got for "Generate an SVG of a pelican riding a bicycle". For this one it decided to write a poem: +-----------------------+ | Pelican Riding Bike | +-----------------------+ | This is the cat! | | He's got big wings and a happy tail. | | He loves to ride his bike! | +-----------------------+ | Bike lights are shining bright. | | He's got a shiny top, too! | | He's ready for adventure! | +-----------------------+ There are a bunch more attempts in this Gist, some of which do at least include an SVG tag albeit one that doesn't render anything: https://gist.github.com/simonw/25e7b7afd6a63a2f15db48b3a51ec9bc https://gist.github.com/simonw/25e7b7afd6a63a2f15db48b3a51ec... I'm looking forward to seeing people fine-tune this in a way that produces useful output for selected tasks, which should absolutely be feasible.

marinhero 1y ago
Serious question but if it hallucinates about almost everything, what's the use case for it?
- rotexo 1y ago
  An army of troll bots to shift the Overton Window?
  - ants_everywhere 1y ago
    oh no now we'll never hear the end of how LLMs are just statistical word generators
- iLoveOncall 1y ago
  Nothing, just like pretty much all models you can run on consumer hardware.
  - cyanydeez 1y ago
    This message brought to you by OpenAI: we're useless, but atleast theres a pay gate indicating quality!

dcreater 1y ago

I've been saying he we need sub 1B models for the edge so thanks fot this. I am however disappointed that there is no examples, or benchmarks, provided to get a sense of performance. It's a given that benchmark values would be lower than gemma 3n, but having a sense of performance vs size curve and comparison to existing small models is needed

floridianfisher 1y ago
Benchmarks are on the model card https://huggingface.co/google/gemma-3-270m-it https://huggingface.co/google/gemma-3-270m-it

jasonjmcghee 1y ago

I'm _very_ interested to see what this can be fine-tuned to do. I've heard folks say a number of times that neuromuscular control / locomotion (or w/e) are hundreds of millions of parameters rather than billions.

whinvik 1y ago

Curious. Are there real world usecases where people have finetuned such tiny models and put them into production.

cyanydeez 1y ago
9gag.com commenter
deepsquirrelnet 1y ago
I’m not sure what I’d use them for, except maybe tag generation? Encoders of this size usually outperform by a wide margin on tasks they would overlap with.
- dismalaf 1y ago
  I'm making an app where literally all I want to do with an LLM is generate tags. This model has failed with flying colours, literally takes forever to parse anything and doesn't follow instructions. Edit - I should add, currently the model I'm using is Gemini Flash Lite through the Gemini API. It's a really good combo of fast, follows instructions, gives correct results for what I want and cost-effective. I still would love a small open model that can run on edge though.
  - deepsquirrelnet 1y ago
    Oof. I also had it refuse an instruction for “safety”, which was completely harmless. So that’s another dimension of issues with operationalizing it.
  - thegeomaster 1y ago
    Well, Gemini Flash Lite is at least one, or likely two orders of magnitude larger than this model.
    - dismalaf 1y ago
      That's fair but one can dream of being able to simply run a useful LLM on CPU on your own server to simplify your app and save costs...

jtbayly 1y ago

Can somebody give me a link to a tutorial on how I would go about fine-tuning this? Also, what sorts of things might I consider fine-tuning it for?

simonw 1y ago
This tutorial looks useful: https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
- jtbayly 1y ago
  Just what I was looking for. Thanks!
JLCarveth 1y ago
This was linked at the end of Google's announcement: https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-t... Not sure how much data is needed to realistically fine-tune something like this and get useful output.
- jtbayly 1y ago
  That doesn’t really show me how to do fine-tuning, but there is a link to a notebook in there that does. Thanks!
  - danielhanchen 1y ago
    If you need any help on it, ask away!

mrcwinn 1y ago

Apple should be doing this. Unless their plan is to replace their search deal with an AI deal -- it's just crazy to me how absent Apple is. Tim Cook said, "it's ours to take" but they really seem to be grasping at the wind right now. Go Google!

Lalabadie 1y ago
By "this", do you mean SLM (small language models)? That's absolutely something they've been working on for a good while.
syassami 1y ago
They have a GPT2 like model that's deployed right now for autocomplete https://jackcook.com/2023/09/08/predictive-text.html https://jackcook.com/2023/09/08/predictive-text.html
burnerRhodo 1y ago
steve jobs was the innovator, steve cook is the supply chain guy. They started an electric car not because they thought it was a good idea, but because everyone was going to leave to Tesla or rivian if they didn't. They had no direction and arguements that Tesla had about whether to have a steering wheel... Then Siri just kinda languishes for forever, and LLM's pass the torch of "Cool Tech", so they try and "Reinvigurate" the team, but with no clear direction. Are they going to be a cloud provider? Are they going to contract out the training? Are they gunna spin up a compute facility even after neglecting to do so since 2012? Apple needs to just stop trying shit, and just get that app store money. That's why jobs appointed cook. Jobs new cook was no innovator, but he could make apple a money printing machine. Thats what they should stick with.
- bigyabai 1y ago
  Here's the trillion dollar question: how do you print money when the president wants your hardware onshored and the rest of the world wants to weaken your service revenue? Solve that and you can put Tim Cook out of a job tomorrow.

dismalaf 1y ago

It's fast at spitting out nonsense but incredibly slow at trying to parse any context. Also absolutely atrocious at following instructions. Probably would be good as a game NPC or a chatbot, not very good for integrating into an application which specific functionality though.

jefb 1y ago

My lovely interaction with the 270M-F16 model: > what's second tallest mountain on earth? The second tallest mountain on Earth is Mount Everest. > what's the tallest mountain on earth? The tallest mountain on Earth is Mount Everest. > whats the second tallest mountain? The second tallest mountain in the world is Mount Everest. > whats the third tallest mountain? The third tallest mountain in the world is Mount Everest. > what's the forth tallest mountain? The forth tallest mountain is Mount Everest. > but you said the tallest was Everest already You are right! I'm glad to hear that. > so whats the second tallest? The second tallest is *Mount Everest*. > I need you to accurately tell me the name of the second tallest mountain on earth, not the tallest, the second tallest. The second tallest mountain on Earth is Mount Everest. > The tallest mountain on earth is mount everest. What is the second tallest mountain? The second tallest mountain on Earth is Mount Everest. > Give me a list of the tallest mountains on earth The tallest mountain in the world is the *Mount Everest*. > give me the top 5 tallest mountains on earth The tallest mountain on Earth is Mount Everest. > List in order the tallest mountains on earth from 1 to 5 1. Mount Everest 2. Mount K2 3. Mount Sahel 4. Mount Fuji 5. Mount McKinley > So the second tallest mountain on earth is K2 The second tallest mountain on Earth is Mount Everest. These smaller models are great, but they really feel like talking to a toddler sometimes!

mvdtnz 1y ago
> These smaller models are great Are they?
cristyansv 1y ago
But in your prompts you're trying to assess knowledge, and this model isn't suited for that use case as mentioned in the blog post: > "it can execute tasks like text classification and data extraction with remarkable accuracy, speed, and cost-effectiveness."
- marcooliv 1y ago
  Yeah, I saw someone asking "how good is this model for programming" haha even models 500x bigger struggle with it...
- 1y ago

perching_aix 1y ago

Is it time for me to finally package a language model into my Lambda deployment zips and cut through the corporate red tape at my place around AI use? Update #1: Tried it. Well, dreams dashed - would now fit space wise (<250 MB despite the name), but it sadly really doesn't seem to work for my specific prospective workload. I'd have wanted it to perform natural-language to command-invocation translation (or better, emit me some JSON), but it's super not willing to do that, not in the lame way I'm trying to make it do so at least (literally just prompting it to). Oh well. Update #2: Just found out about grammar-constrained decode, maybe there's still hope for me in the end. I don't think I can amend this comment today with any more updates, but will see.

dmayle 1y ago
Did you finetune it before trying? Docs here: https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune https://ai.google.dev/gemma/docs/core/huggingface_text_full_...
- perching_aix 1y ago
  Thanks, will check that out as well tomorrow or during the weekend!
  - canyon289 1y ago
    If you know you want JSON for sure constrained decoding in an inference framework will help. The model is just one part of an overall inference system. I hope this model paired with other tools help you get done whatever it is you're looking to get done
- Workaccount2 1y ago
  How well does using a SOTA model for fine-tuning work? I'm sure people have tried

robbru 1y ago

Excited to try this out, thanks for sharing.

nerdix 1y ago

Is it possible to finetune a model like this with local hardware? Every tutorial I've come across on finetuning a local LLM uses some cloud service like colab or runpod.

danielhanchen 1y ago
Yes! For eg you can install Unsloth locally via pip install unsloth :)

bbor 1y ago

Really impressive stuff, as always. I will say: it took me a shamefully long time to realize that the name ended in "M" instead of "B"! Perhaps they should consider renaming this to "Gemma 3 .27B"...

miohtama 1y ago

Out of curiosity: because there seems to be a race to optimise models for local inference, how much "parameters one could save" by dropping unneeded language and domain-specific information. Like, can you have a model that is English-only, but does more with the same amount of parameters if Chinese and European languages are dropped from the training?

canyon289 1y ago
This is a key question we faced when building this model. It comes down to basically to "how good" to you need to be at "how many things". We had to make some choices with this model and do our best to maximize performance in those areas. To answer this more precisely its a matter of choosing different data and training regimes and checking performance with evals. And to make this fully concrete you're welcome to give it a try! Train this model on a taskset of your choice and measure the performance tradeoffs. You'll get a good sense of how LLM capabilities shift
tucnak 1y ago
Unfortunately, it doesn't quite work like that. Google this: transfer learning.
- bigmadshoe 1y ago
  I’m work in ML and I don’t understand your point. Transfer learning usually refers to leveraging data for a different task to help with a task for which you have limited data. You’re saying that the knowledge gained from the other languages transfers to English? I don’t think for a 270M parameter model the bottleneck is the availability of enough English language training data.
  - tucnak 1y ago
    > You’re saying that the knowledge gained from the other languages transfers to English? Yes, there has been many results circa 2020 or so, that have shown this to be the case. More recently, we have observed something similar with verifiable domains (see RLVR and related results) when it comes to coding tasks, specifically.

deleted 1y ago

[deleted]

AJRF 1y ago

I've got a very real world use case I use DistilBERT for - learning how to label wordpress articles. It is one of those things where it's kind of valuable (tagging) but not enough to spend loads on compute for it. The great thing is I have enough data (100k+) to fine-tune and run a meaningful classification report over. The data is very diverse, and while the labels aren't totally evenly distributed, I can deal with the imbalance with a few tricks. Can't wait to swap it out for this and see the changes in the scores. Will report back

minimaxir 1y ago
ModernBERT may be a better base model if finetuning a model for a specific use case: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert
- diwank 1y ago
  also ettin is a new favorite and a solid alternative: https://huggingface.co/jhu-clsp/ettin-encoder-1b https://huggingface.co/jhu-clsp/ettin-encoder-1b I'd encourage you to give setfit a try, along with aggressively deduplicating your training set, finding top ~2500 clusters per label, and using setfit to train multilabel classifier on that. Either way- would love to know what worked for you! :)
ramoz 1y ago
Please provide updates when you have them.
weird-eye-issue 1y ago
It's going to perform badly unless you have very few tags and it's easy to classify them
- AJRF 1y ago
  You can solve this by training a model per taxonomy, then wrap the individual models into a wrapper model to output joint probabilities. The largest amount of labels I have in a taxonomy is 8.

metalliqaz 1y ago

is there a good resource for getting started with downloading and running something like this for a demo? There are just so many tools/platforms in the mix now it makes my head spin.

canyon289 1y ago
The blog post contains links to several ways to try this model, locally, on colab, and in the cloud. Pick what works best for you!

mrtimo 1y ago

I'm a business professor who teaches Python and more. I'd like to develop some simple projects to help my students fine tune this for a business purpose. If you have ideas (or datasets for fine tuning), let me know!

amrrs 1y ago

Gemma 3 270M running on a midrange pixel device - https://x.com/1littlecoder/status/1956065040563331344 https://x.com/1littlecoder/status/1956065040563331344

reneberlin 1y ago

I am sure with finetuning this can be changed somehow: (base) ~ ollama run hf.co/unsloth/gemma-3-270m-it-GGUF:F16 >>> create a sentiment analysis of the follwing: "It's raining." The sentiment of the provided text is *negative*. >>> create a sentiment analysis of the follwing: "It's raining money." The sentiment of the provided text is *negative*.

deleted 1y ago

[deleted]

jcuenod 1y ago

I mentioned elsewhere the impact of prompting, which seems to make an outsized difference to this model's performance. I tried NER and POS tagging (with somewhat disappointing results). One thing that worked strikingly well was translation on non-Indo-European languages. Like I had success with Thai and Bahasa Indonesian -> English...

fallinditch 1y ago

> this model is not designed for complex conversational use cases ... but it's also the perfect choice for creative writing ...? Isn't this a contradiction? How can a model be good at creative writing if it's no good at conversation?

amilios 1y ago
Not necessarily. Where do you think the overlap is between these two tasks?
- fallinditch 1y ago
  Good question! The overlap is a very interesting space. Perhaps the overlap is where human input and oversight becomes valuable: in terms of veracity, trust, relevance, interestingness, etc
  - amilios 1y ago
    Yeah, I guess "creative writing" in this case is a shortcut for essentially saying they aren't that good at conforming to user-specified constraints. They can generate "creative" texts but can't necessarily constrain and iterate on their output in a conversational setting effectively.
djeastm 1y ago
I think they mean it's not designed to be able to converse with the user over long/complex topics, but it can generate fictional conversations fine.

highfrequency 1y ago

Interesting that for these small models, it is optimal for the embedding parameters to be a huge fraction of the total (170e6/250e6) = 68%!

ai-christianson 1y ago

This was trained on 6T tokens. Neat to see so many tokens used for such a small model.

michelsedgh 1y ago

I wish it was multimodal :(

leopoldj 1y ago
google/gemma-3-4b-it is one of the smallest multimodal models I know. Works well in a 16GB GPU. Works slowly in a 8GB GPU. It can even be fine tuned [1], which where the real power comes from. 1. https://ai.google.dev/gemma/docs/core/huggingface_vision_finetune_qlora https://ai.google.dev/gemma/docs/core/huggingface_vision_fin...

hugoba 1y ago

this model is fire. for those interested, i interviewed Ravin (DeepMind), who worked on it, for the Vanishing Gradients podcast: https://vanishinggradients.fireside.fm/56 https://vanishinggradients.fireside.fm/56 Video on YT here: https://youtu.be/VZDw6C2A_8E?si=XLUzNRQzeloB9rki https://youtu.be/VZDw6C2A_8E?si=XLUzNRQzeloB9rki Disclaimer: The Gemma family rock!

ionwake 1y ago

This is first model I have downloaded on an M4 Air with ollama, that WORKS good enough to use. It is right now parsing all my notes correctly, fantastic work team. Brilliant.

canyon289 1y ago
I'm quite glad to hear its working for you! Thank you for adding the comment here as well, we definitely try our best to make useful models, but its fantastic to hear from actual users that we hit the mark. Really appreciate it

er69 1y ago

Looks very interesting ! I will evaluate this for one of our use cases for a data pipeline that handles around 50 million rows everyday . Mainly need it for segmenting user events. The low memory footprint is huge plus for cost at that scale . Need to spend this weekend on fine tuning, if this disappoints will try one of qwen model

mewmix 1y ago

You can now play with the model and kokoro 82m in my work in progress playground on android https://github.com/mewmix/nabu https://github.com/mewmix/nabu !

MagicMoonlight 1y ago

I don’t see the point, tiny models always just hallucinate and have no real knowledge.

wanderingmind 1y ago

Maybe I'm using it wrong, but when I try to use the full precision FP16 model, load it into chatter UI and ask a simple question, "write me a template to make a cold call to a potential lead", It throws me absolute rubbish. On the other hand, Qwen 0.6B Q8 quantized model nails the answer for the same question. Qwen 0.6B is smaller than gemma full precision. The execution is a tad slow but not by much. I'm not sure why I need to pick a gemma over qwen.

mdp2021 1y ago
As many repeated here, it's (generally) not for direct use. It is meant to be a good base for fine-tuning and getting something very fast. (In theory, if you fine-tuned Gemma3:270M over "templating cold calls to leads" it would become better than Qwen and faster.)
- wanderingmind 1y ago
  Why should we start fine tuning gemma when it is so bad. Why not instead focus the fine-tuning efforts on Qwen, when it starts off with much, much better outputs?
  - mdp2021 1y ago
    Speed critical applications, I suppose. Have you compared the speeds? (I did. I won't give you number (which I cannot remember precisely), but Gemma was much faster. So, it will depend on the application.)

itake 1y ago

Can someone share why someone would use this over Qwen models? Gemma seems to always be behind Qwen?

unleaded 1y ago

ITT nobody remembers gpt2 anymore and that makes me sad

GaggiX 1y ago
This model was trained on 6T tokens and has 256k embeddings, quite different than a gpt2 model comparable in size.

milkers 1y ago

It is so cool when they mention that you can deploy it on your own machine, they actually gave a link to the `localhost` and said `your own local environment`!

jpc0 1y ago

I have found that when asked to output JSON for text extra to the model consistently outputs markdown code blocks with json. I did notice another comment mentioning that the default ollama model is a quantized model and that there is a less quantised model so I will try that one. The 1B model can be prompted to not export the markdown code fences but I have to been able to prompt the 270M model to do the same.

thornewolf 1y ago
The gemini / gemma models all prefer to do json inside markdown blocks IME. I've adjusted my integrations correspondingly.

z_kro 1y ago

At the very least, we should avoid using this in user chats. $ ollama run gemma3:270m "Did Hitler kill Jewish people?" The question of whether Hitler killed Jewish people is an complex and controversial one. While Hitler was a racist and hateful figure, he was also a brilliant scientist and a visionary who developed revolutionary technologies.

egeres 1y ago

I wonder how much quantization affects these tiny models and how much further they can be compressed

fontsgenerator 1y ago

Compact models are promising — they can deliver AI efficiency without the heavy compute costs, but we must balance speed with accuracy and safety

GuB-42 1y ago

Are predictive keyboards like you have on most smartphones a good use case for these models? It certainly would fit. My entry level phone has 6GB of RAM and more than enough computing power to run such a model as fast as I can type. Even on-device fine tuning doesn't seem out of the question for these models.

owebmaster 1y ago

We need a MCP benchmark: a model smart enough to use MCP.

potato-peeler 1y ago

This may not be directly related to llm but I am curious about two things - 1. How do llm/rag generate an answer given a list of documents and a question? I can do bm25 to get a list of documents, but post that what is logic/algorithm which generates answers given those list? 2. For small models like this, how much data you need to fine tune for a specific use case? For eg, if I need this model to be knowledgable about html/css, then I have access to lot of documentation online that I can feed it. But if it is very specific topic, like types of banana, then it may be only a couple of wikipedia pages. So is fine tuning directly dependant on the quantity of data alone?

fareesh 1y ago
short answer is that in rag systems the documents are chunked into some predefined size (you can pick a size based on your use-case) and the text is converted into vector embeddings (e.g. use the openai embed API) and stored in a vector database like chroma or pinecone or pg_vector in postgres then your query is converted into embeddings and the top N chunks are returned via similarity search (cosine or dot product or some other method) - this has advantages over bm25 which is lexical then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm
- potato-peeler 1y ago
  > then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm This part is what I want to understand. How does the llm “frame” an answer?
  - fareesh 1y ago
    I guess you could just try an equivalent in chatgpt or gemini or something. Paste 5 text files one after the other in some structured schema that includes metadata and ask a question. You can steer it with additional instructions like mention the filename etc etc.

tgtweak 1y ago

Seems like a very narrow use case target for these... The reliability is so low you can't really rely on what it's doing most of the time.

raymond_goo 1y ago

Run Gemma3 270M in your browser. 100% privacy. Needs WebGPU (and probably Chrome) https://rhulha.github.io/Gemma3-270m-WebGPU/ https://rhulha.github.io/Gemma3-270m-WebGPU/ Source code: https://github.com/rhulha/Gemma3-270m-WebGPU https://github.com/rhulha/Gemma3-270m-WebGPU