14 ms·
Gemma 4 12B: A unified, encoder-free multimodal model
- nickandbro 5mo agoWow Google is becoming the new pre Llama 4 Meta when it comes to releasing open weights models.
- embedding-shape 5mo agoI dunno, feels a bit unfair to companies that actually do FOSS releases (Gemma 4 being released under Apache 2.0 license) to compare them to a company that never done any FOSS releases, and mostly done proprietary "available to download" releases.
- seba_dos1 5mo agoNote that a binary released under Apache 2.0 license does not yet make it FOSS.
- embedding-shape 5mo agoAgreed, miles ahead though from "proprietary" which is what Meta been using for most model releases. Ideally companies would share the fucking datasets and training code already, but no, no one wants to talk about the source of those or even share the ones they have as then who knows what comes out of Pandora's box...
- jimmy76615 5mo agoNVIDIA does a pretty good job on that front.
- redman25 5mo agoIDK this model release is a bit disappointing considering the community has been chomping at the bit for the 124ba4b model. There was some leaked info about it but people suspect it was not released because it was too close to gemini flash in performance.
- brianwawok 5mo agoEvery other Google model I have tried felt very weak compared to qwen models. I dont have a ton of use case for multimodal though, so its very possible this is a fantastic multimodal model.
- wongarsu 5mo agoGemma 4 27b and 32b feel pretty capable for text and visionn. Comparable with qwen, maybe a bit better on tool calling heavy tasks I am not overly impressed with the smaller gemma models. And gemma 3 was a bit of a mixed bag, great at some things, bad at most others
- verdverm 5mo agoqwen3.6 was my favorite, then I tried the deepseek-v4-{flash,pro} still making my way through deep dives on the chinese open weights, they are all pretty good and way more cost / resource effective
- thot_experiment 5mo agoHard disagree, Qwen multimodal is way better than google's, but Gemma 31b runs laps around Qwen 27B in complex engineering tasks. Maybe Qwen is better at slopcoding web framework CRUD, but for embedded dev there's no comparison.
- avadodin 5mo agoE4B is decent at instruction following. It managed to produce a deliverable on par with the lowest tier of paid models. Even higher tiers often just ignore all rules when they feel like it. I wish it was an 8BA1B MoE model with the newer acceleration 1B or maybe even a tailor-made sub-1B slapped on top. That would make it an awesome local model for the average laptop.
- kordlessagain 5mo agoCool!
- minimaxir 5mo agoThe big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is robust enough. https://developers.googleblog.com/gemma-4-12b-the-developer-guide/ https://developers.googleblog.com/gemma-4-12b-the-developer-... > Small enough to run locally on consumer laptops with 16GB of RAM, it unlocks powerful multimodal and agentic experiences right on your machine. I am assuming that involves quantization, which due to the quality loss makes that statement somewhat misleading IMO.
- GaggiX 5mo ago> That's technically encoding Isn't that just projecting the patches into the d_model size vectors that the models takes? >I am assuming that involves of quantization 12B model in 16GB seems very reasonable to me, int8 is top quality for running models.
- minimaxir 5mo agoThe guide describes it as projection although there is apparently an extra step: "A factorized coordinate lookup (X and Y matrices) attaches spatial location information directly to the input." 12B at int8 would take up 12G memory, or 75% of the system memory which technically fits within 16GB but the OS will not like that. EDIT: On my 18G memory MacBook Pro, LM Studio reports a "partial GPU offload" for the int8 MLX weights. Can't test because the `gemma_unified" architecture is NYI.
- WhitneyLand 5mo agoYeah and it’s pretty memory efficient with only 8 attention layers so at int8 in 16GB ram maybe you still get 64k-128k context. The part I hate though is that I’d bet none of the performance claims are based on int8. Why do we care about bf16 benchmarks when no one will be using that with this model.
- ethanpil 5mo agoWhat's Google's business case for releasing open models? Don't get me wrong, I am grateful and appreciative of these releases. I'm trying to understand how it fits into their bigger picture as a for profit company? Are they not helping competitors build on the novel technology they have developed? Is it simply goodwill and/or marketing? Or am I missing something strategic?
- mmarian 5mo agoMarketing + Pro Serv if I had to take a guess.
- XzAeRosho 5mo agoGoogle's MO since always has been to release great products or services for free, position themselves high and then abandon them or just find uses for Enterprise sales. I'm pretty sure they are doing it because they get some research experience by shrinking and improving these models, and because they know that by doing this they get some good PR among the dev community.
- Aachen 5mo agoGoogle's "free" is and was ad-supported, even if some products now have a paid tier. These models don't include ads. Doesn't seem like the same underlying reason
- theturtletalks 5mo agoMaybe they are hedging against a future where local models are just as good as cloud models? Or maybe they can go the Taalas route and start hardcoding Gemma on a chip and hardware manufacturers can use it for local private AI.
- onlyrealcuzzo 5mo agoIf you're an AI lab, you definitely want research teams in this space - as this is where you can most easily iterate and make improvements which you'll then bake into larger, frontier models. The question is: do you want to release your models, or use them purely for R&D? Since everyone else is already releasing models of similar qualities, it's hard to say you're shooting yourself in the foot if you join the chorus. The added cannibalization of releasing them is effectively zero, so the reputational benefits are likely to be worth it.
- deleted 5mo ago[deleted]
- zuminator 5mo agoHow does it compare with e4b, aside from being larger?
- thomasjb 5mo agoThat's what I want to know too. A smarter E4B that's happy in opencode would be a good selfhosted model for me
- anonova 5mo agoThere's a comparison of all the Gemma 4 models (+ Gemma 3 27B) on the Huggingface model card: https://huggingface.co/google/gemma-4-12B-it#benchmark-results https://huggingface.co/google/gemma-4-12B-it#benchmark-resul...
- dwa3592 5mo agoThis is a pretty good update. The demo video is a bit funny though - the tester asks to turn the release into bullet points. okay, the model obliges. then the tester says draft an email with this content. BAM! the LLM turns the content from bullets to passages even though it was not asked and it undid the last good thing that it did. i am not sure if it's an email etiquette to not put bullets in the email.
- Zambyte 5mo agoIs this Mac only? Or is that an Ollama issue that it only supports this release of models on Mac? It seems like every tag with the MLX badge is only supported on Mac[0], and that includes all of the tags in this release. [0] https://ollama.com/library/gemma4/tags https://ollama.com/library/gemma4/tags Edit: MLX being Mac-only is independent of the model being MLX (and therefore Mac) only. The latter is what I am asking about.
- embedding-shape 5mo agoMLX is quite literally macOS-specific technology, for other platforms you want non-MLX. I was sure "MLX" stood for "Metal-something-something" but can't find any reference to that somehow, anywho, "Metal" is hardware-accelerated graphics on Apple platforms FWIW. Edit: about the actual release on Ollama, if you're on non-Apple hardware you probably want the NVFP4 variant ("gemma4:12b-nvfp4") which was uploaded 45 minutes ago, especially if you're with a recent nvidia GPU.
- sambaumann 5mo agoI still get "this model requires macOS" when trying to pull that one
- embedding-shape 5mo agoI don't use Ollama myself anymore, but seems others been having similar issues for quite some time, maybe one of these fit your environment exactly? https://github.com/ollama/ollama/issues?q=is%3Aissue%20state%3Aopen%20%22this%20model%20requires%20macOS%22 https://github.com/ollama/ollama/issues?q=is%3Aissue%20state...
- Patrick_Devine 5mo agoI realize this is a little confusing; we're working w/ the MLX team to bring MLX to other platforms, but we're not quite there yet. The `gemma4:12b-nvfp4` model is specifically for the MLX engine. For the GGUF 4bit variant (i.e. non-macs) you'll need `gemma4:12b-it-q4_K_M` which I just pushed. You'll also need to upgrade to version 0.30.4 which we're just about to release (it's in prerelease and we're running through our last regression tests).
- randomNumber7 5mo ago> Novel unified architecture: No multimodal encoders. The vision and audio inputs flow directly into the LLM backbone. I would be interested in how this actually works. I couldn't find a description of the model architecture (and I did check the links in the Google blog)
- spott 5mo agohttps://newsletter.maartengrootendorst.com/p/a-visual-guide-to-gemma-4-12b https://newsletter.maartengrootendorst.com/p/a-visual-guide-... (in a link from here: https://developers.googleblog.com/gemma-4-12b-the-developer-guide/ https://developers.googleblog.com/gemma-4-12b-the-developer-..., which was linked in the text of the post, but not the linkdump at the end).
- toldnotmywrath 5mo agoMy understanding is that early (and most extant) visual language models have a component module (called the image encoder) that transforms images into representations (called embeddings) the model's inner layers can process. This is often a separate module grafted onto the main model, and further pre-trained (e.g. OpenAI's CLIP, SigLIP used in the Gemma 3 and PaliGemma series). The image encoder approach has a few problems. One problem is that many like Gemma 3's encoder have fixed image resolution constraints and inputs must be resized with all the attendant distortions that causes with spatial understanding. However, the Gemma 4 series image encoders overcame this and can handle variable-dimension inputs. Two, these image encoders are somewhat large (ranging from 300-500M parameters) requiring extra memory and FLOPs to run. Three, say we need to fine-tune a vision language model, updates to its weights, may affect its understanding of the representations generated by the image encoder if we don't fine-tune both together. The new Gemma-4-12B replaces the encoder (with its many attention layers and large parameter count) with a simple linear projection to generate the embeddings for images. That reduces the computational requirements and simplifies the input pipelines for image processing. I don't have any expertise on the topic though and might very well be wrong on some details.
- djyde 5mo agoWhat are the use cases for these small models? Is there anyone using models of this scale in their daily life who could share their experience?
- Xiol 5mo agoI've yet to see someone answer a question like this with a decent, useful answer.
- Aachen 5mo ago"Small" models are the ones I can run myself on my own terms. LLMs aren't useful enough for me to justify spending hundreds of euros on a GPU with 16GB VRAM or something, and that's assuming I have the rest of the desktop just laying around. Back when I checked (before the RAM price hike), these models weren't meaningfully better than 4-8GB ones anyway, you'd have to go for the top tier cards at 24 or 32 GB iirc to get something vaguely in the direction of the SaaS versions, and that was absolutely out of my budget. Even if that changed, so have hardware prices so it'd probably still work out the same
- properbrew 5mo agoI think small models have a very good niche for specific tasks. I utilise a fine tuned Phi-4 model (smaller than this one) that fits in about 3.5gb of RAM (not vram) for the document processing side of things for the desktop app I develop (a bit of a shameless plug - whistle-enterprise.com). If you have a very specific idea for local model use you can find a way to make it work very well, you don't even need to have a graphics card or NPU chip. You just have to be extremely constrained in how it's used. I think as a generic chatbot they're not great, I'd use a hosted SOTA model and I'm a big fan of local LLMs myself.
- SeriousM 5mo agoThank you for sharing your usecase! I like your product very much! Could you talk a bit how you did the finetuning? Did you use unsloth or any other tool and how went the verification to proof the outcome?
- jdelman 5mo agoI can’t help but wonder if this is the basis of the model they’ve helped tune for Apple.
- digdugdirk 5mo agoI do enjoy the immediate out of touch signaling with the "runs on your 16gb vram laptop" line. Because everyone has a laptop with 16gb vram, or can just pop out and buy a new one, right?
- deleted 5mo ago[deleted]
- vehemenz 5mo agoThis comment has me a bit confused. Consumers were complaining about the standard 8GB with the early 2020 refresh of MacBook Pros, many OSes ago. Sure, it might be workable for many tasks (as evidenced by the recent sales of the MacBook Neo), but users with a mere 8GB shouldn't have expectations of LLM performance. Even 16GB feels like a stretch.
- NekkoDroid 5mo agoI think you are mixing up RAM and VRAM.
- Schiendelman 5mo agoOn a Mac they are the same thing; they're shared. Of course you need some amount for the OS, but if you have an Apple Silicon Mac with 24GB of RAM, you can likely run a 16GB model.
- crims0n 5mo agoThey are effectively one and the same on Apple Silicon.
- NekkoDroid 5mo agoWhich most people as a matter of fact don't use. A majority of people with laptop have separate memory pools and the VRAM of them is nowhere near that and even on most gaming laptops you aren't getting 16GB VRAM.
- Havoc 5mo agoQuite a niche release. The MoE outperforms it on score and will likely be faster thanks to lower active weights. So this really only makes sense for specific ram constrained applications that can’t fit a quantized MoE
- dist-epoch 5mo agoThe un-quantized MoE outperforms it. But between same (V)RAM requirement 4 bit 26B-A3B and 8 bit 12B it's unclear which one will win, especially given one is MoE and the other dense. All the launch benchmarks are at 16 bit.
- ComputerGuru 5mo agoQuite aside from the architectural changes, I suppose this is the answer to why Google had such a glaring hole in the (pretrained) Gemma4 model lineup between the Gemma4 4b and Gemma4 26b models! A model that comfortably fits in 16GB of VRAM (allowing room for context) is a welcome upgrade.
- claysmithr 5mo agoI don’t see the download in lm studio
- deckar01 5mo agoIt also says it is supposed to be available in their own Edge Gallery app and it’s not there (on iOS).
- corgihamlet 5mo agohttps://lmstudio.ai/models/gemma-4 https://lmstudio.ai/models/gemma-4
- claysmithr 5mo agoThanks looks like they just added it 1 hour ago
- BiraIgnacio 5mo agousing an embedder instead of a decoder is quite clever. Not sure who came up with that first but it's a cool idea.
- lxgr 5mo agoAm I missing something or are the Ollama versions of this (https://ollama.com/library/gemma4/tags https://ollama.com/library/gemma4/tags) text-only for now?
- philipkglass 5mo agoSince ollama has diverged from llama.cpp, it will take a bit of time for ollama to support multi-modality. If you're using plain llama.cpp it looks like a PR has already merged for this model with vision and audio support: https://github.com/ggml-org/llama.cpp/pull/24077 https://github.com/ggml-org/llama.cpp/pull/24077
- zozbot234 5mo agoThey've actually gone back to (a lightly patched) llama.cpp with the 0.30 release a few weeks ago, and have now vendored-in an up to date release. Needless to say this is great news for both projects!
- satvikpendem 5mo agoJust use llama.cpp or Unsloth Studio which wraps it, I don't know why anyone use Ollama anymore.
- verdverm 5mo agoI switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746 https://github.com/ggml-org/llama.cpp/issues/22746
- Jabrov 5mo agoStop using ollama
- thot_experiment 5mo agoOllama is a shitty project that steals from the open source community, don't use it, use llama.cpp instead.
- deleted 5mo ago[deleted]
- mlmonkey 5mo agoIs there some place where we can try it before downloading the gigabytes of weights?
- spott 5mo agoIs there a paper on this? I'm curious how they pre-trained it... I feel like it must have had audio/image output that they chopped off. I wonder how hard it would be to add it back on.
- joaogui1 5mo agoI mean Claude is multimodal on input but not output, why couldn't this also be?
- zkmon 5mo agoI'm waiting for FP8 quant, preferably from Google.
- mdp2021 5mo agoIf you accept the "ggml-org": https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/tree/main https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/tree/mai... https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/blob/main/gemma-4-12B-it-Q8_0.gguf https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/blob/mai...
- zkmon 5mo agoDo they run well on vLLM?
- RandyOrion 5mo agoA small dense multimodal model with audio support, interesting. Wait, *Excluding Chinese language. This is ... curious. P.S. Where is gemma 4 124b?
- kylehotchkiss 5mo agoWhere are the computers we could purchase to run 124b models :’(
- thot_experiment 5mo agoYou can get SXM V100s for like $100 off ebay, if you're willing to do the troubleshooting work to get em running with adapters you can build a computer capable of fitting a Q4 quant of a 120b model in VRAM for something like fifteen hundred dollars. (assuming you already have some RAM sticks laying around T___T)
- comma_at 5mo agoAre there qwen or minimax or other open weight models of same hardware requirements that outperform this?
- senko 5mo agoI ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12b-q4.html https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12... The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run just fine. So, with those qualifiers, I think it's a decent local coding model. It roughly compares with GPT-4.1 (!!), released 14 months ago, on the output: https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.html https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.ht... (actually I'd call it better, but those syntax errors...) I ran the quantized version (4-bit GGUF) on my consumer-grade card with 12G of VRAM and got 5t/s for output. Not for interactive use for coding, but fairly capable model. To me, it's fascinating how much progress we got in over a year. GPT-4.1 was considered an extremely capable coding model. Now we got something with 12B of params performing roughly the same (in this specific benchmark, disclaimers, etc). Lists of various models I tested: https://senko.net/vibecode-bench/ https://senko.net/vibecode-bench/
- frikk 5mo agoThank you for sharing this. Do you think the syntactical issues could be addressed with fine tuning or some other kind of parameter tweaking? That's frustrating hah.
- profunctor 5mo agoWith a harness you could feed the code to a linter and if there are errors feed that to a model automatically. It’s amazing that the models are good enough that I haven’t bothered doing this
- zigzag312 5mo ago> It roughly compares with GPT-4.1 (!!), released 14 months ago I think the mayor win for coding was reasoning. That's why such a small model can match GPT-4.1 in coding, but I suspect that GPT-4.1 still wins in general world knowledge due to bigger size.
- julianlam 5mo agoLast time I tried Gemma 4 (26B-A4B) its memory usage would balloon and consume all of my swap until my machine died. Qwen 3.6 on the other hand barely uses any memory at all for its KV cache.
- verdverm 5mo agoTurns out when you block people from the best and biggest hardware, they get innovative. It reminds me of the Pentium days when everyone was shipping inefficient programs because the processor would be better next year.
- iknowstuff 5mo agowe never stopped doing that!
- zkmon 5mo agoIt's quite interesting to see the quants pour into the HF page. I keep refreshing it and see many new quants every few mins.
- petercooper 5mo agoIts image processing is terrible. I ran several tests against it against Qwen 3.5 0.8b (yes, 7% the size) and Qwen beat it every time with Gemma often getting things entirely wrong. I even gave it a plain image saying "This is a test" and it thought for 6 minutes trying to analyze it and failed. Qwen 3.5 0.8b confidently got it in under a second. It may be that the Q6 quant I got is borked (or my LM Studio is), but either way, the 0.8b's performance is mind boggling in comparison.
- ma2kx 5mo agoI guess Google implements more / stronger guard rails than Alibaba and thus confuses these small models. At least this was my impression with Gemma3 models where it often said that the image contains some nudity / sex scenes and therefore it cannot give a description of the image. Never understood the point of this behavior....
- jimmy76615 5mo agoThe biggest problem with all the Google models has always been RLHF, particularly safety training. They take a good, smart model and make it behave like a corporate person that has been to far to many forced anti-{sexism, racism...} seminars so that it is now living in fear of saying something that could be construed as wrong by some moral standard.
- staticman2 5mo agoThis is almost certainly not true. If it was, they wouldn't need to be using the classifiers they are using to warn Gemini about problematic prompts.
- ai_fry_ur_brain 5mo ago[flagged]
- thot_experiment 5mo agoI've always found the Gemma models to vastly under-perform on vision tasks compared to Qwen so that's nothing new.
- semiinfinitely 5mo agoPerfection is achieved, not when there is nothing more to add, but when there is nothing left to take away
- anonova 5mo agoDo Gemma 4 models compete with Gemini 3.1 Flash-Lite? I would assume even the smallest Gemini model would outperform even Gemma 4 31B, but I can't really get a sense of performance or output quality difference.
- mchusma 5mo agoGemma 4 31b outperformed Gemini 3.1 Flash-Lite in our app benchmarks (agentic tool use via api in our application as a part of various workflows). But google won't let you pay to use Gemma models, you have to go elsewhere, I think this may be because it would cannabilize Flash-lite.
- verdverm 5mo agoYou can actually get the gemma-4 models on a per-token API basis, you just have to click some extra buttons (in GCP). Not the same for other open weight models. For those they make you run your own hardware. Use OpenCode Go instead: https://opencode.ai/go https://opencode.ai/go
- dTal 5mo agoCurious logic. Does Google want you to use it or not? Do they want to be paid for tokens or not? why segregate open and closed? It's not parameter size - there is apparently such a thing as "Gemini Nano", which famously is downloaded automatically by Chrome. How similar is it to Gemma E4B? And how strange - you have the weights, but you don't "have" them?
- asim 5mo agoWe are now entering the closed loop game. Google doesn't need anyone else to accelerate their models. This is their bread and butter. I'm both shocked but also not surprised that they continue to develop such efficiencies. Honestly it's like silicon and CPU architecture advancement. We kept shrinking it and shrinking it and it kept getting more and more powerful and here we are with AI and it's only going to be 100x more efficient with time. Maybe there's some point of decay but essentially the next 30 years will be more advanced than the last 30 and were going to be living in some sort of futurist blade runner scenario where gene editing is repairing ageing cells, organs and curing all sorts of cancers that haven't even appeared yet. Beyond our lifetimes people will live to 125 quite steadily and with great mobility and then obviously people will look to how do we get to living 1000 years, which of anyone is religious knows Noah and others lived to that age in a totally different era. Anyway I'm going off on some tangent but look back 30 years. Now look forward 30 years. It's going to be insane. May God protect us.
- bityard 5mo ago> We kept shrinking it and shrinking it and it kept getting more and more powerful and here we are with AI and it's only going to be 100x more efficient with time. It's definitely an exciting time, but in terms of advancements in the state of the art, there is a lot of low-hanging fruit left to pick. There IS a bottom, however, as you can only encode so much "knowledge" in a small number of parameters. This feels to me a lot like what the early days of what radio or aviation must have been like. Or, heck, microcomputers even.
- asim 5mo agoIt's definitely a core component of a bigger system. We are effectively trying to recreate intelligence and human life through models and robotics. So the key insights for me, the LLM is the cerebral cortex but we have a lot more to recreate. Once you map in sensory input continuously and give it physical robotics, things start to change. But even before that leaving these things in simulated realities is what will happen, and right now we have things that operate based on our commands, but a complete step function will be the things that act on their own and that will be a very dangerous time but also where we see some very surreal things happening. They might not necessarily be made in the same way either, they might operate on entirely different types of architecture.
- SuperV1234 5mo agoHow does this compare to frontier models?
- powera 5mo agoI'm seeing very low quality results on LMStudio with this model. Worse than Gemma 3 12B. It is getting questions like "David has 18 apples and Ivan has 7 apples. How many apples do they have together?" wrong half the time, while Gemma3 12B could very consistently answer that. Other smoke tests (like Chinese translation, and the infamous "Rs in Strawberry" test) also show poor results. I don't know if it is a quantization/release issue, if the parameters needed for accurate responses have changed (i.e. it needs "thinking" tokens to handle its base error rate), or if the model has been so focused on audio/video that the text processing is bad.
- szmarczak 5mo agoI just ran this 25 times and every time I got 25. Ran this via ollama.
- Lapsa 5mo ago[dead]
- __natty__ 5mo agoIt’s fascinating for me to see how small language models grow recently in capabilities while still consumer friendly in size to run on their machines
- deleted 5mo ago[deleted]
- thomasjb 5mo agoUnfortunately there's no gguf quants of the assistant model yet: https://huggingface.co/models?other=base_model:quantized:google/gemma-4-12B-it-assistant https://huggingface.co/models?other=base_model:quantized:goo...
- kristjansson 5mo agoI think MTP Gemma4 support is still WIP https://github.com/ggml-org/llama.cpp/pull/23398 https://github.com/ggml-org/llama.cpp/pull/23398 ?
- dofm 5mo agoThis has been my impression. The underlying LiteRT-LM framework used in the edge gallery does support the MTP drafters for the smaller models, but according to: https://developers.google.com/edge/litert-lm/models/gemma-4 https://developers.google.com/edge/litert-lm/models/gemma-4 > Note: LiteRT-LM supports E2B and E4B models today, with support for larger models coming soon. So even Google aren't shipping MTP support for the 26B and 31B models yet.
- thot_experiment 5mo ago[dead]
- kristjansson 5mo agoUpdate: not any more!
- scirob 5mo agoQuickly deployed it to check some benchmarks relevant for German language. These are results for CohereLabs/include-base-44 german only : Gemma 4 12B %61.9 Gemma 4 26B (a4b MoE) 0.647 Qwen 3 14B 0.621 Gemma 4 12B 0.618 Ministral 14B 2512 0.604 Gemma 3 12B 0.547 The quwen 3 14B vs Gemma 4 12B difference is within random variance they same in some repeat runs they actually got the exact same score. Next step up Gemma 4 31B gets 0.676 on this. Or let in some reasoning Qwen 3 14B (reasoning) 0.676. I'll run some cheat-proof benchmarks ones tomorrow see if qwen is still on top.
- kordlessagain 5mo agoI just ran a short tool use test and it's doing pretty well.
- christina97 5mo agoIt seems worse in all aspects to the 26B A4B? I would have thought dense models beat MoE still on many benchmarks? Is the entire point of this model then that it runs if you don’t have enough GPU memory to load the 26B? That one runs faster anyway due to lower active params.
- SubiculumCode 5mo ago"Laptop ready: Small enough to run locally with just 16GB of VRAM or unified memory." I wish. I just have 12.
- adt 5mo agohttps://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- dyauspitr 5mo agoJust tried this out. Jesus Christ. Google does some things so well.
- deleted 5mo ago[deleted]
- SwellJoe 5mo agoI mean, they did invent the technology. It's actually kinda surprising they're not the leader in the space. They kinda got Kodaked (though the story is still playing out, and I guess they're still somewhat competitive in the space even if Anthropic and OpenAI are the leaders).
- 4k4 5mo agoI'm actually thinking how much this is bett3r (besides multimedia) over prismml's 1.5bit model based on qwen2.5 or sth.
- foota 5mo agoIt feels like this would be beneficial to give the model more of a deep understanding of visual knowledge.
- jamwise 5mo ago"Small enough to run locally with just 16GB of VRAM or unified memory" With many laptops dropping back down to 8GB because of the memory shortage there's some interesting pressures building in the industry.
- kristianp 5mo agoWhat quantisation do the creators intend this to be run at? They talk about 16GB of ram, so should it be run at 8 bit? People here are talking about using q4, but I would have thought a smaller model like this wouldn't perform well at such low bits per parameter. Edit, it looks like their bechmarks would have been done at 16 bit float, as the hugging face release is that size: https://huggingface.co/google/gemma-4-12B https://huggingface.co/google/gemma-4-12B . Which is a little deceptive: they're advertising an 8 bit size will fit on 16GB laptops, while releasing a 16bit size. I guess we have to wait for someone to produce perplexity curves at different Q's.
- easygenes 5mo agoThey haven't made one for this new model, but Unsloth has a comprehensive quant KLD map of Gemma 4 26B A4B here: https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtRVN97QDzO0Pq7SscC7x%2Fgemma%20426b%20bench.png?alt=media&token=80b4da76-efe9-4554-8e31-cca6494d456c https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-p...
- easygenes 5mo agoI want to like the vision capabilities of the model. However, when I gave it an image which Gemma 26B A4B and Qwen 3.6 35B A3B has no problem correctly describing in detail, including identifying the Taj Mahal in the background it utterly failed. Its sense of the image was that it was a "distorted wide panorama" and even when I asked directly if it was the Taj Mahal it said no. The reference models saw it correctly as a normal square image taken from a fairly rectilinear lens (iPhone main camera).
- easygenes 5mo agoI have now also tried it on this scatter plot: https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtRVN97QDzO0Pq7SscC7x%2Fgemma%20426b%20bench.png?alt=media&token=80b4da76-efe9-4554-8e31-cca6494d456c https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-p... Similarly, the 26B A4B Gemma 4 and the 35B A3B Qwen 3.6 identify it clearly, give me the title and trends analysis fairly accurately. While this 12B spits out gobbledygook about it having something to do with hard-drive capacity. It's like it can barely see, gets the very broad strokes (knows it's looking at some kind of chart), but can't identify any details clearly.
- alienjesus 5mo agogood one, wanna try on Cerebras inference in Agentic Browsing
- dgacmu 5mo agoI was excited about this until I fed it one of my local test problems: coin identification. I then spent 10 minutes arguing with it that a photo of a 1998 washington quarter was not, in fact, a Morgan Silver Dollar. I mean, I wish it was. It went into a crash loop on a british columbia 1 dollar coin. This happened with both Q4_1 and Q8. Maybe I'm holding it wrong or it's just really bad for this task. In contrast, gemma4 gets the british columbia coin right though it also mis-identifies the quarter. gemini 3.1-flash-lite nails them both. Was getting about 50 t/s output on a 3090 with Q8 which seems ok.
- sureglymop 5mo agoWhy would you expect it to be good for this particular highly specific task? Curious.
- dgacmu 5mo agoAh! Good question: Google's non-open-weights models (Gemini, etc) have almost always outperformed on image recognition tasks compared to any other models. I use a mix of in-house and Gemini for image classification tasks for $startup. No other models have done as well, and I had hoped that some of that would spill over into their open source models. It does to a degree - bigger Gemma models are okay.
- wuyunhuo 5mo agoThe optimal small-model solution, delivering multimodal, reasoning, and coding experiences on affordable hardware that were remarkably close to those of mid-to-large models at the time.
- OhioMan2943 5mo agoAsked it to name the director who wears a rolex and likes submarines. It said christopher nolan.
- keyle 5mo agoNot terribly impressed with this one. I asked it for recommendation between Paris to Berlin and option 3 was Rome... and option 4 was Tokyo. mmmkay.
- benbojangles 5mo agoI run gemma-4-26b-bf16 in mtp mode and it runs very smooth, spitting out answers in seconds and outputting text 30x faster than i can read.
- benbojangles 5mo agowhy combine audio & image analysis into an llm though, why not allow the user to choose their own audio & image analysis alongside their own llm choice?
- Miles_Stone 5mo ago[flagged]
- briansm 5mo agoStrange that they are feeding raw audio in. Even in humans, there is a hardware transform to the frequency domain (the cochlea) before data is fed to the brain, effectively doing this part in the LLM seems inefficient.
- nialse 5mo agoThe FFT is essentially just a matrix multiplication, or two. No need for fancy conversions. Just a huge amount of training data and a very large array.
- outageroom 5mo agoI really like the idea of small models that you can get the most out of. If I weren't a programmer, I wouldn't even know what I would use Opus 4.8 or GPT 5.5 models for.
- ThouYS 5mo agoironically, coding is the area that small models excel most at. qwen 3.6 27b is an insane agentic model. but Opus can be much more than a programmer: language tutor, knowledgeable "friend", therapist
- baalimago 5mo agoI don't understand why Google does this. If I can run this locally, why would I need a subscription or use any inference provider, including Google..? Scorched earth tactics to make anthropic and openai IPO fail?
- tgtweak 5mo agoIt's a disruption game - releasing competent open models disrupts smaller labs trying to release their own or commercialize their own. It's a similar rationale behind the Chinese labs releasing near-frontier open-weighted models, the goal is to disrupt and lift the barrier of entry for would-be competitors.
- accrual 5mo agoSplendid model, it reminds me of Gemma3 27B which was my favorite local model last year. Gemma always had a bit more warmth/empathy compared to Qwen and Mistral in my experience and I found it more useful for personal questions. My system has a 4080 Super (16GB) installed and using llama.cpp (b9333-35c9b1f39) I got these results on a test prompt: * Qwen3.5-9B-Q6_K.gguf - Prompt: 1492.0 t/s | Generation: 81.0 t/s * gemma-4-12b-it-Q4_K_M.gguf - Prompt: 1329.2 t/s | Generation: 72.3 t/s * gemma-4-12b-it-Q8_0.gguf - Prompt: 504.4 t/s | Generation: 25.2 t/s
- synergy20 5mo agoollama does not support this yet, what else can I try
- tmuhlestein 5mo ago[flagged]
- fluffyspork 5mo agoAccording to Gemini to run full 256k context window with unified RAM 4-bit Quantized (Q4_K_M GGUF): You need at least 25 GB of total RAM. This is the most practical configuration for consumer hardware. 8-bit Quantized (Q8_0 / SFP8): You need at least 32 GB to 36 GB of RAM Uncompressed 16-bit (BF16): You will need upwards of 45 GB to 50 GB of RAM to account for both the 26.7 GB base model and the massive KV cache.
- rft 5mo agoThose numbers are wrong for most use cases. Likely the LLM did not take SWA (Sliding Window Attention) of G4 into account. Without SWA those numbers could be correct, I can't load a q8 without SWA on a 24GB card. I tested this with the at the time newest llama.cpp master on a Linux system with 2 3090 24GB, only one was used for testing. q8 without any KV quant, 256k context, mmproj loaded takes less than 20GB VRAM. This runs at about 1.5k to 2k tok/s pp and 40-50 tok/s gen (slightly lowered power limits & undervolted). q8 with 64k non-quant context and mmproj takes just under 16GB VRAM. Drop down to the q6k model, no mmproj, 64k non-quant context and it fits in 12GB VRAM. All the way down to q4km and some batch size tweaking and it barely fits into 8GB VRAM. 64k context is the minimum for Hermes agent, so a vision capable "agentic" model fits into a 16GB card. This is very impressive. I am currently testing how smart the model is and it does decently so far, had one looping issue it recovered after a lot of tokens, did some basic tool calling.
- LarsKrimi 5mo agoIt seems very good at understanding human language clues even in a 4-bit (Q4_K_S) model, similar in feel to E4B but a great incremental improvement. Interesting for my 8GB VRAM system, but the system RAM requirement seems to balloon quickly, and it starts misspelling words. Also token/s drops off quickly it seems
- macwhisperer 5mo agocheck out a custom 4-bit quant I made today https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense should run perfect for 12-16gb with maybe 10-20k context seems intelligent enough that I would recommend this as a daily driver for friends who just want a local ai that can do most things relatively quickly (getting 10 tps on my m2 air)
- Saris 5mo agoFrom playing with it a bit it doesn't seem as reliable as qwen 3.6 for tool calls, it almost immediately got stuck in some kind of loop.