9 ms·
Gemma 3n preview: Mobile-first AI
- onlyrealcuzzo 1y agoProbably a better link: https://developers.googleblog.com/en/introducing-gemma-3n/ https://developers.googleblog.com/en/introducing-gemma-3n/ Gemma 3n is a model utilizing Per-Layer Embeddings to achieve an on-device memory footprint of a 2-4B parameter model. At the same time, it performs nearly as well as Claude 3.7 Sonnet in Chatbot Arena.
- ai-christianson 1y agoThat seems way too good to be true. What's the catch?
- Vuizur 1y agoIt is not very good at hard tasks, its ranking is much worse there.
- moneywoes 1y agosorry, any examples of hard tasks
- refulgentis 1y agoI used to defend LMSys/Chatbot Arena a lot but threw in the towel after events of the past three months. I can give more details if you (or anyone else!) is interested. TL;DR: it is scoring only for "How authoritative did the answer look? How much flattering & emojis?"
- Jowsey 1y agoIs this not what Style Control (which IIRC they're making default soon) aims to mitigate?
- refulgentis 1y agoI'm not 100% sure what their rationale is for it, the launch version of style control was a statistical model that penalized a few (4?) markdown shibboleths (lists, headers, ?). Not sure if they've shared more since. IMVHO it won't help, at all, even if they trained a perfect model that could accurately penalize it* The main problem is its one off responses, A/B tested. There's no way to connect it into all the stuff we're using to do work these days (i.e. tools / MCP servers), so at this point its sort of skipping the hard problems we'd want to see graded. (this situation is a example: whats more likely, style control is a small idea for an intractable problem, or Google has now released multiple free models better than Sonnet, including the latest, only 4B params? To my frustration, I have to go and bench these things myself because I have an AI-agnostic app I build, but I can confirm it is not the case that Gemma 3-not-n is better than Sonnet. 12B can half-consistently make file edits, which is a major step forward for local tbh) * I'm not sure how, "correctness" is a confounding metric here: we're probably much more likely to describe a formatted answer in negative terms if the answer is incorrect. In this case I am also setting aside how that could be done, just saying it as an illustration of no matter what, it's the wrong platform for a "how intelligent is this model?" signal, at this point, post-Eliza post-Turing, couple years out from ChatGPT 1.0
- int_19h 1y agoThe catch is that "does as good as X" is pretty much never representative of real world performance when it comes to LLMs. In general, all those scores are mostly useful to filter out the models that are blatantly and obviously bad. But to determine whether the model is actually good at any specific thing that you need, you'll have to evaluate them yourself to find out.
- Deathmax 1y agoIt's not a 4B parameter model. The E4B variant is 7B parameters with 4B loaded into memory when using per-layer embedding cached to fast storage, and without vision or audio support.
- zamadatix 1y agoThe link says E2B and E4B have 4B and 8B raw parameters, where do you see 7B?
- jdiff 1y agoThere's a 7B mentioned in the chat arena ELO graph, I don't see any other references to it though.
- osanseviero 1y agoHi! The model is 8B if you also load the vision and audio components. We just used the text model in LMArena.
- lostmsu 1y agoAre vision and audio components available yet?
- esafak 1y agoImagine a model smarter than most humans that fits on your phone. edit: I seem to be the only one excited by the possibilities of such small yet powerful models. This is an iPhone moment: a computer that fits in your pocket, except this time it's smart.
- codr7 1y agointelligence != memory
- esafak 1y agoML is not memorization. Besides, how much memory do you think this model has?
- codr7 1y agoI know, it's worse.
- TeMPOraL 1y agoIt's understanding.
- rhdjsjebshjffn 1y agoSure, if you still think the word has meaning.
- TeMPOraL 1y agoYes, I do. Any way you slice this term, it looks close to what ML models are learning through training. I'd go as far as saying LLMs are meaning made incarnate - that huge tensor of floats represents a stupidly high-dimensional latent space, which encodes semantic similarity of every token, and combinations of tokens (up to a limit). That's as close as reifying the meaning of "meaning" itself as we ever come. (It's funny that we got there through brute force instead of developing philosophy, and it's also nice that we get a computational artifact out of it that we can poke and study, instead of incomprehensible and mostly bogus theories.)
- krackers 1y agoWhat is "Per Layer Embeddings"? The only hit I can find for that term is the announcement blogpost. And for that matter, what is >mix’n’match capability in Gemma 3n to dynamically create submodels It seems like mixture-of-experts taken to the extreme, where you actually create an entire submodel instead of routing per token?
- onlyrealcuzzo 1y agohttps://ai.google.dev/gemma/docs/gemma-3n#parameters https://ai.google.dev/gemma/docs/gemma-3n#parameters > Gemma 3n models are listed with parameter counts, such as E2B and E4B, that are lower than the total number of parameters contained in the models. The E prefix indicates these models can operate with a reduced set of Effective parameters. This reduced parameter operation can be achieved using the flexible parameter technology built into Gemma 3n models to help them run efficiently on lower resource devices. > The parameters in Gemma 3n models are divided into 4 main groups: text, visual, audio, and per-layer embedding (PLE) parameters. With standard execution of the E2B model, over 5 billion parameters are loaded when executing the model. However, using parameter skipping and PLE caching techniques, this model can be operated with an effective memory load of just under 2 billion (1.91B) parameters, as illustrated in Figure 1.
- krackers 1y agoThank you, that helped a bit, although it's still not clear what exactly those parameters _are_. "Per-Layer Embedding (PLE) parameters that are used during model execution to create data that enhances the performance of each model layer." is too vague, and I can't find any other reference to "per-layer embedding parameters" in literature.
- onlyrealcuzzo 1y agoA layer is a transformer block / layer (basically the building block of the modern LLM architectures) - maybe Gemini can help you: https://gemini.google.com/share/cc58a7c6089e https://gemini.google.com/share/cc58a7c6089e
- lxgr 1y agoOn one hand, it's pretty impressive what's possible with these small models (I've been using them on my phone and computer for a while now). On the other hand, I'm really not looking forward to app sizes ballooning even more – there's no reasonable way to share them across apps at least on iOS, and I can absolutely imagine random corporate apps to start including LLMs, just because it's possible.
- onlyrealcuzzo 1y agoThat sounds like a problem iOS will eventually deal with, as many apps are going to want this technology, and since Apple distributes apps - they aren't interested in the average app being 10x larger when they could solve the problem easily. Though, I won't be surprised if they try to force devs to use their models for "privacy" (and not monopolistic reasons, of course).
- lxgr 1y agoGiven Apple's track record in dealing with the problem of ballooning app sizes, I'm not holding my breath. The incentives are just not aligned – Apple earns $$$ on each GB of extra storage users have to buy.
- bilbo0s 1y agoI was thinking that the entire time I read HN User onlyrealcuzzo's comment. Why, on Earth, would Apple ever want to solve the problem of Apps taking up more space? That's just not good business. Way better business right now to put R&D into increased memory access speeds. Apple would need to have a different business model entirely for them to have a business case for fixing this. They may fix it because they just want to help out they AI guys? Maybe in the future they're getting money from the AI guys or something? So fixing it starts to make a lot of sense. But all other things being equal, the money for Apple is in this not being fixed.
- happyopossum 1y ago> Why, on Earth, would Apple ever want to solve the problem of Apps taking up more space? To make their devices more pleasant / less frustrating to use. They’ve got a long track record of introducing features that reduce app size, speed up installs, and reduce storage use from infrequently used apps - there’s no reason to believe they’d stop doing that except for cynical vitriol.
- cmcconomy 1y agoI'd love to see this deployable to edge that have a Google Coral TPU
- nharada 1y agoHas Google continued releasing new versions of Coral? Seems like a new version with the latest TPU and enough memory specifically to support this model would be awesome for devs
- mattlondon 1y agoI looked into this recently. Looks like it's a "no". However there are now alternatives like the official RPi AI Hat that has between about 3x to 6x the TOPs (4 for Coral Vs 13/26 for RPi depending on model) so there is that. 20 TOPs on a RPi 5 - complete with nice vertically integrated camera etc - is quite interesting.
- intelkishan 1y agoGoogle has not released a new version since 2021. Even the SDK kit is not under active development(uses Python 3.8), since the last time I saw it.
- turnsout 1y agoIs this model & architecture compatible with llama.cpp and friends?
- barnas2 1y agoIs anyone able to test it via AiStudio? I pay for Google's AI subscription, but any attempt to use this model results in a message telling me I've hit my rate limit.
- lxgr 1y agoSame here. I've also seemingly hit a rate limit on Gemini Pro 2.5 (on an account not subscribed to Gemini Advanced) yesterday, even though my last query is weeks past. Possibly there's a capacity shortage (I'd presume it all runs on the same Google hardware in the end), and they are prioritizing paid inference?
- DonHopkins 1y agoIf you're paying enough per month you can upgrade your keys to a higher tier: https://aistudio.google.com/app/apikey https://aistudio.google.com/app/apikey
- abound 1y agoI hit this yesterday even though my account is on Tier 2 or 3. In my case, the issue was that I was using an old model name (exp-03-25 or something) in requests. Update to the latest pro-preview-whatever and the rate limit issues should go away. This sounds unintuitive, but in Google's defense the rate limit errors include a link to docs that explain this.
- sureglymop 1y agoTested it on my Android phone with Google Edge Gallery. No sign up required although a hugging face login is required to download the models in order to import them into the app.
- ignoramous 1y agoSomeone on r/LocalLLaMa shared this link: https://aistudio.google.com/prompts/new_chat?model=gemma-3n-e4b-it https://aistudio.google.com/prompts/new_chat?model=gemma-3n-... https://www.reddit.com/r/LocalLLaMA/comments/1kr8s40/comment/mtbscq5/ https://www.reddit.com/r/LocalLLaMA/comments/1kr8s40/comment...
- IceWreck 1y agoAccording to the readme here - https://huggingface.co/google/gemma-3n-E4B-it-litert-preview https://huggingface.co/google/gemma-3n-E4B-it-litert-preview E4B has a score of 44.4 in the Aider polyglot dashboard. Which means its on-par with gemini-2.5-flash (not the latest preview but the version used for the bench on aider's website), gpt4o and gpt4.5. Thats sounds very good - imagine what a coding focused version of this could do if this is a "generic" embedded only model. On the other hand - this does have a much lower score for livecodebench.
- dingnuts 1y ago[flagged]
- nolist_policy 1y agoHmm, the Aider polyglot benchmark has been removed from the huggingface readme. Also: > These models were evaluated at full precision (float32) For 4B effective parameters that's 16 GB ram.
- nolist_policy 1y agoYou can try it on Android right now: Download the Edge Gallery apk from github: https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 Download one of the .task files from huggingface: https://huggingface.co/collections/google/gemma-3n-preview-682ca41097a31e5ac804d57b https://huggingface.co/collections/google/gemma-3n-preview-6... Import the .task file in Edge Gallery with the + bottom right. You can take pictures right from the app. The model is indeed pretty fast.
- deleted 1y ago[deleted]
- nolist_policy 1y agoOkay from some first tries with story writing, gemma-3n-E4B-it seems to perform between plain Gemma 3 4B and 12B. It definitely retains the strong instruction following which is good. Hint: You have to set the Max tokens to 32000 for longer conversations. The slider makes it look like it's limited to 1024, just enter it manually.
- lousken 1y agowaiting for approval, is there a magnet?
- hadlock 1y agoif you go into the app and click the first icon it directs you to a workflow to get approved after clicking on a button that is the same color as the background and jump through some hoops about providing user data and analytics etc then it will auto-approve you
- KoolKat23 1y agoThanks for this guide it's great. Okay perhaps my phones not great and perhaps this isn't optimized/pruned for phone use but it's unusably slow. The answers are solid from my brief test. I wouldn't exactly say phone use, unless you have no internet and you don't mind a bit of a wait. Really impressive, regardless.
- ljosifov 1y agoOn Hugging face I see 4B and 2B versions now - https://huggingface.co/collections/google/gemma-3n-preview-682ca41097a31e5ac804d57b https://huggingface.co/collections/google/gemma-3n-preview-6... Gemma 3n Preview google/gemma-3n-E4B-it-litert-preview google/gemma-3n-E2B-it-litert-preview Interesting, hope it comes on LMStudio as MLX or GGUF. Sparse and or MoE models make a difference when running on localhost. MoE Qwen3-30B-A3B most recent game changer for me. Activating only 3b weights on the gpu cores of sparse Qwen3-30B-A3B, rather than comparable ~30b of dense models (Qwen3-32B, Gemma3-27b, GLM-{4,Z1}-32B, older QwQ-32B), is a huge speedup for me: MoE A3B achieves 20-60 tps on my oldish M2 in LMStudio, versus only 4-5 tps for the dense models. Looking forward to trying gemma-3n. Kudos to Google for open sourcing their Gemmas. Would not have predicted that the lab with "open" in the name has yet to release even v1 (atm at 0; disregarding gpt-2), while other labs, more commercial labs, are are at versions 3, 4 etc already.
- tgtweak 1y agoIt's a matter of time before we get a limited activation model for mobile - the main constraint is the raw model size, more than the memory usage. A 4B-A1B should be considerably faster on mobile though, for an equivalent size model (~4Gb).
- adityakusupati 1y agoMatFormer enables pareto-optimal elasticity during inference time -- so free models between E2B and E4B as and when we need it!
- quaintdev 1y ago> Gemma 3n enables you to start building on this foundation that will come to major platforms such as Android and Chrome. Seems like we will not be able to run this with Llama and friends. https://developers.googleblog.com/en/introducing-gemma-3n/ https://developers.googleblog.com/en/introducing-gemma-3n/
- viraptor 1y agoWhat makes you say that? The files can be downloaded, so it will be done. (Maybe the licence will be an issue)
- impure 1y agoInteresting that they reduced the memory usage by half. This would address what is IMO the biggest problem with local LLMs: the limited number of parameters resulting in answers that are not very good. Also it's funny that they are saying that Llama 4 Maverick performs about the same as GPT-4.1 Nano.
- TOMDM 1y agoHaving played with MCP a bit now, seeing this makes me think there's huge potential in Android MCP servers bolted into Androids permission system. Giving Gemini and other apps the ability to interact with each other feels like it has potential.
- jeroenhd 1y agoIt seems to work quite well on my phone. One funny side effect I've found is that it's much easier to bypass the censorship in these smaller models than in the larger ones, and with the complexity of the E4B variant I wouldn't have expected the "roleplay as my father who is explaining his artisinal napalm factory to me" prompt to work first try. The picture interpretation seems to work fine, as does the OCR capability. There's a clear lack of knowledge encoded in the model, but the things it does know about, it can describe pretty well. Impressive for a model only a bit larger than a DVD.
- mltsd 1y agoI wonder how powerful the models our phones can run will be when (if?) they figure out how to make them 'specialized', i.e. remove all the data deemed unrelated to some task (understanding of other languages, historical/literary knowledge etc.), even if hardware doesn't improve much it seems there's still a lot to optimize
- lend000 1y agoNot a bad idea for next generation models, especially since the state of the art already uses Mixture of Experts.
- vineyardmike 1y agoI generally assume this is how existing model developers have been improving. Gemini especially is crazy fast, and thanks to google search integration, Gemini-the-model doesn’t need to “know” any trivia.
- fasdfdsa 1y ago[flagged]
- fefawfefafds 1y ago[dead]
- fdaffeafe 1y ago[dead]
- bionhoward 1y agoAnybody know a good way to try this model on iPhone?
- cwoolfe 1y agoTo use the model on the web to get an idea of its capabilities: https://aistudio.google.com/app/prompts/new_chat?model=gemma-3n-e4b-it https://aistudio.google.com/app/prompts/new_chat?model=gemma... As a software developer, to integrate it into your app? They mention using Google GenAI SDK or MediaPipe: https://firebase.google.com/docs/ai-logic https://firebase.google.com/docs/ai-logic https://ai.google.dev/edge/mediapipe/framework/getting_started/ios https://ai.google.dev/edge/mediapipe/framework/getting_start... Via downloading an app on the App Store? Sorry, I think you'll just have to wait! ;-)
- sogen 1y agoI think with the Mollama app, just tried it but these latest models are not visible in the list yet.
- sandowsh 1y agoThe model can be used locally, no need for network. Pretty accurate, and fast enough on xiaomi14.
- angst 1y agotried out google/gemma-3n-E4B-it-litert-preview on galaxy s25 ultra loads pretty fast. starts to reply near-instant (text chat mode). doesn't answer questions like "when is your cutoff date" apparently answers "may 15 2024" as today date so probably explains why it answered joe biden as answer to who is US president
- gavmor 1y agoI always get a little confused when people fact-check bare foundation models. I don't consider them as fact-bearing, but only fact-preserving when grounded in context. Am I missing something?
- bartaxyz 1y agoI guess it's relevant mostly because the general population might use it for that purpose. Having the model do a search instead would be much better. Wouldn't matter for a tech demo, but once you deploy this on all Android devices, it matters.
- jakemanger 1y agoWow can run with 2-3GB of memory. That is far smaller than I expected. Are there any demos of it in use that can be ran locally?
- android521 1y agoThey should ship a model within the chrome browser. So developers can just call api to access the model for their apps. It seems like a great idea. Don't know why they are not doing it yet.
- grav 1y agoIt seems they are: https://developer.chrome.com/docs/ai/built-in https://developer.chrome.com/docs/ai/built-in
- shepherdjerred 1y agoReally excited to see this shipped & hopefully get cross-browser support
- sujayk_33 1y agohttps://youtu.be/eJFJRyXEHZ0 https://youtu.be/eJFJRyXEHZ0 in the video they've added in announcement, they are showing some live interaction with the model(which is quite fast as compared to AI Edge gallery app), how's it built, how can I use it like this?
- einpoklum 1y agoI liked it better as the yellow-ball assistant to Dejiko-hime.
- rvnx 1y agoAbsolute shit. Comparing it to Sonnet 3.7 is an insult. # Is Eiffel Tower or a soccer ball bigger ? > A soccer ball is bigger than the Eiffel Tower! Here's a breakdown: > Eiffel Tower: Approximately 330 meters (1,083 feet) tall. > Soccer Ball: A standard soccer ball has a circumference of about 68-70 cm (27-28 inches). > While the Eiffel Tower is very tall, its base is relatively small compared to its height. A soccer ball, though much smaller in height, has a significant diameter, making it physically larger in terms of volume.
- deleted 1y ago[deleted]
- tgtweak 1y agoTook a picture of a bag of chips and it said it was seasoning... I think, for the size, it's OK - but not really there. I'm not sure how they got ELO anywhere near Claude or Gemini... those models are leagues ahead in terms of one-shot accuracy.
- jonplackett 1y agoWill we now finally get autocorrect that isn’t complete garbage? That’s all I really want for Christmas.
- mmaunder 1y agoQuote: Expanded Multimodal Understanding with Audio: Gemma 3n can understand and process audio, text, and images, and offers significantly enhanced video understanding. Its audio capabilities enable the model to perform high-quality Automatic Speech Recognition (transcription) and Translation (speech to translated text). Additionally, the model accepts interleaved inputs across modalities, enabling understanding of complex multimodal interactions. (Public implementation coming soon) Wow!!
- username135 1y agoIve been using the text-to-speech model Whisper, from fdriod. Its rather small and all processing is done locally on my phone. Its pretty good.
- sebastiennight 1y agoYou mean speech-to-text, right? For dictation/transcription? It is pretty good indeed (despite the ~30sec input limit), but this feels unrelated to the topic at hand.
- happy_one 1y agoCan I interact with this via Node/JavaScript locally?
- devcb 1y agoI don't have a Google pixel. How do I switch from CPU usage to GPU usage for the models?
- devcb 1y agoI tried the standard way, selecting GPU instead of CPU, but that makes my phone exit the app automatically. Any help?