6 ms·
Probably a better link: https://developers.googleblog.com/en/introducing-gemma-3n/ https://developers.googleblog.com/en/introducing-gemma-3n/ Gemma 3n is a mod
by onlyrealcuzzo 1y ago
Probably a better link: https://developers.googleblog.com/en/introducing-gemma-3n/ https://developers.googleblog.com/en/introducing-gemma-3n/
Gemma 3n is a model utilizing Per-Layer Embeddings to achieve an on-device memory footprint of a 2-4B parameter model.
At the same time, it performs nearly as well as Claude 3.7 Sonnet in Chatbot Arena.
- ai-christianson 1y agoThat seems way too good to be true. What's the catch?
- Vuizur 1y agoIt is not very good at hard tasks, its ranking is much worse there.
- moneywoes 1y agosorry, any examples of hard tasks
- refulgentis 1y agoI used to defend LMSys/Chatbot Arena a lot but threw in the towel after events of the past three months. I can give more details if you (or anyone else!) is interested. TL;DR: it is scoring only for "How authoritative did the answer look? How much flattering & emojis?"
- Jowsey 1y agoIs this not what Style Control (which IIRC they're making default soon) aims to mitigate?
- refulgentis 1y agoI'm not 100% sure what their rationale is for it, the launch version of style control was a statistical model that penalized a few (4?) markdown shibboleths (lists, headers, ?). Not sure if they've shared more since. IMVHO it won't help, at all, even if they trained a perfect model that could accurately penalize it* The main problem is its one off responses, A/B tested. There's no way to connect it into all the stuff we're using to do work these days (i.e. tools / MCP servers), so at this point its sort of skipping the hard problems we'd want to see graded. (this situation is a example: whats more likely, style control is a small idea for an intractable problem, or Google has now released multiple free models better than Sonnet, including the latest, only 4B params? To my frustration, I have to go and bench these things myself because I have an AI-agnostic app I build, but I can confirm it is not the case that Gemma 3-not-n is better than Sonnet. 12B can half-consistently make file edits, which is a major step forward for local tbh) * I'm not sure how, "correctness" is a confounding metric here: we're probably much more likely to describe a formatted answer in negative terms if the answer is incorrect. In this case I am also setting aside how that could be done, just saying it as an illustration of no matter what, it's the wrong platform for a "how intelligent is this model?" signal, at this point, post-Eliza post-Turing, couple years out from ChatGPT 1.0
- int_19h 1y agoThe catch is that "does as good as X" is pretty much never representative of real world performance when it comes to LLMs. In general, all those scores are mostly useful to filter out the models that are blatantly and obviously bad. But to determine whether the model is actually good at any specific thing that you need, you'll have to evaluate them yourself to find out.
- Deathmax 1y agoIt's not a 4B parameter model. The E4B variant is 7B parameters with 4B loaded into memory when using per-layer embedding cached to fast storage, and without vision or audio support.
- zamadatix 1y agoThe link says E2B and E4B have 4B and 8B raw parameters, where do you see 7B?
- jdiff 1y agoThere's a 7B mentioned in the chat arena ELO graph, I don't see any other references to it though.
- osanseviero 1y agoHi! The model is 8B if you also load the vision and audio components. We just used the text model in LMArena.
- lostmsu 1y agoAre vision and audio components available yet?
- esafak 1y agoImagine a model smarter than most humans that fits on your phone. edit: I seem to be the only one excited by the possibilities of such small yet powerful models. This is an iPhone moment: a computer that fits in your pocket, except this time it's smart.
- codr7 1y agointelligence != memory
- esafak 1y agoML is not memorization. Besides, how much memory do you think this model has?
- codr7 1y agoI know, it's worse.
- TeMPOraL 1y agoIt's understanding.
- rhdjsjebshjffn 1y agoSure, if you still think the word has meaning.
- TeMPOraL 1y agoYes, I do. Any way you slice this term, it looks close to what ML models are learning through training. I'd go as far as saying LLMs are meaning made incarnate - that huge tensor of floats represents a stupidly high-dimensional latent space, which encodes semantic similarity of every token, and combinations of tokens (up to a limit). That's as close as reifying the meaning of "meaning" itself as we ever come. (It's funny that we got there through brute force instead of developing philosophy, and it's also nice that we get a computational artifact out of it that we can poke and study, instead of incomprehensible and mostly bogus theories.)