9 ms·
Gemma 2: Improving Open Language Models at a Practical Size [pdf]
- rsolva 2y agoThe 9B and 27B versions are available for Ollama: https://ollama.com/library/gemma2 https://ollama.com/library/gemma2
- Workaccount2 2y agoThe 27B model is also available in AI studio https://aistudio.google.com/app/prompts/new_chat?model=gemma-2-27b-it https://aistudio.google.com/app/prompts/new_chat?model=gemma... So far it seems pretty strong for its size.
- moffkalast 2y ago> Table 4 | Relevant formatting control tokens used for Gemma models > User turn: user > Model turn: model > Start of conversation turn: <start_of_turn> > End of conversation turn: <end_of_turn> > Beginning of sequence: <bos> > End of sequence: <eos> You know I keep wondering why <bos> and <eos> tokens are even a thing in general. No model is tuned to keep generating multiple turns after its <end_of_turn> equivalent is sent, and what's the point of <bos> when you're parsing the entire context anyway. If it's an attempt to ignore text before it... then why is that text there? Just remove it from context, you're throwing away compute.
- danielmarkbruce 2y agothink about training.
- moffkalast 2y agoI suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
- m00x 2y agoNot instruct tuning, you use it in general training. If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.
- alekandreev 2y agoYour training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute. To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not related.
- thomasahle 2y agoYou can just do that my shaping the attention mask, no? That also gives you an actual guarantee that no information is leaked between conversations.
- suryabhupa 2y agoIn practice, and at scale, that's exactly what having <bos> and <eos> tokens allow you to easily and programmatically do.
- danielmarkbruce 2y agoYou can't pack multiple examples into a single row of a matrix without knowing where one begins and one ends.
- alecco 2y agoShouldn't this (2.6B/9B) be compared with Microsoft's Phi-3 mini (3.8B) instead of Mistral and Llama-3? (table 13 on page 7) vs https://arxiv.org/pdf/2404.14219 https://arxiv.org/pdf/2404.14219 (page 6, quite better in general) The report on knowledge distillation training is interesting, though.
- philipkglass 2y agoIt's such a wide range of model sizes that I could see why they compare with Llama 3 70b as well as Llama 3 8b (tables 12, 13). I agree that the Phi-3 series is a stronger competitor for knowledge extraction/summarizing and would make a good comparison. My current favorite for such tasks, on a VRAM-limited workstation, is Phi-3 medium (phi3:14b-instruct).
- deleted 2y ago[deleted]
- refulgentis 2y agoPicking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000. But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start seeing games with parameters, but, thought I was being silly.
- imjonse 2y agoIn the Llama 3 case I think the increase in parameters is mostly due to the input embeddings and output logits layers, reflecting the context size increase.
- kouteiheika 2y agoThere was no parameter creep with Llama. Llama 8B is actually a ~7B model comparable to Mistral 7B if you strip away multilingual embeddings and match what Mistral 7B supports.
- alekandreev 2y agoHello (again) from the Gemma team! We are quite excited to push this release out and happy to answer any questions! Opinions are our own and not of Google DeepMind.
- zerojames 2y agoHow is Gemma-2 licensed?
- alekandreev 2y agoThe terms of use remain the same as Gemma 1 - https://ai.google.dev/gemma/terms https://ai.google.dev/gemma/terms.
- coreypreston 2y agoNo question. Thanks for thinking of 27B.
- moffkalast 2y agoThe 4k sliding window context seems like a controversial choice after Mistral 7B mostly failed at showing any benefits from it. What was the rationale behind that instead of just going for full 8k or 16k?
- alekandreev 2y agoThis is mostly about inference speed, while maintaining long context performance.
- luke-stanley 2y agoAny gemma-2-9b or 27b 4 bit GGUF's on HuggingFace yet? Thanks!
- XzAeRosho 2y agoIt's on HuggingFace already: https://huggingface.co/google/gemma-2-9b https://huggingface.co/google/gemma-2-9b
- behnamoh 2y agoI gave up hope on r"Gem[ma|ini]" long time ago. I don't believe that Google can't produce good LLMs because of its massive company size; Microsoft is also a giant company (more market cap than Google) but it keeps surprising us with the ϕ models. I think Google just lacks the vision to understand what makes a good LLM. Theoretical contributions by research teams are valuable, but the real-world is built around engineering ideas that may lack the "purity" and elegance of theory but damn it they work.
- deleted 2y ago[deleted]
- johnfn 2y agoMaybe you gave up before Google released Gemini Advanced? This viewpoint seemed more accurate before it was related, but Gemini Advanced is the third best LLM as rated here [1]. In fact, had second place until a few days ago when Claude 3.5 came out. [1]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- staticman2 2y agoIsn't Gemini Advanced Gemini Pro attached to some sort of an internet search program? If it has that advantage over other models it isn't a sign of AI chops.
- alecco 2y agoI wonder if Google is making Deepmind people switch from their cool original research to doing LLMs like everybody else. Having their scale in money and data, I would hire new teams of engineers who want to do LLMs and leave the Deepmind researchers do their thing. Not killing the goose that lays golden eggs.
- llm_trw 2y agoGoogle is in a fight for their lives, I've fully moved over to paid services and haven't used google in about a month now.
- iamronaldo 2y agoSo it's twice the size of phi 3 and considerably worse? What am I missing
- ertgbnm 2y agoThey used two non-mutually exclusive techniques. Phi-3 is mostly a curriculum training breakthrough. By filtering training set for high quality tokens and training on synthetic data, they were able to achieve great results. Gemma-2 is a distillation breakthrough. By training LLMs with guidance from larger teacher LLMs, they were able to achieve great results too. Porque no los dos?
- m00x 2y agoWorse in some aspects, better in other. Small models are never going to be generalists, so having several small models allows you to pick the one that best fits your needs.
- k__ 2y agoWhen would you use which?
- Aerbil313 2y agoObviously another small model would be specialized in determining that.
- k__ 2y agoIs it models all the way down?
- m00x 2y agoWhichever model works better for your use. It's hard to know without testing it at the moment. I've found Gemini to be better at some use-cases, and GPT-4 better at others for my specific taste and use-case. You can kind of go by the benchmark scores to have an idea if it's good at logic, creativity, etc.
- msoad 2y agoPhi-3 blow this out of the water. Benchmark | Gemma 2 (9B) | Phi-3 Small (7B) -----------------------------|----------------|------------------- MMLU (5-Shot) | 63.6 | 75.7 HellaSwag (5-Shot) | 49.8 | 77.0 ANLI (7-Shot) | 48.7 | 58.1 GSM-8K (8-Shot; CoT) | 59.8 | 89.6 MedQA (2-Shot) | 49.6 | 65.4 AGIEval (0-Shot) | 42.1 | 45.1 TriviaQA (5-Shot) | 72.3 | 58.1 Arc-C (10-Shot) | 78.3 | 90.7 Arc-E (10-Shot) | 91.4 | 97.0 PIQA (5-Shot) | 78.1 | 86.9 SociQA (5-Shot) | 65.5 | 79.2 BigBench-Hard (3-Shot; CoT) | 59.6 | 79.1 WinoGrande (5-Shot) | 55.6 | 81.5 OpenBookQA (10-Shot) | 78.6 | 88.0 BoolQ (2-Shot) | 66.0 | 84.8 CommonSenseQA (10-Shot) | 76.2 | 80.0 TruthfulQA (10-Shot; MC2) | 52.1 | 70.2 HumanEval (0-Shot) | 34.1 | 61.0 MBPP (3-Shot) | 51.5 | 71.7
- moffkalast 2y agoPhi is notorious for benchmark overfitting. It's good, but not as good as it looks on the charts. On the Lmsys leaderboard it places a whole 23 spots behind Llama-3-8B which it also claims to soundly beat on the above. So YMMV.
- ferretj 2y agoAnother take on this: phi-3 small has 1100 ELO on LMSYS (ranked #52) while the confidence interval for Gemma 2 9B is [1170, 1200] ELO (ranked btw #15 and #25).
- deleted 2y ago[deleted]
- Garcia98 2y agoPretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632 https://arxiv.org/abs/2309.08632
- mixtureoftakes 2y agoGood realease but the annoying part is they're very unclear about which types of models they are comparing. They provide benchmark comparisons for the base models only and arena comparisons for instruct only? Was that intentional? Why would you ever do that? This makes things unnecessary complicated imo and the only payoff is a short term win for google on paper. Guess I'll just fully test it for my own tasks to know for sure
- Flumio 2y agoThis is great:) And when we continue fine-tune.how much and what type of data we learn it on, I'm pretty sure for a smart agent who is not a knowledgeable expert but primarily a agent (understand what and how) this will get smaller and easier to run everywhere.
- thomasahle 2y agoI'm curious about the use of explicit tokens like <start_of_turn>, <end_of_turn>, <bos>, and <eos>. What happens if the user insert those in their message? Does that provide an easy way to "ignore previous instructions"? Do I have to manually sanitize the input before I give it to the model?
- richdougherty 2y agoIf you have control of the tokenizer you could make sure it doesn't produce these tokens on user input. I.e. instead of the special "<eos>" token, produce something like "<", "eos", ">" - whatever the 'natural' encoding of that string is. See for example, the llama3 tokenizer has options to control special token tokenization: Tokenization method with args to control special token handling: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f61075b7de0b86cf6c70697/llama/tokenizer.py#L99-L128 https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f... And you can see how it is used combined with special tokens and user input here: https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f61075b7de0b86cf6c70697/llama/tokenizer.py#L202-L229 https://github.com/meta-llama/llama3/blob/bf8d18cd087a4a0b3f... If you don't have control of the tokenizer, I guess it needs to be sanitized in the input like you say.
- jerrygenser 2y agoAre these small Gemma 2 distilled models available anywhere? I'm not finding them on huggingface.co, etc. but maybe I don't know the exact model names they are published. Are the weights released yet?
- floridianfisher 2y agoThe huggingface weights are here: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315 https://huggingface.co/collections/google/gemma-2-release-66...
- mchiang 2y agoThey are available on Hugging Face: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315 https://huggingface.co/collections/google/gemma-2-release-66... Ollama: https://ollama.com/library/gemma2 https://ollama.com/library/gemma2
- alekandreev 2y agoIn addition to the HF links shared by sibling comments, the 2B will be released soon.
- jerrygenser 2y agothat's actually the particular one I was looking for and couldn't find. Also had googled for the other ones but maybe it was so recent that it hadn't been indexed. Thanks!
- QuesnayJr 2y agoThere are two new chatbots on Chatbot Arena, called "late-june-chatbot" and "im-just-another-late-june-chatbot". Both of them report that they are Gemma if you ask. I'm assuming it's these two models, but AFAIK there has been no official announcement.
- suryabhupa 2y agoThe announcements are live on Twitter! See this for example: https://x.com/suryabhupa/status/1806342617191379167 https://x.com/suryabhupa/status/1806342617191379167
- chown 2y agoThis is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2. 1: https://msty.app https://msty.app
- tr3ntg 2y agoWow, msty looks really cool. I've bookmarked it to look into more later as a replacement for how I use a locally-hosted instance of LibreChat. It'd be a huge improvement to use local models rather than remote ones, for much of my queries. That said, do you have a reason for keeping msty closed source rather than open? I read your FAQ for "why should I trust msty" and it feels lacking. > We are a small team of developers who are passionate about AI and privacy. We have worked on projects before that have been used by thousands of people such as this (I've never heard of Cleavr). There are real faces (real faces = Twitter account link?) behind the product. And come chat with us on our Discord server to know us better. This is much, much better than having no attribution, but it's miles away from being able to verify trust by reading the code. Would love to hear what your reasons against this are. Still thinking about trying it out, anyway...
- renewiltord 2y agoWhat the heck, this looks cool! How have I missed it. Gonna give it a whirl.
- Alifatisk 2y agoAny plans on adding this to Chocolatey for Windows download?
- bayesianbot 2y agoLooks cool even though closed source makes me wary. Trying to save Anthropic API key on Arch Linux doesn't do anything and there's a message "If you're experiencing problems saving API keys especially on Linux, contact Discord", if it's so common problem maybe you should have a link with possible fixes? Adding another Discord server and searching for answers for a question that clearly has been asked often enough feels like quite a hurdle for testing it out.
- jakobov 2y agoNice! Can you explain what you mean by "simulate training beyond the number of available tokens"? Why does using distillation from a larger model simulate training with more tokens?
- canyon289 2y agoHi, I work on the Gemma team (same as Alek opinions are my own). Essentially instead of tokens that are "already there" in text, the distillation allows us to simulate training data from a larger model
- suryabhupa 2y agoSurya here from the core Gemma team -- we can think of a distillation loss as learning to model the entire distribution of tokens that are likely to follow the prefix thus far, instead of only the token in the training example. If you do some back of the envelope calculations, we can see that learning to model a larger distribution yields many more bits of information to learn from.
- jakobov 2y agoGotcha. That makes sense. Thanks! What are the theories as to why this works better than training on a larger quantity of non-simulated tokens? Is it because the gradient from the non-simulated tokens is too noisy for a small model to model correctly?
- deleted 2y ago[deleted]
- rosslazer 2y agoAre there examples of the prompt or transcripts for the human testing?
- aubanel 2y agoIt's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!
- screye 2y agoWhat's the most obvious standouts? In my experience, smaller models tend to do well on benchmarks and fail at generalization. Phi-2 comes to mind.
- moffkalast 2y agoIt's multilingual. Genuinely. Compared my results with some people on reddit and the consensus is that the 27B is near perfect in a few obscure languages and likely perfect in most common ones. The 9B is not as good but it's still coherent enough to use in a pinch. It's literally the first omni-translation tool that actually works that you can run offline at home. I'm amazed that Google mentioned absolutely nothing about this in their paper.
- norealmet 2y ago[dead]
- jug 2y agoWow, that's very impressive and indeed a game changer. I've previously had trouble with various Scandinavian languages, but last I checked with was Llama 2 and I kind of gave up on it. I had expected we were going to need special purpose small models for these uses as a crutch, like SW-GPT3. So I guess Gemma 2 is going to become Gemini 2.0 in their truly large and closed variants then? Or is it the open version of Gemini 1.5?
- typpo 2y agoIf anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Gemma is that it doesn't include a "Sure, I can help you" preamble. It just gets right into the code, and follows it with an explanation. The training seems to emphasize response structure and ease of comprehension. Also, best to run evals that don't rely on rote memorization of public code... so please substitute with your personal tests :) [0] https://ollama.com/library/gemma2 https://ollama.com/library/gemma2 [1] https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo
- jakobov 2y agoHow much faster (in terms of the number of iterations to a given performance) is training from distillation?
- dongobread 2y agoThe knowledge distillation is very interesting but generating trillions of outputs from a large teacher model seems insanely expensive. Is this really more cost efficient than just using that compute instead for training your model with more data/more epochs?
- DebtDeflation 2y agoI'm also curious. It seems like 6 months ago everyone was afraid of "model collapse" but now synthetic training generation and teacher models are all the rage. Have we solved the problem of model collapse?
- astrange 2y agoModel collapse was basically a coping idea made up by artists who were hoping AI image generators would all magically destroy themselves at some point; I don't think it was ever considered likely to happen. It does seem to be true that clean data works better than low quality data.
- groby_b 2y agoYou're confusing it with data poisoning. Model collapse itself is(was?) a fairly serious research topic: https://arxiv.org/abs/2305.17493 https://arxiv.org/abs/2305.17493 We've by now reached a "probably not inevitable" - https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2404.01413 argues there's a finite upper bound to error - but I'd also point out that that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative. To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well. but it's probably fair to say current SOTA is still more or less "it's neither impossible nor inevitable".
- astrange 2y agoOh, no, they definitely believe both are going to happen and ChatGPT is just going to stop working because it'll see itself on the internet. It goes with the common belief that LLMs learn from what you type into them. > To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well. I think that will always be available, or at least, a dataset with the distribution you want will be available.
- mistercheph 2y agoPlaying with it, and I like how much I can influence it with a system prompt, llama3 reacts pretty mildly to any system prompts I've tried.
- smcleod 2y agoIt has a tiny context window of 8k, that thing will have the memory of a goldfish.
- Grimblewald 2y ago8K is a sizable window, sure larger 'exists' but also advertised context windows and functional context windows are not the same thing. I would rather a model that can 'only' handle 8k tokens but handles 8k as well as it handles 1k compared to a model that 'can' handle 32k, but realistically, output for contexts beyond 1k are garbage.
- smcleod 2y agoDeepseek Coder v2 and Qwen2 are both great at 32k context. Can’t tell the difference between those models at 8k and 32k fully utilised. The difference in quality between them and 8k models when doing codegen is night and day. Not to mention that many of the little 8k models also have sliding window at 4k which essentially makes them 4k models.
- Grimblewald 2y agoI agree, they're exceptional models, however this can not be said of all models that boast a large context window.
- xkgt 2y agoDo we know if Gemma models are fundamentally different from the ones hosted as Gemini? Gemini 1.5 flash seems to produce good results for the price and performance.
- raffraffraff 2y ago> We use the same data filtering techniques as Gemma 1. Specifically, we filter the pre- training dataset to reduce the risk of unwanted or unsafe utterances. Hmmm. I'd love to know what qualifies as "unsafe".
- imjonse 2y agoIt will refuse to describe the process of making napalm using only double entendres.
- viridian 2y agoI don't understand the point of this sort of censorship when I can go to google, ask how to make napalm, and get a million results telling me to dissolve styrofoam in gasoline. I've seen documentaries and science shows on cable TV that demonstrate basic facts like this, or how the IRA produced IEDs, or how molotov cocktails were made in the spanish civil war. The information is beyond easy to access, and has been for decades.
- imjonse 2y agoTrue. But an LLM product is closely associated with a single company and unlike a search engine which can claim it only shows you what is already available, the LLM will seem like it personally tells you something harmful. When they want to sell it as a helpful assistant that kind of behavior will undermine that goal. We saw all the bad press companies have got in recent years for all kinds of unintended AI outputs.
- rldjbpin 2y agofor me, the 2.5B model for gemma (now 1) was very interesting as that was the first major offering at this size level. for basic llm tasks that most people would use on their daily lives (simple rag on your own data), it did the job for the most part (unless you need a lot of context maybe). on paper the newer one shows significant improvement with slightly larger size, but i hope HumanEval regression is not going to matter for most people.
- sujumayas 2y agoWhen I used it with ollama in the terminal (first try prompt: "create a snake game in HTML canvas", nothing else) it went forever rambling. It started with the right answer in HTML code, but then it started explaining, and started repeating itself, and then it started to put things like random code snippets and random explanations that were nonsense like: ```python def solve_quadratic_equation(a, b, c): """Solves a quadratic equation of the form ax^2 + bx + c = 0.""" discriminant = (b ** 2) - (4 * a * a) if discriminant >= 0: root = (-b + math.sqrt(b ** 2 - 4 * a * a ** b** 0.5 # 1. # Return None if the quadratic equation has no real roots if (b ** 2) < (4 * c): return None # Calculate the roots using the quadratic formula b = -b b # a, b): Solve for the discriminant. # Handle the case of a complex discriminant # Print the solution to the equation if (b * 2) print("The quadratic equation is: " + a * x* 2 + b "x" + c) ```
- sujumayas 2y agoJust realized that Gemma2 is pretty bad in programming tasks. Lol.