7 ms·
Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
- brucethemoose2 3y agoNot to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
- throwaway19423 3y agoI am confused how all these things are able to interoperate. Are the creators of these models following the same IO for their models? Won't the tokenizer or token embedder be different? I am genuinely confused by how the same code works for so many different models.
- brucethemoose2 3y agoIt's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don't work with the hyper optimized llama runtumes. Also, there's a lot of magic going on behind the scenes with configs stored in gguf/huggingface format models, and the libraries that use them. There are different tokenizers, but they mostly follow the same standards.
- null_point 3y agoI found the magic! https://github.com/search?q=repo%3Aggerganov%2Fggml%20magic&type=code https://github.com/search?q=repo%3Aggerganov%2Fggml%20magic&...
- null_point 3y agoHey, c'mon now. Just being playful about the "magic" string used in GGUF files to detect that it is in-fact a GGUF file.
- jebarker 3y agoI doubt there'd be confusion as the names are totally different
- a1o 3y agoIf I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
- replete 3y agoI used gemm:2b with ollama last night and the model was around 1.3gb IIRC
- samus 3y agoDepends how much you quantize the model. For most general-purpose LLMs, the model completely dwarfs the size of the binary code.
- coder543 3y agohttps://ollama.com/library/gemma/tags https://ollama.com/library/gemma/tags You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional. The size of anything but the model is going to be rounding error compared to how large the models are, in this context.
- sorenjan 3y agoWhat's the use case of models this small? Can you use the "knowledge" encoded in them and ask them questions and get relevant answers, or are they used as text processors to summarize documents etc?
- trisfromgoogle 3y agoGemma 2B generation quality is excellent in my own very-biased opinion. I asked it to write a response to your comment: -- Large language models (LLMs) have achieved significant progress in recent years, with models like GPT-3 and LaMDA demonstrating remarkable abilities in various tasks such as language generation, translation, and question answering. However, 2b parameter models are a much smaller and simpler type of LLM compared to GPT-3. While they are still capable of impressive performance, they have a limited capacity for knowledge representation and reasoning. Despite their size, 2b parameter models can be useful in certain scenarios where the specific knowledge encoded in the model is relevant to the task at hand. For example: - Question answering: 2b parameter models can be used to answer questions by leveraging their ability to generate text that is similar to the question. - Text summarization: 2b parameter models can be used to generate concise summaries of documents by extracting the most important information. - Code generation: While not as common, 2b parameter models can be used to generate code snippets based on the knowledge they have learned. Overall, 2b parameter models are a valuable tool for tasks that require specific knowledge or reasoning capabilities. However, for tasks that involve general language understanding and information retrieval, larger LLMs like GPT-3 may be more suitable. -- Generated in under 1s from query to full response on together.ai https://api.together.xyz/playground/chat/google/gemma-2b-it https://api.together.xyz/playground/chat/google/gemma-2b-it
- brucethemoose2 3y ago...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
- Havoc 3y agoThat does seem to be the consensus unfortunately. Would have been better for everyone if google’s foray into open model a la FB made a splash
- brucethemoose2 3y agoYeah, especially with how much Google is hyping it. It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.
- Vetch 3y agoWas it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?
- brucethemoose2 3y agoI eval'd it with vllm. One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab. Many Chinese llms have a similar "default sampling" issue. But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.
- trisfromgoogle 3y agoAny chance you can share more details on your measurement setup and eval protocols? You're likely seeing some config snafus, which we're trying to track down.
- colesantiago 3y agoIsn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI. Has this perception changed or pretty much the same?
- beoberha 3y agoGemini - maybe, though I find it pretty unlikely it’ll happen anytime soon. Not sure what you mean about Gemma considering it’s not a service. You can download the model weights and the inference code is on GitHub. Everything is local!
- brucethemoose2 3y agoThis is not necessarily a production backend, as it mentions in the readme. There are some very interesting efforts in JAX/TPU land like https://github.com/erfanzar/EasyDeL https://github.com/erfanzar/EasyDeL
- ertgbnm 3y agoThe weights are downloadable so there isn't much of a risk if Google stops hosting Gemma apart from the fact that it won't get new versions that you swap out in the future.
- cyanydeez 3y agoeven if there's a new model, I'm not seeing how these models provide any reliability metric. if you figure out a money making software/service, you're gonna be tied to that model to some significant degree.
- brokensegue 3y agodoes anyone have stats on cpu only inference speed with this?
- austinvhuang 3y agoany particular hardware folks are most interested in?
- brokensegue 3y agoI'm just looking for ballpark figures. Maybe a common aws instance type
- notum 3y agoNot sure if this is of any value to you, but Ryzen 7 generates 2 tokens per second for the 7B-Instruct model. The model itself is very unimpressive and I see no reason to play with it over the worst alternative from Hugging Face. I can only imagine this was released for some bizarre compliance reasons.
- brokensegue 3y agothe metrics suggest it's much better than that
- janwas 3y agoFor the 7B IT and a short factual query I see 5.3 tps on a 5 year old Skylake Gold 6154 CPU @ 3.00GHz, 16 threads. Expect a slight increase as we improve scalability. FYI using the NUQ (4.5-bit) quantization improves throughput by about 1.4x.
- austinvhuang 3y agoHi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? gemma.cpp is a direct implementation of gemma, in its current form it's aimed at experimentation + research and portability + easy modifiable rather than a general purpose deployment framework. - this initial implementation is cpu simd centric. we're exploring options for portable gpu support but the cool thing is it will build and run on a lot of environments you might not expect an llm to run, so long as you have the memory to load the model. - I'll let other colleagues answer questions about the Gemma model itself, this is a C++ implementation of the model, but relatively independent of the model training process. - Although this is from Google, we're a very small team that wanted such a codebase to exist. We have lots of plans to use it ourselves and we hope other people like it and find it useful. - I wrote a twitter thread on this project here: https://twitter.com/austinvhuang/status/1760375890448429459 https://twitter.com/austinvhuang/status/1760375890448429459
- rgbrgb 3y agoThanks for releasing this! What is your use case for this rather than llama.cpp? For the on-device AI stuff I mostly do, llama.cpp is better because of GPU/metal offloading.
- austinvhuang 3y agollama.cpp is great, if it fit your needs you can use it. I think at this point llama.cpp is effectively a platform that's hardened for production. In its current form, I think of gemma.cpp is more of a direct model implementation (somewhere between the minimalism of llama2.c and the generality of ggml). I tend to think of 3 modes of usage: - hacking on inference internals - there's very little indirection, no IRs, the model is just code, so if you want to add support for your own runtime support for sparsity/quantization/model compression/etc. and demo it working with gemma, there's minimal barriers to do so - implementing experimental frontends - i'll add some examples of this in the very near future. but you're free to get pretty creative with terminal UIs, code that interact with model internals like the KV cache, accepting/rejecting tokens etc. - interacting with the model locally with a small program - of course there's other options for this but hopefully this is one way to play with gemma w/ minimal fuss.
- einpoklum 3y agoCome on Dejiko, we don't have time for this gema! https://www.youtube.com/watch?v=9FSAqDVZHhU https://www.youtube.com/watch?v=9FSAqDVZHhU
- a-french-anon 3y agoGlad I wasn't alone.
- einpoklum 3y agoWell, it was just so nostalgic for me nyo :-\
- sillysaurusx 3y agoEvery time I see Gemma all I hear is Jubei screaming Genmaaaa since the n is almost silent. https://youtu.be/TFR9-cZecWo?si=rMED2LEh-fssHeeG https://youtu.be/TFR9-cZecWo?si=rMED2LEh-fssHeeG
- austinvhuang 3y agolol
- ofermend 3y agoAwesome work on getting this done so quickly. We just added Gemma to the HHEM leaderboard - https://huggingface.co/spaces/vectara/leaderboard https://huggingface.co/spaces/vectara/leaderboard, and as you can see there its doing pretty good in terms of low hallucination rate, relative to other small models.
- swozey 3y ago> LLM hallucinations I wasn't familiar with the term, good article - https://masterofcode.com/blog/hallucinations-in-llms-what-you-need-to-know-before-integration https://masterofcode.com/blog/hallucinations-in-llms-what-yo...
- ed 3y agoKarpathy offers a more concise (and whimsical) explanation https://x.com/karpathy/status/1733299213503787018 https://x.com/karpathy/status/1733299213503787018
- swozey 3y agoThe velocity of the LLM open source ecosystem is absolutely insane. I just got into hobby projects with diffusion a week ago and I'm seeing non-stop releases. It's hard to keep up. It's a firehose of information, acronyms, code etc. It's been a great python refresher.
- deleted 3y ago[deleted]
- austinvhuang 3y agoDon't be discouraged, you don't have to follow everything. In fact it's probably better to dive deep into one hobby project like you're doing than constantly context switch with every little news item that comes up. While working on gemma.cpp there were definitely a lot of "gee i wish i could clone myself and work on that other thing too".
- next_xibalba 3y agoIs this neutered in the way Gemini is (i.e. is the "censorship" built in) or is that a "feature" of the Gemini application?
- deleted 3y ago[deleted]
- ComputerGuru 3y agoIt depends on the model you load/use, the team released both censored and "PT" versions.
- jonpo 3y agoThese models (Gemma) are very difficult to jailbreak.
- throwaway19423 3y agoCan any kind soul explain the difference between GGUF, GGML and all the other model packaging I am seeing these days? Was used to pth and the thing tf uses. Is this all to support inference or quantization? Who manages these formats or are they brewing organically?
- austinvhuang 3y agoI think it's mostly an organic process arising from the ecosystem. My personal way of understanding it is this - the original sin of model weight format complexity is that NNs are both data and computation. Representing the computation as data is the hard part and that's where the simplicity falls apart. Do you embed the compute graph? If so, what do you do about different frameworks supporting overlapping but distinct operations. Do you need the artifact to make training reproducible? Well that's an even more complex computation that you have to serialize as data. And so on..
- moffkalast 3y agoIt's all mostly just inference, though some train LoRAs directly on quantized models too. GGML and GGUF are the same thing, GGUF is the new version that adds more data about the model so it's easy to support multiple architectures, and also includes prompt templates. These can run CPU only, be partially or fully offloaded to a GPU. With K quants, you can get anywhere from a 2 bit to an 8 bit GGUF. GPTQ was the GPU-only optimized quantization method that was superseded by AWQ, which is roughly 2x faster and now by EXL2 which is even better. These are usually only 4 bit. Safetensors and pytorch bin files are raw float16 model files, these are only really used for continued fine tuning.
- Gracana 3y ago> and also includes prompt templates That sounds very convenient. What software makes use of the built-in prompt template?
- moffkalast 3y agoOf the ones I commonly use, I've only seen it read by text-generation-webui, in the GGML days it had a long hardcoded list of known models and which templates they use so they could be auto-selected (which was often wrong), but now it just grabs it from any model directly and sets it when it's loaded.
- Wissan 3y ago[flagged]
- Wissan 3y ago[dead]
- sintax 3y agoDemo when model quantized to q0_K?
- tarruda 3y agoIs it not possible to add Gemma support on Llama.cpp?
- austinvhuang 3y agoGemma support has been added to llama.cpp, in fact it was added almost immediately after the release: https://twitter.com/ggerganov/status/1760293079313973408 https://twitter.com/ggerganov/status/1760293079313973408 However, be aware that there were some quality issues with quantization initially (hopefully they're resolved but i haven't followed too closely): https://twitter.com/ggerganov/status/1760418864418934922 https://twitter.com/ggerganov/status/1760418864418934922
- zoogeny 3y agoI know a lot of people chide Google for being behind OpenAI in their commercial offerings. We also dunk on them for the over-protective nature of their fine-tuning. But Google is scarily capable on the LLM front and we shouldn't count them out. OpenAI might have the advantage of being quick to move, but when the juggernaut gets passed its resting inertia and starts to gain momentum it is going to leave an impression. That became clear to me after watching the recent Jeff Dean video [1] which was posted a few days ago. The depth of institutional knowledge that is going to be unlocked inside Google is actually frightening for me to consider. I hope the continued competition on the open source front, which we can really thank Facebook and Llama for, keeps these behemoths sharing. As OpenAI moves further from its original mission into capitalizing on its technological lead, we have to remember why the original vision they had is important. So thank you, Google, for this. 1. https://www.youtube.com/watch?v=oSCRZkSQ1CE&ab_channel=RiceKenKennedyInstitute https://www.youtube.com/watch?v=oSCRZkSQ1CE&ab_channel=RiceK...
- whimsicalism 3y agoRealistically, if Google has all this talent, they should have gotten the juggernaut moving in 2020. Google has had years to get to this stage, and they've lost a lot of the talent that made their initial big splashes to OAI and competitors. Try finding someone on a sparse MoE paper from Google prior to 2022 who is still working there and not at OAI. With respect, they can hardly even beat Mistral, resorting to rounding down a 7.8b model (w/o embeddings) to 7b.
- freedomben 3y agoOrganizational dysfunction can squash/squander even the most talented engineers. Especially in a big org in big tech. My bet is that their inability to deliver before is probably a result of non-comittal funders/decision makers, product whiplash, corporate politics, and other non-technical challenges. Google has been the home of the talent for many years. They came on my radar in the late 00s when I used Peter Norvig's textbook in college, and they hired Ray Kurzweil in like 2012 or 2013 IIRC. They were hiring ML PhDs with talent for many years, and they pioneered most of the major innovations. They just got behind on productizing and shipping.
- kwantaz 3y agonice
- namtranase 3y agoThank the team for the awesome repo. I have navigated gemma.cpp and run it from the first day, it is smooth in my view. So I hope gemma.cpp will continue to add cool features (something like k-quants, server,...) so it can serve more widely. Actually, I have developed a Python wrapper for it: https://github.com/namtranase/gemma-cpp-python https://github.com/namtranase/gemma-cpp-python The purpose is to use easily and update every new technique from gemma.cpp team.
- austinvhuang 3y agoNice, this is really cool to see! There were other threads that expressed interest in something like this.
- dontupvoteme 3y agoAt the risk of being snarky, it's interesting that Llama.cpp was a 'grassroots' effort originating from a Bulgarian hacker google now launches a corporatized effort inspired by it. I wonder if there's some analogies to the 80s or 90s in here.
- trisfromgoogle 3y agoTo be clear, this is not comparable directly to llama.cpp -- Gemma models work on llama.cpp and we encourage people who love llama.cpp to use them there. We're also launched with Ollama. Gemma.cpp is a highly optimized and lightweight system. The performance is pretty incredible on CPU, give it a try =)
- alekandreev 3y agoAs a fellow Bulgarian from the 80s and 90s myself, and now a part of the Gemma team, I’d say Austin, Jan, and team very much live up to the ethos of hackers I'd meet on BBSes back then. :) They are driven entirely by their own curiosity and a desire to push computers to the limit. Combined with their admirable low-level programming skills, you get a very solid, fun codebase, that they are sharing with the world.
- natch 3y agoApart from the fact that they are different things, since they came out of the same organization I think it’s fair to ask: Do these models have the same kind of odd behavior as Gemini?
- xrd 3y agoI was discussing LLMs with a non technical person on the plane yesterday. I was explaining why LLMs aren't good at math. And, he responded, no, chatgpt is great a multivariate regression, etc. I'm using LLMs locally almost always and eschewing API backed LLMs like chatgpt. So I'm not very familiar with plugins, and I'm assuming chatgpt plugs into a backend when it detects a math problem. So it isn't the LLM doing the math but to the user it appears to be. Does anyone here know what LLM projects like llama.cpp or gemma.cpp support a plugin model? I'm interested in adding to the dungeons and dragons system I built using llama.cpp. Because it doesn't do math well, the combat mode is terrible. But I was writing my own layer to break out when combat mode occurs, and I'm wondering if there is a better way with some kind of plugin approach.
- staticman2 3y agoSillytavern is a front end for local and cloud models. They have a simple scripting language and there's been some experiments with adding game functionality with it: https://chub.ai/characters/creamsan/team-neko-e4f1b2f8 https://chub.ai/characters/creamsan/team-neko-e4f1b2f8 This one says it uses javascript as well: https://chub.ai/characters/creamsan/tessa-c4b917f9 https://chub.ai/characters/creamsan/tessa-c4b917f9 Thise are the only two listed as SFW. There's some others if you hit the nsfw toggle and search for the scripted tag.I don't know if this is the right approach but you could also write a module for Sillytavern Extras.
- manlobster 3y agoI wonder why they didn't use bazel as their build system.
- jvolkman 3y agoLooks like they're working on it: https://github.com/google/gemma.cpp/issues/16 https://github.com/google/gemma.cpp/issues/16
- f46f7ab7de71 3y ago[dead]
- olegbask 3y agoIt would be amazing to add support for M1 aka Metal: I was able to run Q8 version with llama.cpp and it's blazingly fast. The problem: I don't know how much accuracy it loses and https://huggingface.co/google/gemma-2b-it/tree/main https://huggingface.co/google/gemma-2b-it/tree/main takes too much memory which results in OOMs. Do you have any estimates on getting Metal support similar to how llama.cpp works? Why `.gguf` files are so giant compared to `.sbs`? Is it just because they use fp32?
- ManasRao 3y agoHello Austin, I would like to inquire about the accessibility of the code and weights for the Gemma Model. Is this information publicly available?
- austinvhuang 3y agoHi, haven't followed this thread for a while so just happened to see this now. I'm assuming you mean in other languages/implementations? (since the gemma.cpp repo linked above has code + links for gemma.cpp specific weights) If so, you can find the weights here https://www.kaggle.com/models/google/gemma https://www.kaggle.com/models/google/gemma - each of the "model variations" (flax, jax, pytorch, keras, etc.) has a download for the weights and links to its code. If you're comfortable with flax, that's DM's own reference implementation: https://github.com/google-deepmind/gemma https://github.com/google-deepmind/gemma