3 ms·
Run Google Gemma 2 2B 100% Locally
Google just released Gemma 2 2B and you can run it on Mac (and other devices), 100% local, powered by llama.cpp!
1. brew install llama.cpp
2. ./llama-cli
--hf-repo google/gemma-2-2b-it-GGUF \
--hf-file 2b_it_v2.gguf \
-p "Write a poem about cats as a labrador" -cnv
With this, I created a Local RAG Knowledge and Answering System with Google Gemma 2 2B and Marqo. Check it out: https://github.com/ellie-sleightholm/marqo-google-gemma2
Link also in the comments!
- esleightholm 2y agohttps://github.com/ellie-sleightholm/marqo-google-gemma2 https://github.com/ellie-sleightholm/marqo-google-gemma2
- kozak 2y agoI tried this code, and it cannot download the model from https://huggingface.co/google/gemma-2-2b-it-GGUF/resolve/main/2b_it_v2.gguf https://huggingface.co/google/gemma-2-2b-it-GGUF/resolve/mai... without authentication.
- esleightholm 2y agoThere are 2 things you will need to do. 1. You will need to request access to the model in HuggingFace and accept the license. Head to https://huggingface.co/google/gemma-2-2b-it-GGUF/tree/main https://huggingface.co/google/gemma-2-2b-it-GGUF/tree/main and there should be an option to request access. This will be approved almost immediately and you will receive an email saying you've been granted access. 2. Create a User Access Token in HuggingFace to download the model. Visit https://huggingface.co/settings/tokens https://huggingface.co/settings/tokens and create a new token. Then, set the token in your environment by running: `export HF_TOKEN=<your_huggingface_token>` Hope that helps! Any further issues, feel free to reply to this comment and I'd be happy to help.
- schappim 2y agoWhat is the advantage of this over running: ollama run gemma2:2b ?
- umtksa 2y agoand I tried both of them and ollama some how handled everything better for gemma2
- mappu 2y agoollama is a thin wrapper over llama.cpp, so i'd pose the opposite question - what does ollama give you over using llama.cpp directly?
- schappim 2y agoModel management, customisable HTTP APIs, monitoring, security features, "parallel requests" (batch processing), no requirement for HF auth etc...
- yjftsjthsd-h 2y agoEase of use. Rather like arduino or like docker vs chroot/jails/zones, there's nothing wrong with just using the underlying tech, but lowering friction has value.
- stavros 2y agoWhat's new about this? LM studio runs much more powerful models with a few clicks, in a nice interface. Ollama does the same thing well on the cli.
- vineyardmike 2y agoThe new thing added here is a disguised advertisement by the head of developer relations (of the database) :)
- schappim 2y agoGood pickup: [1] https://www.linkedin.com/in/elliesleightholm/?original_referer=https%3A%2F%2Fwww%2Egoogle%2Ecom%2F&originalSubdomain=uk https://www.linkedin.com/in/elliesleightholm/?original_refer...
- mysore 2y ago[flagged]