3 ms·
llama.cpp offers compiled releases for multiple platforms. This release has the new vision features: https://github.com/ggml-org/llama.cpp/releases/tag/b5332 ht
by simonw 2y ago
llama.cpp offers compiled releases for multiple platforms. This release has the new vision features: https://github.com/ggml-org/llama.cpp/releases/tag/b5332 https://github.com/ggml-org/llama.cpp/releases/tag/b5332
On macOS I downloaded the llama-b5332-bin-macos-arm64.zip file and then had to run this to get it to work:
unzip llama-b5332-bin-macos-arm64.zip
cd build/bin
sudo xattr -rd com.apple.quarantine llama-server llama-mtmd-cli *.dylib
Then I could run the interactive terminal (with a 3.2GB model download) like this (borrowing from https://news.ycombinator.com/item?id=43943370R https://news.ycombinator.com/item?id=43943370R)
./llama-mtmd-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
Or start the localhost 8080 web server (with a UI and API) like this:
./llama-server -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
I wrote up some more detailed notes here: https://simonwillison.net/2025/May/10/llama-cpp-vision/ https://simonwillison.net/2025/May/10/llama-cpp-vision/
- ngxson 2y agoFor brew users, you can specify --HEAD when installing the package. This way, brew will automatically build the latest master branch. Btw, the brew version will be updated in the next few hours, so after that you will be able to simply "brew upgrade llama.cpp" and you will be good to go!
- ngxson 2y agoAnd btw, -ngl is automatically set to max value now, you don't need to -ngl 99 anymore! Edit: sorry this is only true on Metal. For CUDA or other GPU backends, you still need to manually specify -ngl
- danielhanchen 2y agoOH WHAT! So just -ngl? Oh also do you know if it's possible to auto do 1 GPU then the next (ie sequential) - I have to manually set --device CUDA0 for smallish models, and probs distributing it amongst say all GPUs causes communication overhead!
- ngxson 2y agoAh no I mean we can omit the whole "-ngl N" argument for now, as it is internally set to -1 by default in CPP code (instead of being 0 traditionally), and -1 meaning offload everything to GPU I have no idea how to specify custom layer specs with multi GPU, but that is interesting!
- danielhanchen 2y agoWAIT so GPU offloading is on by DEFAULT? Oh my fantastic! For now I have to "guess" via a Python script - ie I sum sum up all the .gguf split files in filesize, then detect CUDA memory usage, and specify approximately how many GPUs ie --device CUDA0,CUDA1 etc
- ngxson 2y agoAhhh no sorry I forgot that the actual code controlling this is inside llama-model.cpp ; sorry for the misinfo, the -ngl only set to max by default if you're using Metal backend (See the code in side llama_model_default_params())
- danielhanchen 2y agoOh no worries! I re-edited my comment to account for it :)
- danielhanchen 2y agoI'm also extremely pleased with convert_hf_to_gguf.py --mmproj - it makes quant making much simpler for any vision model! Llama-server allowing vision support is definitely super cool - was waiting for it for a while!