6 ms·

This covers three things: Llama.cpp (Mac/Windows/Linux), Ollama (Mac), MLC LLM (iOS/Android) Which is not really comprehensive... If you have a lin

This covers three things: Llama.cpp (Mac/Windows/Linux), Ollama (Mac), MLC LLM (iOS/Android) Which is not really comprehensive... If you have a linux machine with GPUs, i'd just use hugging face inference (https://github.com/huggingface/text-generation-inference https://github.com/huggingface/text-generation-inference). And I am sure there are other things that could be covered.

Patrick_Devine 3y ago
Ollama works with Windows and Linux as well too, but doesn't (yet) have GPU support for those platforms. You have to compile it yourself (it's a simple `go build .`), but should work fine (albeit slow). The benefit is you can still pull the llama2 model really easily (with `ollama pull llama2`) and even use it with other runners. DISCLAIMER: I'm one of the developers behind Ollama.
- mschuster91 3y ago
  > DISCLAIMER: I'm one of the developers behind Ollama. I got a feature suggestion - would it be possible to have the ollama CLI automatically start up the GUI/daemon if it's not running? There's only so much stuff one can keep in a Macbook Air's auto start.
  - jmorgan 3y ago
    Good suggestion! This is definitely on the radar, so that running `ollama` will start the server when it's needed (instead of erroring!): https://github.com/jmorganca/ollama/issues/47 https://github.com/jmorganca/ollama/issues/47
- DennisP 3y ago
  I've been wondering, is the M2's neural engine usable for this?
  - npsomaratna 3y ago
    I think you'd need to offload the model into CoreML to do so, right? My understanding is that none of the current popular inference frameworks do this (not yet, at least).
    - DennisP 3y ago
      I did find some support from Apple: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
- sanex 3y ago
  Matt Williams talk today at that conference was a great intro to Ollama.
krisoft 3y ago
> If you have a linux machine with GPUs How much VRAM one needs to run inference with llama 2 on a GPU approximately?
- lolinder 3y ago
  Depends on which model. I haven't bothered doing it on my 8GB because the only model that would fit is the 7B model quantized to 4 bits, and that model at that size is pretty bad for most things. I think you could have fun with 13B with 12GB VRAM. The full size model would require >35GB even quantized.
- novaRom 3y ago
  16Gb is minimum to run 7B model with float16 weights; out of the box, with no further efforts.
- zacmps 3y ago
  70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB). Quantization still seems to have some issues for the 70B model.
lolinder 3y ago
Just a note that you have to have at least 12GB VRAM for it to be worth even trying to use your GPU for LLaMA 2. The 7B model quantized to 4 bits can fit in 8GB VRAM with room for the context, but is pretty useless for getting good results in my experience. 13B is better but still not anything near as good as the 70B, which would require >35GB VRAM to use at 4 bit quantization. My solution for playing with this was just to upgrade my PC's RAM to 64GB. It's slower than the GPU, but it was way cheaper and I can run the 70B model easily.
- dc443 3y ago
  I have 2x 3090 do you know if it's feasible to use that 48GB total for running this?
  - eurekin 3y ago
    Yes, it runs totally fine. I ran it in Oobabooga/text generation web ui. Nice thing about it is that it autodownloads all necessary gpu binaries on it's own and creates a isolated conda env. I asked same questions on the official 70b demo and got same answers. I even got better answers with ooba, since the demo cuts text early Ooobabooga: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui Model: TheBloke_Llama-2-70B-chat-GPTQ from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ ExLlama_HF loader gpu split 20,22, context size 2048 on the Chat Settings tab, choose Instruction template tab and pick Llama-v2 from the instruction template dropdown Demo: https://huggingface.co/blog/llama2#demo https://huggingface.co/blog/llama2#demo
    - zakki 3y ago
      Is there any specific settings to make 2x3090 work together?
      - kwerk 3y ago
        Interested in this too
robotnikman 3y ago
Llama.cpp has been fun to experiment around with. I was suprised with how easy it was to set up, much easier than when I tried to set up a local llm almost a year ago.
lhl 3y ago
As an article aimed towards newbies I think it's pretty good, but I'd agree "comprehensive" is a bit of a reach. I think a better title would have been something like "Get LLama2 running locally on every platform easily." Anyone w/ more than a single consumer GPU probably has a good grip on their options (a vllm vs hf shootout would be neat for exmaple), but I'd add a few more projects for those taking the next step for local inferencing: * exllama - while llama.cpp has matched its token generation performance, exllama is still largely my preferred inference engine because it is so memory efficient (shaving gigs off the competition) - this means you can run a 33B model w/ 2K context easily on a single 24GB card. It also scales almost perfectly for inferencing on 2 GPUs. It's been tested to run a llama2-70b w/ 16K context (NTK RoPE scaling) sneaking in at 47GB. Ridiculous. * AutoGPTQ - while it's fallen a bit behind for inference, if you are using an older (eg Pascal) cards, it's worth taking a look. It's also the easiest tool for making GPTQ quants. * ggml - for non-llama models, this covers running quants of almost everything else out there; GPU acceleration only w/ cuBLAS/CLBlast, so not the fastest, but fast enough * python-llama-cpp and LocalAI - while these are technically llama.cpp bindings, they're pretty useful/worth mentioning since they replicate the OpenAI API making it easy as a drop-in replacement for a whole ecosystems of tools/apps * A lot of hobbyists like oobabooga, kobold.cpp, silly tavern, etc but I haven't gotten around to poking into those as much. They seem like a lot of work, always behind their mainline dependencies, and while featureful and interesting, they also feel like they're one update away from breaking (eg, automatic1111 vibes).

Patrick_Devine 3y ago

Ollama works with Windows and Linux as well too, but doesn't (yet) have GPU support for those platforms. You have to compile it yourself (it's a simple `go build .`), but should work fine (albeit slow). The benefit is you can still pull the llama2 model really easily (with `ollama pull llama2`) and even use it with other runners. DISCLAIMER: I'm one of the developers behind Ollama.

mschuster91 3y ago
> DISCLAIMER: I'm one of the developers behind Ollama. I got a feature suggestion - would it be possible to have the ollama CLI automatically start up the GUI/daemon if it's not running? There's only so much stuff one can keep in a Macbook Air's auto start.
- jmorgan 3y ago
  Good suggestion! This is definitely on the radar, so that running `ollama` will start the server when it's needed (instead of erroring!): https://github.com/jmorganca/ollama/issues/47 https://github.com/jmorganca/ollama/issues/47
DennisP 3y ago
I've been wondering, is the M2's neural engine usable for this?
- npsomaratna 3y ago
  I think you'd need to offload the model into CoreML to do so, right? My understanding is that none of the current popular inference frameworks do this (not yet, at least).
  - DennisP 3y ago
    I did find some support from Apple: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
sanex 3y ago
Matt Williams talk today at that conference was a great intro to Ollama.

krisoft 3y ago

> If you have a linux machine with GPUs How much VRAM one needs to run inference with llama 2 on a GPU approximately?

lolinder 3y ago
Depends on which model. I haven't bothered doing it on my 8GB because the only model that would fit is the 7B model quantized to 4 bits, and that model at that size is pretty bad for most things. I think you could have fun with 13B with 12GB VRAM. The full size model would require >35GB even quantized.
novaRom 3y ago
16Gb is minimum to run 7B model with float16 weights; out of the box, with no further efforts.
zacmps 3y ago
70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB). Quantization still seems to have some issues for the 70B model.

lolinder 3y ago

Just a note that you have to have at least 12GB VRAM for it to be worth even trying to use your GPU for LLaMA 2. The 7B model quantized to 4 bits can fit in 8GB VRAM with room for the context, but is pretty useless for getting good results in my experience. 13B is better but still not anything near as good as the 70B, which would require >35GB VRAM to use at 4 bit quantization. My solution for playing with this was just to upgrade my PC's RAM to 64GB. It's slower than the GPU, but it was way cheaper and I can run the 70B model easily.

dc443 3y ago
I have 2x 3090 do you know if it's feasible to use that 48GB total for running this?
- eurekin 3y ago
  Yes, it runs totally fine. I ran it in Oobabooga/text generation web ui. Nice thing about it is that it autodownloads all necessary gpu binaries on it's own and creates a isolated conda env. I asked same questions on the official 70b demo and got same answers. I even got better answers with ooba, since the demo cuts text early Ooobabooga: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui Model: TheBloke_Llama-2-70B-chat-GPTQ from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ ExLlama_HF loader gpu split 20,22, context size 2048 on the Chat Settings tab, choose Instruction template tab and pick Llama-v2 from the instruction template dropdown Demo: https://huggingface.co/blog/llama2#demo https://huggingface.co/blog/llama2#demo
  - zakki 3y ago
    Is there any specific settings to make 2x3090 work together?
    - kwerk 3y ago
      Interested in this too

robotnikman 3y ago

Llama.cpp has been fun to experiment around with. I was suprised with how easy it was to set up, much easier than when I tried to set up a local llm almost a year ago.

lhl 3y ago

As an article aimed towards newbies I think it's pretty good, but I'd agree "comprehensive" is a bit of a reach. I think a better title would have been something like "Get LLama2 running locally on every platform easily." Anyone w/ more than a single consumer GPU probably has a good grip on their options (a vllm vs hf shootout would be neat for exmaple), but I'd add a few more projects for those taking the next step for local inferencing: * exllama - while llama.cpp has matched its token generation performance, exllama is still largely my preferred inference engine because it is so memory efficient (shaving gigs off the competition) - this means you can run a 33B model w/ 2K context easily on a single 24GB card. It also scales almost perfectly for inferencing on 2 GPUs. It's been tested to run a llama2-70b w/ 16K context (NTK RoPE scaling) sneaking in at 47GB. Ridiculous. * AutoGPTQ - while it's fallen a bit behind for inference, if you are using an older (eg Pascal) cards, it's worth taking a look. It's also the easiest tool for making GPTQ quants. * ggml - for non-llama models, this covers running quants of almost everything else out there; GPU acceleration only w/ cuBLAS/CLBlast, so not the fastest, but fast enough * python-llama-cpp and LocalAI - while these are technically llama.cpp bindings, they're pretty useful/worth mentioning since they replicate the OpenAI API making it easy as a drop-in replacement for a whole ecosystems of tools/apps * A lot of hobbyists like oobabooga, kobold.cpp, silly tavern, etc but I haven't gotten around to poking into those as much. They seem like a lot of work, always behind their mainline dependencies, and while featureful and interesting, they also feel like they're one update away from breaking (eg, automatic1111 vibes).