4 ms·

You can run it on a Macbook M1/M2 with 64GB of RAM.

by moyix 3y ago

2OEH8eoCRo0 3y ago
How? It's larger than 64GB.
- coder543 3y ago
  Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
  - 2OEH8eoCRo0 3y ago
    I'm aware but is it still LLaMA 70B at that point?
    - coder543 3y ago
      Yes. Quantization does not reduce the number of parameters. It does not re-train the model.
    - deleted 3y ago
      [deleted]
    - andy99 3y ago
      It's a legit question, the model will be worse in some way... I've seen it discussed that all things being equal more parameters is better (meaning it's better to take a big model and quantized it to fit in memory than use a smaller unquantized model that fits), but a quantized model wouldn't be expected to run identically to or as well as the full model.
      - coder543 3y ago
        You don’t stop being andy99 just because you’re a little tired, do you? Being tired makes everyone a little less capable at most things. Sometimes, a lot less capable. In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program. If we’re not willing to declare that you are a completely separate person when you’re tired, or that 32-bit and 64-bit versions are completely different programs, then I don’t think it’s worth getting overly philosophical about quantization. A quantized model is still the same model. The quality loss from using 4+ bit quantization is minimal, in my experience. Yes, it has a small impact on accuracy, but with massive efficiency gains. I don’t really think anyone should be running the full models outside of research in the first place. If anything, the quantized models should be considered the “real” models, and the full fp16/fp32 model should just be considered a research artifact distinct from the model. But this philosophical rabbit hole doesn’t seem to lead anywhere interesting to me. Various papers have shown that 4-bit quantization is a great balance. One example: https://arxiv.org/pdf/2212.09720.pdf https://arxiv.org/pdf/2212.09720.pdf
        cjbprime 3y ago
        I don't like the metaphor: when I'm tired, I will be alert again later. Quantization is lossy compression: the human equivalent would be more like a traumatic brain injury affecting recall, especially of fine details. The question of whether I am still me after a traumatic brain injury is philosophically unclear, and likely depends on specifics about the extent of the deficits.
        coder543 3y ago
        The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the analogy out further, if the model was able to get an A on a test before quantization, it would likely still get a B at worst afterwards, given a drop in the score of less than 10%. Depending on the task, the measured impact could even be negligible. It’s far more similar to the model being perpetually tired than it is to a TBI. You may nitpick the analogy, but analogies are never exact. You also ignore the other piece that I pointed out, which is how we treat other software that comes in multiple slightly different forms.
    - manmal 3y ago
      Sure, quantization reduces information stored for each parameter, not the parameter count.
  - sbrother 3y ago
    Can I ask how many tok/s you're getting on that setup? I'm trying to decide whether to invest in a high-end NVIDIA setup or a Mac Studio with llama.cpp for the purposes of running LLMs like this one locally.
    - coder543 3y ago
      On a 33B model at q4_0 quantization, I’m seeing about 36 tokens/s on the RTX 3090 with all layers offloaded to the GPU. Mixtral runs at about 43 tokens/s at q3_K_S with all layers offloaded. I normally avoid going below 4-bit quantization, but Mixtral doesn’t seem phased. I’m not sure if the MoE just makes it more resilient to quantization, or what the deal is. If I run it at q4_0, then it runs at about 24 tokens/s, with 26 out of 33 layers offloaded, which is still perfectly usable, but I don’t usually see the need with Mixtral. Ollama dynamically adjusts the layers offloaded based on the model and context size, so if I need to run with a larger context window, that reduces the number of layers that will fit on the GPU and that impacts performance, but things generally work well.
      - sbrother 3y ago
        Thanks! That is really fast for personal use.
      - sorenjan 3y ago
        What's the power consumption and fan noise like when doing that? I assume you're running the model doing inference in the background for the whole coding session, i.e. hours at a time?
        coder543 3y ago
        I don’t use local LLMs for CoPilot-like functionality, but I have toyed with the concept. There are a few things to keep in mind: no programmer that I know is sitting there typing code for hours at a time without stopping. There’s a lot more to being a developer than just typing, whether it is debugging, thinking, JIRA, Slack, or whatever else. These CoPilot-like tools will only activate after you type something, then stop for a defined timeout period. While you’re typing, they do nothing. After they generate, they do nothing. I would honestly be surprised if the GPU active time was more than 10% averaged over an hour. When actively working on a large LLM, the RTX 3090 is drawing close to 400W in my desktop. At a 10% duty cycle (active time), that would be 40W on average, which would be 320Wh over the course of a full 8-hour day of crazy productivity. My electric rate is about 15¢/kWh, so that would be about 5¢ per day. It is absolutely not running at a 100% duty cycle, and it’s absurd to even do the math for that, but we can multiply by 10 and say that if you’re somehow a mythical “10x developer” then it would be 50¢/day in electricity here. I think 5¢/day to 10¢/day is closer to reality. Either way, the cost is marginal at the scale of a software developer’s salary.
- rgbrgb 3y ago
  Quantization can take it under 30GB (with quality degradation). For example, take a look at the GGUF file sizes here: https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF
reddit_clone 3y ago
I am not too familiar with LLMs and GPUs (Not a gamer either). But want to learn. Could you please expand on what else would be capable of running such models locally? How about a linux laptop/desktop with specific hardware configuration?
- MeImCounting 3y ago
  It pretty much comes down to 2 factors which is memory bandwidth and compute. You need a high enough memory bandwidth to be able to "feed" the compute and you need beefy enough compute to be able to keep up with the data that is being fed in by the memory. In theory a single Nvidia 4090 would be able to run a 70b model with quantization at "useable" speeds. The reason mac hardware is so capable in AI is because of the unified architecture meaning the memory is shared across the GPU and CPU. There are other factors but it essentially comes down to tokens per second advantages. You could run one of these models on an old GPU with low memory bandwidth just fine but your tokens per second would be far too slow for what most people consider "useable" and the quantization necessary might star noticeably effecting the quality.
  - int_19h 3y ago
    A single RTX 4090 can run at most 34b models with 4-bit quantization. You'd need 2-bit for 70b, and at that point quality plummets. Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.
    - MeImCounting 3y ago
      Thats a good point. For example both the RTX 4090 and the RTX 6000 Ada Generation use the AD102 chip. The RTX 6000 Ada though, would be able to run 70b models due to the larger memory pool despite having the same memory interface width.
      - 3y ago

You can run it on a Macbook M1/M2 with 64GB of RAM.

by moyix 3y ago

You can run it on a Macbook M1/M2 with 64GB of RAM.

2OEH8eoCRo0 3y ago

How? It's larger than 64GB.

coder543 3y ago
Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
- 2OEH8eoCRo0 3y ago
  I'm aware but is it still LLaMA 70B at that point?
  - coder543 3y ago
    Yes. Quantization does not reduce the number of parameters. It does not re-train the model.
  - deleted 3y ago
    [deleted]
  - andy99 3y ago
    It's a legit question, the model will be worse in some way... I've seen it discussed that all things being equal more parameters is better (meaning it's better to take a big model and quantized it to fit in memory than use a smaller unquantized model that fits), but a quantized model wouldn't be expected to run identically to or as well as the full model.
    - coder543 3y ago
      You don’t stop being andy99 just because you’re a little tired, do you? Being tired makes everyone a little less capable at most things. Sometimes, a lot less capable. In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program. If we’re not willing to declare that you are a completely separate person when you’re tired, or that 32-bit and 64-bit versions are completely different programs, then I don’t think it’s worth getting overly philosophical about quantization. A quantized model is still the same model. The quality loss from using 4+ bit quantization is minimal, in my experience. Yes, it has a small impact on accuracy, but with massive efficiency gains. I don’t really think anyone should be running the full models outside of research in the first place. If anything, the quantized models should be considered the “real” models, and the full fp16/fp32 model should just be considered a research artifact distinct from the model. But this philosophical rabbit hole doesn’t seem to lead anywhere interesting to me. Various papers have shown that 4-bit quantization is a great balance. One example: https://arxiv.org/pdf/2212.09720.pdf https://arxiv.org/pdf/2212.09720.pdf
      - cjbprime 3y ago
        I don't like the metaphor: when I'm tired, I will be alert again later. Quantization is lossy compression: the human equivalent would be more like a traumatic brain injury affecting recall, especially of fine details. The question of whether I am still me after a traumatic brain injury is philosophically unclear, and likely depends on specifics about the extent of the deficits.
        coder543 3y ago
        The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the analogy out further, if the model was able to get an A on a test before quantization, it would likely still get a B at worst afterwards, given a drop in the score of less than 10%. Depending on the task, the measured impact could even be negligible. It’s far more similar to the model being perpetually tired than it is to a TBI. You may nitpick the analogy, but analogies are never exact. You also ignore the other piece that I pointed out, which is how we treat other software that comes in multiple slightly different forms.
  - manmal 3y ago
    Sure, quantization reduces information stored for each parameter, not the parameter count.
- sbrother 3y ago
  Can I ask how many tok/s you're getting on that setup? I'm trying to decide whether to invest in a high-end NVIDIA setup or a Mac Studio with llama.cpp for the purposes of running LLMs like this one locally.
  - coder543 3y ago
    On a 33B model at q4_0 quantization, I’m seeing about 36 tokens/s on the RTX 3090 with all layers offloaded to the GPU. Mixtral runs at about 43 tokens/s at q3_K_S with all layers offloaded. I normally avoid going below 4-bit quantization, but Mixtral doesn’t seem phased. I’m not sure if the MoE just makes it more resilient to quantization, or what the deal is. If I run it at q4_0, then it runs at about 24 tokens/s, with 26 out of 33 layers offloaded, which is still perfectly usable, but I don’t usually see the need with Mixtral. Ollama dynamically adjusts the layers offloaded based on the model and context size, so if I need to run with a larger context window, that reduces the number of layers that will fit on the GPU and that impacts performance, but things generally work well.
    - sbrother 3y ago
      Thanks! That is really fast for personal use.
    - sorenjan 3y ago
      What's the power consumption and fan noise like when doing that? I assume you're running the model doing inference in the background for the whole coding session, i.e. hours at a time?
      - coder543 3y ago
        I don’t use local LLMs for CoPilot-like functionality, but I have toyed with the concept. There are a few things to keep in mind: no programmer that I know is sitting there typing code for hours at a time without stopping. There’s a lot more to being a developer than just typing, whether it is debugging, thinking, JIRA, Slack, or whatever else. These CoPilot-like tools will only activate after you type something, then stop for a defined timeout period. While you’re typing, they do nothing. After they generate, they do nothing. I would honestly be surprised if the GPU active time was more than 10% averaged over an hour. When actively working on a large LLM, the RTX 3090 is drawing close to 400W in my desktop. At a 10% duty cycle (active time), that would be 40W on average, which would be 320Wh over the course of a full 8-hour day of crazy productivity. My electric rate is about 15¢/kWh, so that would be about 5¢ per day. It is absolutely not running at a 100% duty cycle, and it’s absurd to even do the math for that, but we can multiply by 10 and say that if you’re somehow a mythical “10x developer” then it would be 50¢/day in electricity here. I think 5¢/day to 10¢/day is closer to reality. Either way, the cost is marginal at the scale of a software developer’s salary.
rgbrgb 3y ago
Quantization can take it under 30GB (with quality degradation). For example, take a look at the GGUF file sizes here: https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF

reddit_clone 3y ago

I am not too familiar with LLMs and GPUs (Not a gamer either). But want to learn. Could you please expand on what else would be capable of running such models locally? How about a linux laptop/desktop with specific hardware configuration?

MeImCounting 3y ago
It pretty much comes down to 2 factors which is memory bandwidth and compute. You need a high enough memory bandwidth to be able to "feed" the compute and you need beefy enough compute to be able to keep up with the data that is being fed in by the memory. In theory a single Nvidia 4090 would be able to run a 70b model with quantization at "useable" speeds. The reason mac hardware is so capable in AI is because of the unified architecture meaning the memory is shared across the GPU and CPU. There are other factors but it essentially comes down to tokens per second advantages. You could run one of these models on an old GPU with low memory bandwidth just fine but your tokens per second would be far too slow for what most people consider "useable" and the quantization necessary might star noticeably effecting the quality.
- int_19h 3y ago
  A single RTX 4090 can run at most 34b models with 4-bit quantization. You'd need 2-bit for 70b, and at that point quality plummets. Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.
  - MeImCounting 3y ago
    Thats a good point. For example both the RTX 4090 and the RTX 6000 Ada Generation use the AD102 chip. The RTX 6000 Ada though, would be able to run 70b models due to the larger memory pool despite having the same memory interface width.
    - 3y ago