6 ms·

LLM Falcon 180B Needs 720GB RAM to Run

102 points by behnamoh 3y ago

tux3 3y ago
Surely the next step will be sparsity and other optimizations beyond brute aggressive quantization, maybe an MoE model like OAI is doing
- brucethemoose2 3y ago
  > brute aggressive quantization Quantization like ExLlama's EX2 or even llama.cpp's k-quants is already far from brute force: https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization > The format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight. Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on performance. Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target average bitrate. Llama.cpp is also working on a feature that let's a small model "guess" the output of a big model which then "checks" it for correctness. This is more of a performance feature, but you could also arrange it to accelerate a big model on a small GPU.
nullc 3y ago
Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), without any tuning effort.
- nullc 3y ago
  [A chat between a curious user and an artificial intelligence assistant named Falcon. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please give a short explination of 'clickbait'.\nFalcon:] Certainly! Clickbait is a term used to describe online content that is designed to attract attention and encourage users to click on a link to a particular webpage. It often uses sensational or provocative headlines, images, or videos to entice users to click through to the content. The goal of clickbait is typically to generate more traffic to a website and increase advertising revenue. However, it can sometimes be misleading or deceptive, as the actual content may not live up to the expectations set by the clickbait headline or image.
- nullc 3y ago
  Seriously, my self reply showing output from Falcon 180B is flagged??
  - KirillPanov 3y ago
    I think 100% LLM-generated comments are fairly unpopular. Suggestion: if you feel the need to share some LLM output with HN, prefix it with a human-written sentence or two explaining why it's worth reading.
- jamesblonde 3y ago
  Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory Assuming zero copy, it should be loadable on around 170GB. Reference: https://www.hopsworks.ai/dictionary/llms-large-language-models https://www.hopsworks.ai/dictionary/llms-large-language-mode...
  - stevenhuang 3y ago
    No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
  - nullc 3y ago
    Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4). Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.
natsucks 3y ago
It makes no sense to me why people want to run this model when it barely outperforms Llama 2 while being over twice the size.
- orra 3y ago
  I was going to say: because Falcon 180GB is open source, so you can run it for any purpose, without worrying about license restrictions. However, it transpires Falcon 180B uses a proprietary license “based upon” the Apache license. Despite Falcon's claims to the contrary, it is not open source. And that's disappointing because Falcon 40B is actually open source (and Apache licensed).
  - deleted 3y ago
    [deleted]
  - dartos 3y ago
    Yep. No truly open source big models around rn, but people keep calling them open source. Some amazing marketing by meta.
    - netdur 3y ago
      I am not sure how can a binary have open source license at first place
      - dartos 3y ago
        Through the magic of marketing
      - jazzyjackson 3y ago
        not so much about reading/modifying the source, more about what you can build on top, but point taken, could use a different word, "public domain" would be nice if anyone actually wanted to give something back to the public after digesting all the information ever produced
      - 3y ago
TrendyCPU 3y ago
That is 180 billion parameters not 180 bytes, for anyone else confused.
- toxik 3y ago
  Yep and 16 bits per parameter means 360 GiB, then because you typically first allocate an empty model then separately load the weights, 720 GiB.
leiframming 3y ago
Interesting article, but the premise is wrong. You don't need 720GB ram to load the model. The author claims that you need to first load up the model (360GB) and then the load the weights (another 360 GB) and the replace the first with the latter. HF allows initializing empty models that do not take up any space using accelerate, especially so that you don't run into that problem. Still, 180B is a best to load up.
- netdur 3y ago
  later he said HF's tensor has almost zero copy, so 720GB is not needed using this format
  - leiframming 3y ago
    No, I think the author gets this mixed up. Zero copy does not play a role here - the problem is how the model is being loaded in the steps that are shown in the article and they have nothing to do with Safe tensors' zero copy ability. Check out the article by Hugging face to see how to do it with only 360GB instead of 720: https://huggingface.co/docs/accelerate/usage_guides/big_modeling https://huggingface.co/docs/accelerate/usage_guides/big_mode...
- kristianp 3y ago
  > Still, 180B is a best to load up. Best -> Beast
ai_ja_nai 3y ago
Current LLMs are in the Pentium IV Netburst stage: hypertrophy of components on old archs have gotten to the limits of what is physically feasible. Get ready for the next phase of leaner models.
hnfong 3y ago
Title here seems clickbait. The title from the article I see it now is "Falcon 180B: Can It Run on Your Computer?" The content does mention 720GB RAM, but quickly clarifies that it can be reduced: QUOTE: Step 1 and step 2 are the ones that consume memory. In total, you would need 720 GB of memory available. This can be CPU RAM, but for fast inference, you may want to use GPUs, e.g., 9 A100 with 80 GB of VRAM. This wouldn’t be enough for batch inference. With either CPU RAM or VRAM, that’s a lot of memory. Fortunately, these requirements can be easily reduced.
haolez 3y ago
On a side note, has anyone here used this model for anything's interesting? Can it outperform popular smaller models?
- Zuiii 3y ago
  Given it's ridiculous license, how can we? I'd rather accept meta's proprietary "open source" license that deal with that nonsense.
  - kyboren 3y ago
    I don't really understand all the fuss about license agreements in this space. Just ignore their stupid license and obtain the model weights via alternative means. Both LLaMA and LLaMA v2 are readily available on BitTorrent[0]. It's not clear to me whether compilations of model weights are really copyrightable in most jurisdictions. Even if they were, enforceability is highly questionable. Anyway downloading is not the same as uploading; the recipient is not the one making the copy. So long as you are not redistributing the model weights but just using those model weights to do something, I don't see how you're violating copyright. If you pirate a compiler and distribute your own software compiled with it--is that compiled software a derivative work of the compiler? Is my proprietary software a derivative work of g++ because I used it to compile my software? Even providing a direct API to inference with or finetuning a model should be OK. I can't see how the output of inference could be a derivative work of the model, which is only part of the input along with the hyperparameters, seed value, potential fine-tuning data, and the prompt. If, alternatively, it was, then the output of training (i.e. the model itself) is surely a derivative work of the input--including my data they slurped up! On the other hand, redistributing finetuned models might need to be done by weight diffs, as those probably are derived works. [0]: e.g. b8287ebfa04f879b048d4d4404108cf3e8014352 and 14ec42c98d0b12f5b12f5b3d670cdf4915c1feaf
    - orra 3y ago
      Your proprietary software won't be a derivative work of most of G++, but it will likely be a derivative work of libgcc (compiler runtime), libc (on Linux), and libstdc++. Anyway, running copyrighted software without a valid license is clear cut unlawful, at least in the EU. That's because running the application requires copying it (into RAM). No, really, the law is that pedantic. "Loading" is a restricted act under Article 4 of the Computer Programs Directive.
alecco 3y ago
> You will need at least 400GB of memory to swiftly run inference with Falcon-180B. [0] https://huggingface.co/tiiuae/falcon-180B https://huggingface.co/tiiuae/falcon-180B
jmorgan 3y ago
A 4-bit quantized version will require (still a whopping) ~100GB of RAM using tools like Ollama and Llama.cpp If you want to try it with Ollama on macOS (keeping in mind you'll need the new Mac Studio with 192GB of memory) this command will work: ollama run falcon:180b Right now it gets about ~5 t/s.. there's quite a bit of work being done to improve inference speeds like speculative sampling which uses a smaller model for a subset of tokens

102 points by behnamoh 3y ago

tux3 3y ago

Surely the next step will be sparsity and other optimizations beyond brute aggressive quantization, maybe an MoE model like OAI is doing

brucethemoose2 3y ago
> brute aggressive quantization Quantization like ExLlama's EX2 or even llama.cpp's k-quants is already far from brute force: https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization > The format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight. Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on performance. Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target average bitrate. Llama.cpp is also working on a feature that let's a small model "guess" the output of a big model which then "checks" it for correctness. This is more of a performance feature, but you could also arrange it to accelerate a big model on a small GPU.

nullc 3y ago

Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), without any tuning effort.

nullc 3y ago
[A chat between a curious user and an artificial intelligence assistant named Falcon. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please give a short explination of 'clickbait'.\nFalcon:] Certainly! Clickbait is a term used to describe online content that is designed to attract attention and encourage users to click on a link to a particular webpage. It often uses sensational or provocative headlines, images, or videos to entice users to click through to the content. The goal of clickbait is typically to generate more traffic to a website and increase advertising revenue. However, it can sometimes be misleading or deceptive, as the actual content may not live up to the expectations set by the clickbait headline or image.
nullc 3y ago
Seriously, my self reply showing output from Falcon 180B is flagged??
- KirillPanov 3y ago
  I think 100% LLM-generated comments are fairly unpopular. Suggestion: if you feel the need to share some LLM output with HN, prefix it with a human-written sentence or two explaining why it's worth reading.
jamesblonde 3y ago
Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory Assuming zero copy, it should be loadable on around 170GB. Reference: https://www.hopsworks.ai/dictionary/llms-large-language-models https://www.hopsworks.ai/dictionary/llms-large-language-mode...
- stevenhuang 3y ago
  No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
- nullc 3y ago
  Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4). Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.

natsucks 3y ago

It makes no sense to me why people want to run this model when it barely outperforms Llama 2 while being over twice the size.

orra 3y ago
I was going to say: because Falcon 180GB is open source, so you can run it for any purpose, without worrying about license restrictions. However, it transpires Falcon 180B uses a proprietary license “based upon” the Apache license. Despite Falcon's claims to the contrary, it is not open source. And that's disappointing because Falcon 40B is actually open source (and Apache licensed).
- deleted 3y ago
  [deleted]
- dartos 3y ago
  Yep. No truly open source big models around rn, but people keep calling them open source. Some amazing marketing by meta.
  - netdur 3y ago
    I am not sure how can a binary have open source license at first place
    - dartos 3y ago
      Through the magic of marketing
    - jazzyjackson 3y ago
      not so much about reading/modifying the source, more about what you can build on top, but point taken, could use a different word, "public domain" would be nice if anyone actually wanted to give something back to the public after digesting all the information ever produced
    - 3y ago

TrendyCPU 3y ago

That is 180 billion parameters not 180 bytes, for anyone else confused.

toxik 3y ago
Yep and 16 bits per parameter means 360 GiB, then because you typically first allocate an empty model then separately load the weights, 720 GiB.

leiframming 3y ago

Interesting article, but the premise is wrong. You don't need 720GB ram to load the model. The author claims that you need to first load up the model (360GB) and then the load the weights (another 360 GB) and the replace the first with the latter. HF allows initializing empty models that do not take up any space using accelerate, especially so that you don't run into that problem. Still, 180B is a best to load up.

netdur 3y ago
later he said HF's tensor has almost zero copy, so 720GB is not needed using this format
- leiframming 3y ago
  No, I think the author gets this mixed up. Zero copy does not play a role here - the problem is how the model is being loaded in the steps that are shown in the article and they have nothing to do with Safe tensors' zero copy ability. Check out the article by Hugging face to see how to do it with only 360GB instead of 720: https://huggingface.co/docs/accelerate/usage_guides/big_modeling https://huggingface.co/docs/accelerate/usage_guides/big_mode...
kristianp 3y ago
> Still, 180B is a best to load up. Best -> Beast

ai_ja_nai 3y ago

Current LLMs are in the Pentium IV Netburst stage: hypertrophy of components on old archs have gotten to the limits of what is physically feasible. Get ready for the next phase of leaner models.

hnfong 3y ago

Title here seems clickbait. The title from the article I see it now is "Falcon 180B: Can It Run on Your Computer?" The content does mention 720GB RAM, but quickly clarifies that it can be reduced: QUOTE: Step 1 and step 2 are the ones that consume memory. In total, you would need 720 GB of memory available. This can be CPU RAM, but for fast inference, you may want to use GPUs, e.g., 9 A100 with 80 GB of VRAM. This wouldn’t be enough for batch inference. With either CPU RAM or VRAM, that’s a lot of memory. Fortunately, these requirements can be easily reduced.

haolez 3y ago

On a side note, has anyone here used this model for anything's interesting? Can it outperform popular smaller models?

Zuiii 3y ago
Given it's ridiculous license, how can we? I'd rather accept meta's proprietary "open source" license that deal with that nonsense.
- kyboren 3y ago
  I don't really understand all the fuss about license agreements in this space. Just ignore their stupid license and obtain the model weights via alternative means. Both LLaMA and LLaMA v2 are readily available on BitTorrent[0]. It's not clear to me whether compilations of model weights are really copyrightable in most jurisdictions. Even if they were, enforceability is highly questionable. Anyway downloading is not the same as uploading; the recipient is not the one making the copy. So long as you are not redistributing the model weights but just using those model weights to do something, I don't see how you're violating copyright. If you pirate a compiler and distribute your own software compiled with it--is that compiled software a derivative work of the compiler? Is my proprietary software a derivative work of g++ because I used it to compile my software? Even providing a direct API to inference with or finetuning a model should be OK. I can't see how the output of inference could be a derivative work of the model, which is only part of the input along with the hyperparameters, seed value, potential fine-tuning data, and the prompt. If, alternatively, it was, then the output of training (i.e. the model itself) is surely a derivative work of the input--including my data they slurped up! On the other hand, redistributing finetuned models might need to be done by weight diffs, as those probably are derived works. [0]: e.g. b8287ebfa04f879b048d4d4404108cf3e8014352 and 14ec42c98d0b12f5b12f5b3d670cdf4915c1feaf
  - orra 3y ago
    Your proprietary software won't be a derivative work of most of G++, but it will likely be a derivative work of libgcc (compiler runtime), libc (on Linux), and libstdc++. Anyway, running copyrighted software without a valid license is clear cut unlawful, at least in the EU. That's because running the application requires copying it (into RAM). No, really, the law is that pedantic. "Loading" is a restricted act under Article 4 of the Computer Programs Directive.

alecco 3y ago

> You will need at least 400GB of memory to swiftly run inference with Falcon-180B. [0] https://huggingface.co/tiiuae/falcon-180B https://huggingface.co/tiiuae/falcon-180B

jmorgan 3y ago

A 4-bit quantized version will require (still a whopping) ~100GB of RAM using tools like Ollama and Llama.cpp If you want to try it with Ollama on macOS (keeping in mind you'll need the new Mac Studio with 192GB of memory) this command will work: ollama run falcon:180b Right now it gets about ~5 t/s.. there's quite a bit of work being done to improve inference speeds like speculative sampling which uses a smaller model for a subset of tokens