3 ms·

FlexGen: Running large language models on a single GPU

192 points by behnamoh 4y ago

deleted 4y ago
[deleted]
aliljet 4y ago
This is absolutely something I was running into with LLaMA. I'm curious if this potentially extends into that particular use case...
stuckinhell 4y ago
This is absolutely stunning work. Excited to try it out on my husband's homelab.
zargon 4y ago
Previous discussion is here (266 comments): https://news.ycombinator.com/item?id=34869960 https://news.ycombinator.com/item?id=34869960
deleted 4y ago
[deleted]
neilv 4y ago
> This project was made possible thanks to a collaboration with ... Yandex Research ... I'm all for global cooperation and fellowship. Are sanctions going to be a barrier for this and related projects?
- lxe 4y ago
  Bing says: I could not find any specific information on how these sanctions affect academic research and cooperation between the US and Russia. Some sources suggest that some academics have canceled conferences, joint projects, and funding with Russian institutions as a form of self-imposed sanctions2, while others indicate that Russian students are still able to secure visas to study abroad3. Therefore, the impact of the sanctions on academic research and cooperation may vary depending on the field, institution, and individual circumstances.
- blagie 4y ago
  It depends on the work and the sanctions regime, but in general: - The best sanctions impact targeted industries (e.g. anything needed to build tanks, warplanes, etc.). - The worst sanctions impact communications and collaboration. Change comes from conversations. Media, non-military education, and non-military academic collaboration are bad usually bad sanctions.
- 01100011 4y ago
  Well, the sanctions are an attempt at denying China access to more powerful GPUs like the A100, so it seems the sanctions may actually accelerate research into running LLMs on lower performing HW.
  - kkielhofner 4y ago
    What's so weird about that (not surprising because lawmakers/politics) is that A100s are actually terrible at inference. Assuming the model can fit in RAM my 3090 handily beats them across the board at everything I've tried. My 4090 blows them away. So by the most commonly accepted definition of "performance" the GPUs banned in China are actually /lower/ performance. The only thing this ban does is slow down from scratch training of very large models. There's a lot of effort on finetuning these large models on < 80GB VRAM hardware but significantly more effort on "running" them on < 24GB VRAM hardware (3090/4090 and lower). In the end your point stands - between any efforts by those impacted by sanctions or just the hobbyist/low-mid range commercial applications research into cramming these huge models into "consumer" hardware are accelerating dramatically.
    - riku_iki 4y ago
      3090 is not allowed to be used in datacenters by nvidia, which China likely doesn't care about, so they are in advantage at the end of the day.
      - kkielhofner 4y ago
        Interestingly that (in)famous language is only included in the driver EULA for the Nvidia Windows driver, not the Linux driver (last I looked).
- SXX 4y ago
  Yandex still operate a lot of services in EU under Yango branding. They trying to hide their relationships with Kremlin-controlled russian Yandex, but do it quite badly. So I not sure why you decide sanctions will affect their research wing. Also it's likely good part of their research staff already work from Israel and Serbia even though they not finished splitting the company into Russian one controlled by Kremlin and international one. PS: As I see only contributor in repository from Yandex is located in Hungary and not in Russia.
stavros 4y ago
What's the currently best-performing LLM that one can run with this?
kkielhofner 4y ago
What's really amazing about a lot of these recent projects is they tend to provide benchmarks running on an Nvidia T4. They use these because they're relatively cheap from cloud providers and you can usually actually get them (as opposed to requesting and getting denied for an A100 or whatever). For those that aren't familiar with it they are tiny power and density optimized GPUs. I have the successor (A2) and total max TDP is 60 watts. Single slot. Slot only power, and passively cooled. Depending on workload I observe it to be roughly 5-10x slower than a 3090, which means for most people at home with a spare Nvidia gaming card (or whatever) you'll see results from these project at a performance multiple of the benchmarks they provide. The one caveat is that the T4/A2 have 16GB VRAM, which makes them more capable (albeit slower) than a "low end" desktop card like the 3070 which has only 8GB VRAM. But as HN readers know there is incredible progress daily to reduce VRAM requirements for these models!
- deleted 4y ago
  [deleted]
- LoganDark 4y ago
  > a "low end" desktop card like the 3070 which has only 8GB VRAM I'm finding that my 12GB 3060 is not enough to run even LLaMA-7B without CPU offload, thank Arceus for quantization...
  - kkielhofner 4y ago
    Give it time - there's a lot of incredible work being done here and we'll likely eventually see some kind of black magic to make it possible.
ByThyGrace 4y ago
I'm just here waiting for the "LLM retard guide" to come out, as it happened for stable diffusion last August.
- arthurcolle 4y ago
  Link to stable diffusion ref you're referring to? I was able to run the model and everything, so pretty familiar, but just wondering if you're referring to a specific document! Haha
  - ByThyGrace 4y ago
    Apparently this is the latest iteration: https://rentry.org/voldy https://rentry.org/voldy
    - arthurcolle 4y ago
      Link is not working
      - ByThyGrace 4y ago
        It is working for me right now.
- throwaway4aday 4y ago
  Copy paste the parts of the install steps for LLaMA C++ into ChatGPT and ask it to explain things simply and include any prerequisite steps you might need to do. If you get stuck, just ask ChatGPT and include any error messages. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
  - LoganDark 4y ago
    LLaMA C++ is easy to set up, but it's sort of slow even pegging all cores of an overclocked i5-12400F, so I'd like to get a 3060 working (with 12GB of VRAM)... it's just it's so difficult to get some sort of ChatGPT-style interface running. Haven't gotten anything working yet even though I have a working CUDA PyTorch. Bleh bleh bleh~
lxe 4y ago
Best way to mess around with FlexGen and LLMs on local hardware in general is https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- boppo1 4y ago
  Can't get it to run on amd 6700xt even though there's ROCm installation instructions. Tried to run llama 7b but got hung up because bitsandbytes calls CUDA.
  - kkielhofner 4y ago
    Par for the course, I'm afraid. Over the years I've made several attempts at AMD/ROCm. I've wasted so much time AMD owes me free GPUs for life ;). At this point I've just accepted that AMD is irrelevant in ML and they're fine with that - they're not making much of an effort to change that. I really and truly wish that wasn't the case but I've accepted that it is.
    - freeqaz 4y ago
      Is there an equivalent to Nvidia-Docker for ROCm to make it brain dead easy? Where does the complexity come in for AMD GPUs?
      - kkielhofner 4y ago
        It's far more complicated than that. Take a look at the Nvidia Frameworks Support Matrix[0]. The Nvidia PyTorch Docker container is 20GB uncompressed(!!!!) and consists of dozens of layers of Nvidia/CUDA tailored software stacks[1] that all come together to do this magical ML/AI stuff we take for granted on CUDA hardware. BTW, we have a nice, clean Nvidia CUDA docker situation because Nvidia has been working on it for years. It's rock solid and universally supported. Moving to a lower level, check out the release notes for the latest Nvidia driver and look in amazement at the sheer number of supported GPUs[2]. In short, literally every GPU they've put in a laptop, desktop, workstation, or datacenter over the past decade (plus their embedded Jetson stuff). There are over 2,000 GPUs listed there and I can tell you from experience the "Unified" in CUDA holds up. If the card is supported by the driver, it has a compatible compute arch, and enough VRAM whatever you throw at it will just work. In many cases even on Windows! People complain about the proprietary driver but the fact is if your GPU says Nvidia on it you can install the one driver on any distro and have these projects up and running in minutes. Compare that to the ROCm "list"[3] of what, maybe a dozen GPUs? I know from experience (unfortunately) that even with "supported" hardware in many cases just getting the driver to work is a nightmare. Then you have basic frameworks randomly crashing, etc. It's a complete mess and like I said - AMD is a decade behind today and Nvidia's lead is only growing. [0] - https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html https://docs.nvidia.com/deeplearning/frameworks/support-matr... [1] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc... [2] - https://download.nvidia.com/XFree86/Linux-x86_64/530.41.03/README/supportedchips.html https://download.nvidia.com/XFree86/Linux-x86_64/530.41.03/R... [3] - https://docs.amd.com/bundle/Hardware_and_Software_Reference_Guide/page/Hardware_and_Software_Support.html https://docs.amd.com/bundle/Hardware_and_Software_Reference_...
htrp 4y ago
Readme updated 19 hours ago. Anything new with the codebase?

192 points by behnamoh 4y ago

deleted 4y ago

[deleted]

This is absolutely something I was running into with LLaMA. I'm curious if this potentially extends into that particular use case...

stuckinhell 4y ago

This is absolutely stunning work. Excited to try it out on my husband's homelab.

zargon 4y ago

Previous discussion is here (266 comments): https://news.ycombinator.com/item?id=34869960 https://news.ycombinator.com/item?id=34869960

deleted 4y ago

[deleted]

neilv 4y ago

> This project was made possible thanks to a collaboration with ... Yandex Research ... I'm all for global cooperation and fellowship. Are sanctions going to be a barrier for this and related projects?

lxe 4y ago
Bing says: I could not find any specific information on how these sanctions affect academic research and cooperation between the US and Russia. Some sources suggest that some academics have canceled conferences, joint projects, and funding with Russian institutions as a form of self-imposed sanctions2, while others indicate that Russian students are still able to secure visas to study abroad3. Therefore, the impact of the sanctions on academic research and cooperation may vary depending on the field, institution, and individual circumstances.
blagie 4y ago
It depends on the work and the sanctions regime, but in general: - The best sanctions impact targeted industries (e.g. anything needed to build tanks, warplanes, etc.). - The worst sanctions impact communications and collaboration. Change comes from conversations. Media, non-military education, and non-military academic collaboration are bad usually bad sanctions.
01100011 4y ago
Well, the sanctions are an attempt at denying China access to more powerful GPUs like the A100, so it seems the sanctions may actually accelerate research into running LLMs on lower performing HW.
- kkielhofner 4y ago
  What's so weird about that (not surprising because lawmakers/politics) is that A100s are actually terrible at inference. Assuming the model can fit in RAM my 3090 handily beats them across the board at everything I've tried. My 4090 blows them away. So by the most commonly accepted definition of "performance" the GPUs banned in China are actually /lower/ performance. The only thing this ban does is slow down from scratch training of very large models. There's a lot of effort on finetuning these large models on < 80GB VRAM hardware but significantly more effort on "running" them on < 24GB VRAM hardware (3090/4090 and lower). In the end your point stands - between any efforts by those impacted by sanctions or just the hobbyist/low-mid range commercial applications research into cramming these huge models into "consumer" hardware are accelerating dramatically.
  - riku_iki 4y ago
    3090 is not allowed to be used in datacenters by nvidia, which China likely doesn't care about, so they are in advantage at the end of the day.
    - kkielhofner 4y ago
      Interestingly that (in)famous language is only included in the driver EULA for the Nvidia Windows driver, not the Linux driver (last I looked).
SXX 4y ago
Yandex still operate a lot of services in EU under Yango branding. They trying to hide their relationships with Kremlin-controlled russian Yandex, but do it quite badly. So I not sure why you decide sanctions will affect their research wing. Also it's likely good part of their research staff already work from Israel and Serbia even though they not finished splitting the company into Russian one controlled by Kremlin and international one. PS: As I see only contributor in repository from Yandex is located in Hungary and not in Russia.

stavros 4y ago

What's the currently best-performing LLM that one can run with this?

kkielhofner 4y ago

What's really amazing about a lot of these recent projects is they tend to provide benchmarks running on an Nvidia T4. They use these because they're relatively cheap from cloud providers and you can usually actually get them (as opposed to requesting and getting denied for an A100 or whatever). For those that aren't familiar with it they are tiny power and density optimized GPUs. I have the successor (A2) and total max TDP is 60 watts. Single slot. Slot only power, and passively cooled. Depending on workload I observe it to be roughly 5-10x slower than a 3090, which means for most people at home with a spare Nvidia gaming card (or whatever) you'll see results from these project at a performance multiple of the benchmarks they provide. The one caveat is that the T4/A2 have 16GB VRAM, which makes them more capable (albeit slower) than a "low end" desktop card like the 3070 which has only 8GB VRAM. But as HN readers know there is incredible progress daily to reduce VRAM requirements for these models!

deleted 4y ago
[deleted]
LoganDark 4y ago
> a "low end" desktop card like the 3070 which has only 8GB VRAM I'm finding that my 12GB 3060 is not enough to run even LLaMA-7B without CPU offload, thank Arceus for quantization...
- kkielhofner 4y ago
  Give it time - there's a lot of incredible work being done here and we'll likely eventually see some kind of black magic to make it possible.

ByThyGrace 4y ago

I'm just here waiting for the "LLM retard guide" to come out, as it happened for stable diffusion last August.

arthurcolle 4y ago
Link to stable diffusion ref you're referring to? I was able to run the model and everything, so pretty familiar, but just wondering if you're referring to a specific document! Haha
- ByThyGrace 4y ago
  Apparently this is the latest iteration: https://rentry.org/voldy https://rentry.org/voldy
  - arthurcolle 4y ago
    Link is not working
    - ByThyGrace 4y ago
      It is working for me right now.
throwaway4aday 4y ago
Copy paste the parts of the install steps for LLaMA C++ into ChatGPT and ask it to explain things simply and include any prerequisite steps you might need to do. If you get stuck, just ask ChatGPT and include any error messages. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- LoganDark 4y ago
  LLaMA C++ is easy to set up, but it's sort of slow even pegging all cores of an overclocked i5-12400F, so I'd like to get a 3060 working (with 12GB of VRAM)... it's just it's so difficult to get some sort of ChatGPT-style interface running. Haven't gotten anything working yet even though I have a working CUDA PyTorch. Bleh bleh bleh~

lxe 4y ago

Best way to mess around with FlexGen and LLMs on local hardware in general is https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui

boppo1 4y ago
Can't get it to run on amd 6700xt even though there's ROCm installation instructions. Tried to run llama 7b but got hung up because bitsandbytes calls CUDA.
- kkielhofner 4y ago
  Par for the course, I'm afraid. Over the years I've made several attempts at AMD/ROCm. I've wasted so much time AMD owes me free GPUs for life ;). At this point I've just accepted that AMD is irrelevant in ML and they're fine with that - they're not making much of an effort to change that. I really and truly wish that wasn't the case but I've accepted that it is.
  - freeqaz 4y ago
    Is there an equivalent to Nvidia-Docker for ROCm to make it brain dead easy? Where does the complexity come in for AMD GPUs?
    - kkielhofner 4y ago
      It's far more complicated than that. Take a look at the Nvidia Frameworks Support Matrix[0]. The Nvidia PyTorch Docker container is 20GB uncompressed(!!!!) and consists of dozens of layers of Nvidia/CUDA tailored software stacks[1] that all come together to do this magical ML/AI stuff we take for granted on CUDA hardware. BTW, we have a nice, clean Nvidia CUDA docker situation because Nvidia has been working on it for years. It's rock solid and universally supported. Moving to a lower level, check out the release notes for the latest Nvidia driver and look in amazement at the sheer number of supported GPUs[2]. In short, literally every GPU they've put in a laptop, desktop, workstation, or datacenter over the past decade (plus their embedded Jetson stuff). There are over 2,000 GPUs listed there and I can tell you from experience the "Unified" in CUDA holds up. If the card is supported by the driver, it has a compatible compute arch, and enough VRAM whatever you throw at it will just work. In many cases even on Windows! People complain about the proprietary driver but the fact is if your GPU says Nvidia on it you can install the one driver on any distro and have these projects up and running in minutes. Compare that to the ROCm "list"[3] of what, maybe a dozen GPUs? I know from experience (unfortunately) that even with "supported" hardware in many cases just getting the driver to work is a nightmare. Then you have basic frameworks randomly crashing, etc. It's a complete mess and like I said - AMD is a decade behind today and Nvidia's lead is only growing. [0] - https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html https://docs.nvidia.com/deeplearning/frameworks/support-matr... [1] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc... [2] - https://download.nvidia.com/XFree86/Linux-x86_64/530.41.03/README/supportedchips.html https://download.nvidia.com/XFree86/Linux-x86_64/530.41.03/R... [3] - https://docs.amd.com/bundle/Hardware_and_Software_Reference_Guide/page/Hardware_and_Software_Support.html https://docs.amd.com/bundle/Hardware_and_Software_Reference_...

htrp 4y ago

Readme updated 19 hours ago. Anything new with the codebase?