4 ms·

There is a bait and switch going on, and sam altman or mark zuckerberg are the first to tell you. “No one can compete with us, but it’s cute to try! Make appli

by b33j0r 3y ago

There is a bait and switch going on, and sam altman or mark zuckerberg are the first to tell you. “No one can compete with us, but it’s cute to try! Make applications though” —almost direct quote from Sam Altman. I have 64gb and an RTX 3090 and a macbook M3, and I already can’t run a lot of the newest models even in their quantized form. The business model requires this to be a subscription service. At least as of today…

kuczmama 3y ago
Realistically, what hardware would be required to run this? I assumed a RTX 3090 would be enough?
- summarity 3y ago
  A Studio Mac with an M1 Ultra (about 2800 USD used) is actually a really cost effective way to run in. Its total system power consumption is really low, even spitting out tokens at full tilt (<250W).
- kkzz99 3y ago
  RTX 3090 has 24GB of memory, a quantized llama70b takes around 60GB of memory. You can offload a few layers on the gpu, but most of them will run on the CPU with terrible speeds.
  - nullc 3y ago
    You're not required to put the whole model in a single GPU. You can buy a 24GB gpu for $150-ish (P40).
    - kuczmama 3y ago
      Wow that's a really good idea. I could potentially buy 4 Nvidia P40's for the same price as a 3090 and run inference on pretty much any model I want.
      - eurekin 3y ago
        Just make sure you're comfortable with manually compiling the bitsandbytes and generally combine a software stack of almost out of date libraries
        kuczmama 3y ago
        That's a good point. Are you referring to the out of date cuda libraries?
        eurekin 3y ago
        I don't remember exactly (either cuda directly or the cudnn version used by the flashattention)... Anyway, /r/localLlama has few instances of such builds. Might be really worthwhile looking that up before buying
        nullc 3y ago
        P40 still works with 12.2 at the moment. I used to use K80s (which I think I paid like $50 for!) which turned into a huge mess to deal with older libraries, especially since essentially all ML stuff is on a crazy upgrade cadence with everything constantly breaking even without having to deal with orphaned old software. You can get gpu server chassis that have 10 pci-slots too! for around $2k on ebay. But note that there is a hardware limitation on the PCI-E cards such that each card can only directly communicate with 8 others at a time. Beware, they're LOUD even by the standards of sever hardware. Oh also the nvidia tesla power connectors have cpu-connector like polarity instead of pci-e, so at least in my chassis I needed to adapt them. Also keep in mind that if you aren't using a special gpu chassis, the tesla cards don't have fans, so you have to provide cooling.
      - frognumber 3y ago
        For reference for readers. SUPPORTED ========= * Ada / Hopper / A4xxx (but not A4000) * Ampere / A3xxx * Turing / Quadro RTX / GTX 16xx / RTX 20XX / Volta / Tesla EOL 2023/2024 ============= * Pascal / Quadro P / Geforce GTX 10XX / Tesla Unsupported =========== * Maxwell * Kepler * Fermi * Tesla (yes, this one pops up over and over, chaotically) * Curie Older don't really do GPGPU much. The older cards are also quite slow relative to modern ones! A lot of the ancient workstation cards can run big models cheaply, but (1) with incredible software complexity (2) very slowly, even relative to modern CPUs. Blender rendering very much isn't ML, but it is a nice, standardized benchmark: https://opendata.blender.org/ https://opendata.blender.org/ As a point of reference: A P40 has a score of 774 for Blender rendering, and a 4090 has 11,321. There are CPUs ($$$) in the 2000 mark, so about dual P40. It's hard for me to justify a P40-style GPU over something like a 4060Ti 16GB (3800), an Arc a770 16GB (1900), or a 7600XT 16GB (1300). They cost more, but the speed difference is nontrivial, as is the compatibility difference and support life. A lot of work is going into making modern Intel / AMD GPUs supported, while ancient ones are being deprecated.
  - trentnelson 3y ago
    Can that be split across multiple GPUs? i.e. what if I have 4xV100-DGXS-32GBs?
- michaelt 3y ago
  You can run a similarly sized model - Llama 2 70B - at the 'Q4_K_M' quantisation level, with 44 GB of memory [1]. So you can just about fit it on 2x RTX 3090 (which you can buy, used, for around $1100 each) Of course, you can buy quite a lot of hosted model API access or cloud GPU time for that money. [1] https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF
ttul 3y ago
A 70B model is quite accessible; just rent a data center GPU hourly. There are easy deployment services that are getting better all the time. Smaller models can be derived from the big ones to run on a MacBook running Apple Silicon. While the compute won’t be a match for Nvidia hardware, a MacBook can pack 128GB of RAM and run enormous models - albeit slowly.
- moyix 3y ago
  My Macbook has a mere 64GB and that's plenty to run 70B models at 4-bit :) LM Studio is very nice for this.
- deleted 3y ago
  [deleted]
- b33j0r 3y ago
  Ok, well now that we’ve downvoted me below the visibility threshold, I was being sincere. And Altman did say that. I am not a hater. So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory. In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet. Hey, my point remains. The models are going to get too expensive for me, personally, to run locally. I suspect we’ll default into subscriptions to APIs because the upgrade slope is too steep.
  - b33j0r 3y ago
    Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference. I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice. I was uselessly recalculating all of my weights to /dev/null and waste heat. I daemonized my process to control it more tightly, but I still see way above expected vram allocation. Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)
YetAnotherNick 3y ago
> At least as of today… This is the exact opposite of bait and switch. The current model couldn't be un-opensourced and over time it will just become easier to run it. Also unless there is reason to believe that prompt engineering of different model families is very different(which honestly I don't believe), there is no effect of baiting. I believe it will always be the case that best 2-3 models would be closed weights.
whimsicalism 3y ago
the connection to the classic bait-and-switch seems tenuous at best
deleted 3y ago
[deleted]

by b33j0r 3y ago

kuczmama 3y ago

Realistically, what hardware would be required to run this? I assumed a RTX 3090 would be enough?

summarity 3y ago
A Studio Mac with an M1 Ultra (about 2800 USD used) is actually a really cost effective way to run in. Its total system power consumption is really low, even spitting out tokens at full tilt (<250W).
kkzz99 3y ago
RTX 3090 has 24GB of memory, a quantized llama70b takes around 60GB of memory. You can offload a few layers on the gpu, but most of them will run on the CPU with terrible speeds.
- nullc 3y ago
  You're not required to put the whole model in a single GPU. You can buy a 24GB gpu for $150-ish (P40).
  - kuczmama 3y ago
    Wow that's a really good idea. I could potentially buy 4 Nvidia P40's for the same price as a 3090 and run inference on pretty much any model I want.
    - eurekin 3y ago
      Just make sure you're comfortable with manually compiling the bitsandbytes and generally combine a software stack of almost out of date libraries
      - kuczmama 3y ago
        That's a good point. Are you referring to the out of date cuda libraries?
        eurekin 3y ago
        I don't remember exactly (either cuda directly or the cudnn version used by the flashattention)... Anyway, /r/localLlama has few instances of such builds. Might be really worthwhile looking that up before buying
      - nullc 3y ago
        P40 still works with 12.2 at the moment. I used to use K80s (which I think I paid like $50 for!) which turned into a huge mess to deal with older libraries, especially since essentially all ML stuff is on a crazy upgrade cadence with everything constantly breaking even without having to deal with orphaned old software. You can get gpu server chassis that have 10 pci-slots too! for around $2k on ebay. But note that there is a hardware limitation on the PCI-E cards such that each card can only directly communicate with 8 others at a time. Beware, they're LOUD even by the standards of sever hardware. Oh also the nvidia tesla power connectors have cpu-connector like polarity instead of pci-e, so at least in my chassis I needed to adapt them. Also keep in mind that if you aren't using a special gpu chassis, the tesla cards don't have fans, so you have to provide cooling.
    - frognumber 3y ago
      For reference for readers. SUPPORTED ========= * Ada / Hopper / A4xxx (but not A4000) * Ampere / A3xxx * Turing / Quadro RTX / GTX 16xx / RTX 20XX / Volta / Tesla EOL 2023/2024 ============= * Pascal / Quadro P / Geforce GTX 10XX / Tesla Unsupported =========== * Maxwell * Kepler * Fermi * Tesla (yes, this one pops up over and over, chaotically) * Curie Older don't really do GPGPU much. The older cards are also quite slow relative to modern ones! A lot of the ancient workstation cards can run big models cheaply, but (1) with incredible software complexity (2) very slowly, even relative to modern CPUs. Blender rendering very much isn't ML, but it is a nice, standardized benchmark: https://opendata.blender.org/ https://opendata.blender.org/ As a point of reference: A P40 has a score of 774 for Blender rendering, and a 4090 has 11,321. There are CPUs ($$$) in the 2000 mark, so about dual P40. It's hard for me to justify a P40-style GPU over something like a 4060Ti 16GB (3800), an Arc a770 16GB (1900), or a 7600XT 16GB (1300). They cost more, but the speed difference is nontrivial, as is the compatibility difference and support life. A lot of work is going into making modern Intel / AMD GPUs supported, while ancient ones are being deprecated.
- trentnelson 3y ago
  Can that be split across multiple GPUs? i.e. what if I have 4xV100-DGXS-32GBs?
michaelt 3y ago
You can run a similarly sized model - Llama 2 70B - at the 'Q4_K_M' quantisation level, with 44 GB of memory [1]. So you can just about fit it on 2x RTX 3090 (which you can buy, used, for around $1100 each) Of course, you can buy quite a lot of hosted model API access or cloud GPU time for that money. [1] https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF

ttul 3y ago

A 70B model is quite accessible; just rent a data center GPU hourly. There are easy deployment services that are getting better all the time. Smaller models can be derived from the big ones to run on a MacBook running Apple Silicon. While the compute won’t be a match for Nvidia hardware, a MacBook can pack 128GB of RAM and run enormous models - albeit slowly.

moyix 3y ago
My Macbook has a mere 64GB and that's plenty to run 70B models at 4-bit :) LM Studio is very nice for this.
deleted 3y ago
[deleted]
b33j0r 3y ago
Ok, well now that we’ve downvoted me below the visibility threshold, I was being sincere. And Altman did say that. I am not a hater. So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory. In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet. Hey, my point remains. The models are going to get too expensive for me, personally, to run locally. I suspect we’ll default into subscriptions to APIs because the upgrade slope is too steep.
- b33j0r 3y ago
  Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference. I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice. I was uselessly recalculating all of my weights to /dev/null and waste heat. I daemonized my process to control it more tightly, but I still see way above expected vram allocation. Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)

YetAnotherNick 3y ago

> At least as of today… This is the exact opposite of bait and switch. The current model couldn't be un-opensourced and over time it will just become easier to run it. Also unless there is reason to believe that prompt engineering of different model families is very different(which honestly I don't believe), there is no effect of baiting. I believe it will always be the case that best 2-3 models would be closed weights.

whimsicalism 3y ago

the connection to the classic bait-and-switch seems tenuous at best

deleted 3y ago

[deleted]