4 ms·

I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have an

by bt1a 3y ago

I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inference can take advantage of splitting the layers among multiple GPUs, high memory GPUs that aren't super expensive are desirable. To share a personal perspective, I have a desktop with a 3090 and an M1 Max Studio with 64GB of memory. I use the M1 for local LLMs because I can use up to 57~GB of memory, even though the output (in terms of tok/s) is much slower than ones I can fit on a 3090.

Dalewyn 3y ago
>models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.
- hnfong 3y ago
  It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worrying rate these days, but let's keep the discussion on topic...
  - Dalewyn 3y ago
    I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.
    - xoranth 3y ago
      People have engineered solutions to make what is available practical (see all the various quantization schemes that have come out). It is just that there's a limit to how much you can compress the models.
    - dekhn 3y ago
      I would say that increasing RAM to avoid engineering a solution has long been a successful strategy. i learned my RAM lesson when I bought my first real linux PC. it had 4MB of RAM, which was enough to run X, bash, xterm, and emacs. But once I ran all that and also wanted to compile with g++, it would start swapping, which in the days of slow hard drives, was death to productivity. I spent $200 to double to 8MB, and then another $200 to double to 16MB, and then finally, $200 to max out the RAM on my machine-- 32MB! And once I did that everything flew. Rather than attempting to solve the problem by making emacs (eight megs and constantly swapping) use less RAM, or find a way to hack without X, I deployed money to max out my machine (which was practical, but not realistically available to me unless I gave up other things in life for the short term). Not only was I more productive, I used that time to work on other engineering problems which helped build my career, while also learning an important lesson about swapping/paging. People demand RAM and what was not practically available is often available 2 years later as standard. Seems like a great approach to me, especially if you don't have enough smart engineers to work around problems like that (see "How would you sort 4M integers in 2M of RAM?")
      - watersb 3y ago
        > I spent $200 to double to 8MB, and then another $200 to double to 16MB, and then finally, $200 to max out the RAM on my machine-- 32MB! Thank you. Now I feel a log better for dropping $700 on the 32MB of RAM when I built my first rig.
    - whiplash451 3y ago
      By the same logic, we’d still be writing assembly code on 640KB RAM machines in 2024.
    - michaelt 3y ago
michaelbrave 3y ago
Right now I have a 3090TI so it's not worth it for me to upgrade to a 4090, but I do run into Vram constraints a lot, especially with merging stable diffusion models, especially as the models get larger (XL-Cascade-etc). As I move toward running multiple LLMs at a time I run into similar problems. I would gladly buy a card that ran a touch slower but had massive Vram, especially if it was affordable, but I guess that puts me into that camp of enthusiasts you mentioned.

by bt1a 3y ago

Dalewyn 3y ago

>models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.

hnfong 3y ago
It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worrying rate these days, but let's keep the discussion on topic...
- Dalewyn 3y ago
  I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.
  - xoranth 3y ago
    People have engineered solutions to make what is available practical (see all the various quantization schemes that have come out). It is just that there's a limit to how much you can compress the models.
  - dekhn 3y ago
    I would say that increasing RAM to avoid engineering a solution has long been a successful strategy. i learned my RAM lesson when I bought my first real linux PC. it had 4MB of RAM, which was enough to run X, bash, xterm, and emacs. But once I ran all that and also wanted to compile with g++, it would start swapping, which in the days of slow hard drives, was death to productivity. I spent $200 to double to 8MB, and then another $200 to double to 16MB, and then finally, $200 to max out the RAM on my machine-- 32MB! And once I did that everything flew. Rather than attempting to solve the problem by making emacs (eight megs and constantly swapping) use less RAM, or find a way to hack without X, I deployed money to max out my machine (which was practical, but not realistically available to me unless I gave up other things in life for the short term). Not only was I more productive, I used that time to work on other engineering problems which helped build my career, while also learning an important lesson about swapping/paging. People demand RAM and what was not practically available is often available 2 years later as standard. Seems like a great approach to me, especially if you don't have enough smart engineers to work around problems like that (see "How would you sort 4M integers in 2M of RAM?")
    - watersb 3y ago
      > I spent $200 to double to 8MB, and then another $200 to double to 16MB, and then finally, $200 to max out the RAM on my machine-- 32MB! Thank you. Now I feel a log better for dropping $700 on the 32MB of RAM when I built my first rig.
  - whiplash451 3y ago
    By the same logic, we’d still be writing assembly code on 640KB RAM machines in 2024.
  - michaelt 3y ago

michaelbrave 3y ago

Right now I have a 3090TI so it's not worth it for me to upgrade to a 4090, but I do run into Vram constraints a lot, especially with merging stable diffusion models, especially as the models get larger (XL-Cascade-etc). As I move toward running multiple LLMs at a time I run into similar problems. I would gladly buy a card that ran a touch slower but had massive Vram, especially if it was affordable, but I guess that puts me into that camp of enthusiasts you mentioned.