5 ms·

Please give us consumer cards with more than 24GB VRAM, Nvidia. It was a slap in the face when the 4090 had the same memory capacity as the 3090. A6000 is 500

by enlyth 4y ago

Please give us consumer cards with more than 24GB VRAM, Nvidia. It was a slap in the face when the 4090 had the same memory capacity as the 3090. A6000 is 5000 dollars, ain't no hobbyist at home paying for that.

andrewstuart 4y ago
Nvidia don't want consumers using consumer GPUs for business. If you are a business user then you must pay Nvidia gargantuan amounts of money. This is the outcome of a market leader with no real competition - you pay much more for lower power than the consumer GPUs and you are forced into ujsing their business GPUs through software license restrictions on the drivers.
- Melatonic 4y ago
  That was always why the Titan line was so great - they typically unlocked features in between Quadro and Gaming cards. Sometimes it was subtle (like very good FP32 AND FP16 performance) or adding full 10 bit colour support if you had a Titan only. Now it seems like they have opened up even more of those features to consumer cards (at least the creative ones) with the studio drivers.
  - koheripbal 4y ago
    Isn't a new Titan RTX 4090 coming out soon?
    - enlyth 4y ago
      An alleged photo of an engineering sample was spotted in the wild a while ago, but no one knows if it's actually going to end up being a thing you can buy.
  - andrewstuart 4y ago
    Hmmm ... "Studio Drivers" ... how are these tangibly different to gaming drivers? According to this, the difference seems to be that Studio Drivers are older and better tested, nothing else. https://nvidia.custhelp.com/app/answers/detail/a_id/4931/~/nvidia-studio-faqs https://nvidia.custhelp.com/app/answers/detail/a_id/4931/~/n... What am I missing in my understanding of Studio Drivers? """ How do Studio Drivers differ from Game Ready Drivers (GRD)? In 2014, NVIDIA created the Game Ready Driver program to provide the best day-0 gaming experience. In order to accomplish this, the release cadence for Game Ready Drivers is driven by the release of major new game content giving our driver team as much time as possible to work on a given title. In similar fashion, NVIDIA now offers the Studio Driver program. Designed to provide the ultimate in functionality and stability for creative applications, Studio Drivers provide extensive testing against top creative applications and workflows for the best performance possible, and support any major creative app updates to ensure that you are ready to update any apps on Day 1. ""
- koheripbal 4y ago
  We're NOT business users, we just want to run our own LLM at home. Given the size of LLMs, this should be possible with just a little bit of extra VRAM.
  - enlyth 4y ago
    Exactly, we're just below that sweet spot right now. For example on 24GB, Llama 30B runs only in 4bit mode and very slowly, but I can imagine a RLHF finetuned 30B or 65B version running in at least 8bit would be actually useful, and you could run it on your own computer easily.
    - bick_nyers 4y ago
      Do you know where the cutoff is? Does 32GB VRAM give us 30B int8 with/without a RLHF layer? I don't think 5090 is going to go straight to 48GB, I'm thinking either 32 or 40GB (if not 24GB).
    - riku_iki 4y ago
      > For example on 24GB, Llama 30B runs only in 4bit mode and very slowly why do you think adding vram, but not cores will make it run faster?..
      - enlyth 4y ago
        I've been told the 4 bit quantization slows it down, but don't quote me on this since I was unable to benchmark at 8 bit locally In any case, you're right it might not be as significant, however, the quality of the output increases with 8/16bit, and running 65B is completely impossible on 24GB
        riku_iki 4y ago
        It's not impossible, there are several projects which load model layer by layer for execution from the disk or ram, but it will be much slower.
  - 4y ago
nullc 4y ago
Nvidia can't do a large 'consumer' card without cannibalizing their commercial ML business. ATI doesn't have that problem. ATI seems to be holding the idiot ball. Port stable diffusion and clip to their hardware. Train an upsized version sized for a 48GB card. Release a prosumer 48gb card... get huge uptake from artists and creators using the tech.

by enlyth 4y ago

andrewstuart 4y ago

Nvidia don't want consumers using consumer GPUs for business. If you are a business user then you must pay Nvidia gargantuan amounts of money. This is the outcome of a market leader with no real competition - you pay much more for lower power than the consumer GPUs and you are forced into ujsing their business GPUs through software license restrictions on the drivers.

Melatonic 4y ago
That was always why the Titan line was so great - they typically unlocked features in between Quadro and Gaming cards. Sometimes it was subtle (like very good FP32 AND FP16 performance) or adding full 10 bit colour support if you had a Titan only. Now it seems like they have opened up even more of those features to consumer cards (at least the creative ones) with the studio drivers.
- koheripbal 4y ago
  Isn't a new Titan RTX 4090 coming out soon?
  - enlyth 4y ago
    An alleged photo of an engineering sample was spotted in the wild a while ago, but no one knows if it's actually going to end up being a thing you can buy.
- andrewstuart 4y ago
  Hmmm ... "Studio Drivers" ... how are these tangibly different to gaming drivers? According to this, the difference seems to be that Studio Drivers are older and better tested, nothing else. https://nvidia.custhelp.com/app/answers/detail/a_id/4931/~/nvidia-studio-faqs https://nvidia.custhelp.com/app/answers/detail/a_id/4931/~/n... What am I missing in my understanding of Studio Drivers? """ How do Studio Drivers differ from Game Ready Drivers (GRD)? In 2014, NVIDIA created the Game Ready Driver program to provide the best day-0 gaming experience. In order to accomplish this, the release cadence for Game Ready Drivers is driven by the release of major new game content giving our driver team as much time as possible to work on a given title. In similar fashion, NVIDIA now offers the Studio Driver program. Designed to provide the ultimate in functionality and stability for creative applications, Studio Drivers provide extensive testing against top creative applications and workflows for the best performance possible, and support any major creative app updates to ensure that you are ready to update any apps on Day 1. ""
koheripbal 4y ago
We're NOT business users, we just want to run our own LLM at home. Given the size of LLMs, this should be possible with just a little bit of extra VRAM.
- enlyth 4y ago
  Exactly, we're just below that sweet spot right now. For example on 24GB, Llama 30B runs only in 4bit mode and very slowly, but I can imagine a RLHF finetuned 30B or 65B version running in at least 8bit would be actually useful, and you could run it on your own computer easily.
  - bick_nyers 4y ago
    Do you know where the cutoff is? Does 32GB VRAM give us 30B int8 with/without a RLHF layer? I don't think 5090 is going to go straight to 48GB, I'm thinking either 32 or 40GB (if not 24GB).
  - riku_iki 4y ago
    > For example on 24GB, Llama 30B runs only in 4bit mode and very slowly why do you think adding vram, but not cores will make it run faster?..
    - enlyth 4y ago
      I've been told the 4 bit quantization slows it down, but don't quote me on this since I was unable to benchmark at 8 bit locally In any case, you're right it might not be as significant, however, the quality of the output increases with 8/16bit, and running 65B is completely impossible on 24GB
      - riku_iki 4y ago
        It's not impossible, there are several projects which load model layer by layer for execution from the disk or ram, but it will be much slower.
- 4y ago

nullc 4y ago

Nvidia can't do a large 'consumer' card without cannibalizing their commercial ML business. ATI doesn't have that problem. ATI seems to be holding the idiot ball. Port stable diffusion and clip to their hardware. Train an upsized version sized for a 48GB card. Release a prosumer 48gb card... get huge uptake from artists and creators using the tech.