14 ms·

Thanks! That rules them out completely EDIT: For training

ShamelessC 3y ago
Are you aware that cards containing “LLM” (40-80GB) levels of VRAM cost substantially more and the status quo for consumer cards hovers around 4-12GB, only going to 24GB for top end cards?
- eurekin 3y ago
  Of course
  - ShamelessC 3y ago
    Your comment, pre-edit, had something of a severe tone given that consideration. Having said that, I've trained/finetuned image models just fine on an RTX 2070 Super with 8 GB of VRAM. This was back when doing so was more fruitful than simply training a more robust model in the first place. Given that is the current status quo - I'm curious what sort of training you're doing whole-network that actually produces results that are noticeably better than doing something few-shot during inference or doing LoRA finetuning? The latter brings you back into the realm of tuning on low-VRAM configs. In general, a single GPU's memory constraints are one of many when training a model _from scratch_. In that case, you're bottlenecked by data and data parallelism. You don't need one or a few GPU's, you need more than would fit in a consumer setup in the first place.
    - deleted 3y ago
      [deleted]
- samspenc 3y ago
  And this is exactly the way NVidia intends to keep it, methinks. Give the consumers / gamers a consumer-priced GPU with a max of 16-24 GB VRAM for the high-end models. By consumer-priced, I mean $500-2000. And make anyone interested in AI / ML / LLM / 3D / creatives pay $3000-10000 for GPUs that are similar in performance but have much higher VRAM. Then top it out with six-figure (or higher) priced GPUs for the FAANG companies which can afford them for their data centers and currently contribute the most revenue (and profit) to NVidia.
loloquwowndueo 3y ago
Why?
- baobabKoodaa 3y ago
  With the current generation of text and image generators, 24GB is the sweet spot
  - Salgat 3y ago
    I wouldn't be shocked if the 5090 is also the only one with 24GB. Seems like NVidia is trying their hardest to suppress memory increases.
    - anonym29 3y ago
      When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k, selling consumer cards with 24GB for $2k is almost a form of charity, by comparison. AMD and Intel GPUs do not have the software ecosystem for AI workloads that Nvidia does, though AMD is rapidly improving. Nvidia has had an effective monopoly on the AI hardware space for the last year or so, and continues to have an effective near-monopoly, but that won't last forever as AMD and Intel catch up. The VRAM is one of the largest differentiators of their cards. Sufficient VRAM allows you to run huge LLMs like 65B in-memory, which is orders of magnitudes faster than system RAM + CPU. Smaller amounts of VRAM require swapping between VRAM and system RAM and incur a major performance penalty. Businesses are fighting to fork over $50k+/card for 40/80GB cards with the same processor as the 24GB consumer cards - it doesn't make economic sense for Nvidia to offer more on the consumer cards, lest they start cannibalizing demand for the enterprise cards.
      - dragonwriter 3y ago
        > When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k ADA 6000 RTX (48GB) — an enterprise (workstation) cars — is about $10K sticker price. The differentiator between those and data center cards in the 40GB-80GB range is, obviously, not just VRAM.
Ekaros 3y ago
Very good. Maybe as a gamer I can get one eventually for reasonable price.
- deleted 3y ago
  [deleted]
- glonq 3y ago
  Depending on your definition of 'reasonable', that ship sailed a few years ago and ain't ever coming back

Are you aware that cards containing “LLM” (40-80GB) levels of VRAM cost substantially more and the status quo for consumer cards hovers around 4-12GB, only going to 24GB for top end cards?

eurekin 3y ago
Of course
- ShamelessC 3y ago
  Your comment, pre-edit, had something of a severe tone given that consideration. Having said that, I've trained/finetuned image models just fine on an RTX 2070 Super with 8 GB of VRAM. This was back when doing so was more fruitful than simply training a more robust model in the first place. Given that is the current status quo - I'm curious what sort of training you're doing whole-network that actually produces results that are noticeably better than doing something few-shot during inference or doing LoRA finetuning? The latter brings you back into the realm of tuning on low-VRAM configs. In general, a single GPU's memory constraints are one of many when training a model _from scratch_. In that case, you're bottlenecked by data and data parallelism. You don't need one or a few GPU's, you need more than would fit in a consumer setup in the first place.
  - deleted 3y ago
    [deleted]
samspenc 3y ago
And this is exactly the way NVidia intends to keep it, methinks. Give the consumers / gamers a consumer-priced GPU with a max of 16-24 GB VRAM for the high-end models. By consumer-priced, I mean $500-2000. And make anyone interested in AI / ML / LLM / 3D / creatives pay $3000-10000 for GPUs that are similar in performance but have much higher VRAM. Then top it out with six-figure (or higher) priced GPUs for the FAANG companies which can afford them for their data centers and currently contribute the most revenue (and profit) to NVidia.

loloquwowndueo 3y ago

Why?

baobabKoodaa 3y ago
With the current generation of text and image generators, 24GB is the sweet spot
- Salgat 3y ago
  I wouldn't be shocked if the 5090 is also the only one with 24GB. Seems like NVidia is trying their hardest to suppress memory increases.
  - anonym29 3y ago
    When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k, selling consumer cards with 24GB for $2k is almost a form of charity, by comparison. AMD and Intel GPUs do not have the software ecosystem for AI workloads that Nvidia does, though AMD is rapidly improving. Nvidia has had an effective monopoly on the AI hardware space for the last year or so, and continues to have an effective near-monopoly, but that won't last forever as AMD and Intel catch up. The VRAM is one of the largest differentiators of their cards. Sufficient VRAM allows you to run huge LLMs like 65B in-memory, which is orders of magnitudes faster than system RAM + CPU. Smaller amounts of VRAM require swapping between VRAM and system RAM and incur a major performance penalty. Businesses are fighting to fork over $50k+/card for 40/80GB cards with the same processor as the 24GB consumer cards - it doesn't make economic sense for Nvidia to offer more on the consumer cards, lest they start cannibalizing demand for the enterprise cards.
    - dragonwriter 3y ago
      > When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k ADA 6000 RTX (48GB) — an enterprise (workstation) cars — is about $10K sticker price. The differentiator between those and data center cards in the 40GB-80GB range is, obviously, not just VRAM.

Ekaros 3y ago

Very good. Maybe as a gamer I can get one eventually for reasonable price.

deleted 3y ago
[deleted]
glonq 3y ago
Depending on your definition of 'reasonable', that ship sailed a few years ago and ain't ever coming back