6 ms·

Nothing above 24GB? Not much point in upgrading, then. Memory size remains the biggest bottleneck.

capableweb 4y ago
Seems they don't want to eat up market share of their "professional" cards with the consumer cards. Want above 24GB? Better put up money for the data center cards.
- bick_nyers 4y ago
  Just spend $6k on a Quadro to get 4090 performance but with 48GB VRAM /s
numlock86 4y ago
Same here. 3090 Ti but memory is usually the bottleneck. And workstation GPUs like the A100 are just too expensive ...
- capableweb 4y ago
  Guess A6000 is in the middle of those two options, comes with 48GB of memory. Not so cheap if you're just using it for gaming, but if you're using your GPU for professional work, could be worth the price.
  - numlock86 4y ago
    €5500 compared to €1200 for just double the memory at the price of less performance is not really worth it in my opinion. > Not so cheap if you're just using it for gaming I am not even sure if there is any recent game that can utilize 24GB of a 3090.
    - ShockedUnicorn 4y ago
      Does VRChat count as a game? In a very populated instance with 50+ people it can take up to about 20gb of vram. Honestly everything VR takes up huge amounts of vram, it's not possible to run No Man's Sky at max settings with only 8gb vram, and you still might need to dial down some settings with 12gb vram. This is with a Quest 2, I'm guessing headsets with higher resolution will be even more demanding.
    - capableweb 4y ago
      > €5500 compared to €1200 for just double the memory at the price of less performance is not really worth it in my opinion. It's not just double the memory, is faster memory as well. Although, less CUDA cores and also doesn't include latest NVENC encoder/decoder, nor AV1 encoder, so... Tradeoffs I guess :) For some people it will be worth it, for others no.
rejectfinite 4y ago
>Memory size remains the biggest bottleneck For what? AI shit? These are for games broski
- lake_vincent 4y ago
  Well, yeah, broski. GPUs are just highly optimized matrix multipliers, so they're for any computation involving matrix multiplication...such computations arise in computer graphics, crypto mining, and machine learning (aka AI shit). And yes, memory size is important for some applications. Is that okay?
  - rrss 4y ago
    this simplification to “everything needs matrix multiply and GPUs are matrix multipliers” doesn’t really match the actual architecture. the hardware for low-precision matrix multiplications in ML is mostly independent of the hardware that’s typically utilized for vertex and fragment shaders in computer graphics. It really only gets used for graphics if using DLSS. It’s not used at all for (integer) computations in cryptomining. Are they for high-precision scientific simulations because it’s possible to multiply matrices on them? These cards are probably quite shit at double-precision matrix multiplications.
  - thomastjeffery 4y ago
    And these ones aren't optimized for machine learning, and that's OK, too.
  - rejectfinite 4y ago
    >And yes, memory size is important for some applications. Is that okay? Yes and they make Quadros for those :)
- Filligree 4y ago
  A 24GB card is not for gaming. It can do games, which is convenient, but no game currently made needs that much.
  - 4y ago
arecurrence 4y ago
Today is a gaming product announcement and I'm not sure that games have a need even beyond 12 GB today. I suspect that the 4090 price not budging much is telling as far as what ram amount demand has been focused on for games. I assume they will soon have a professional card announcement that includes 48GB+ cards. Assuming that the high ram cards have improvements similar to this generational leap in the gaming market, they will be in high demand.
- rejectfinite 4y ago
  >today Sure, but you dont buy a high end GPU and then buy the best one next generation... I mean some do sure but a high end GPU like this you keep for a few years...
  - arecurrence 4y ago
    I believe part of the problem is they went with 24 GB gaming cards a couple years ago and the need for that didn't materialize. Hence, cold feet for launching with 48 GB because there still isn't a need for more than 12 GB and 48 GB impinges on their high margin professional market.
    - vanchor3 4y ago
      > I believe part of the problem is they went with 24 GB gaming cards a couple years ago and the need for that didn't materialize. To be fair the ability to easily purchase the cards didn't materialize either until recently. Why develop for what your consumer can't get?
fuzzy2 4y ago
Is it though? I guess it is for machine learning. But do games require that much memory? Remember that GeForce is the consumer line. I would expect it to mainly target the gamer market.
- Filligree 4y ago
  Realistically speaking, the 3090 targets enthusiasts and hobbyist ML work. There’s not much need for a card of that size for gaming, but there’s definitely room for a card that can do ML and gaming.
wellthisisgreat 4y ago
Yeah, I have the same sentiment. I hope they come out with 4090TI with more memory though
alkonaut 4y ago
If we disregard AI/ML applications for a second, are there really cards that are regularly limiting FPS in games between e.g. 12GB and 16GB?
- philliphaydon 4y ago
  Depends on the game. 4k ultra settings can definitively chew a lot of memory. If you’re doing 1440p then 12g is more than enough.
  - Aardwolf 4y ago
    Why would screen resolution affect memory that much? The textures would be the same, and 4K monitor pixels is megabytes, not gigabytes
    - philliphaydon 4y ago
      you can have different texture resolutions. And more pixels to draw on the screen while trying to push out 100s of frames per second.
      - Aardwolf 4y ago
        At least in the games I play usually texture detail and screen resolution are two independent settings (and if you play windowed it can be any arbitrary resolution), I am mostly a PC gamer though, not sure what console games do.
        philliphaydon 4y ago
        You can absolutely get away with 12gb, 10gb, even 8gb vram and play 4k high refresh rate. But also doesn't account for poor programming. I remember a reddit post about God of War using 15gb of vram after a while of playing, probably a memory leak. Having available vram just gives yourself headroom for newer games with higher quality textures, there was a game which name eludes me which shipped with 8k resolution textures for 4k gaming so it used like 10gb of vram? Or something along those lines. But it can also be used for pre-caching stuff, like if you were playing WoW and the next zone was already precached so you don't need to load it from the disk. I think that new tech Sony and MS were working on, direct storage access? Will play a bigger and better role than higher vram. I only game at 1440p so I don't care for more than 8gb of vram.
blagie 4y ago
Any clue about how well ML frameworks do with multiple cards? I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.
- Filligree 4y ago
  For training? The note cards the better, up to a couple hundred. Custom code may be required, but your bottleneck will be the GPU count. For inference? You can’t generally shard an AI across multiple cards, but you can run multiple instances. Depends on what your trying to do. In no case will it be a simple linear speedup, and you certainly can’t add the memory together.
  - blagie 4y ago
    I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference; it's a dev machine - Training is a bit of fine-tuning; I'm not building big models from scratch. - Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run. - It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time. I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration. Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models. I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.

Seems they don't want to eat up market share of their "professional" cards with the consumer cards. Want above 24GB? Better put up money for the data center cards.

bick_nyers 4y ago
Just spend $6k on a Quadro to get 4090 performance but with 48GB VRAM /s

numlock86 4y ago

Same here. 3090 Ti but memory is usually the bottleneck. And workstation GPUs like the A100 are just too expensive ...

capableweb 4y ago
Guess A6000 is in the middle of those two options, comes with 48GB of memory. Not so cheap if you're just using it for gaming, but if you're using your GPU for professional work, could be worth the price.
- numlock86 4y ago
  €5500 compared to €1200 for just double the memory at the price of less performance is not really worth it in my opinion. > Not so cheap if you're just using it for gaming I am not even sure if there is any recent game that can utilize 24GB of a 3090.
  - ShockedUnicorn 4y ago
    Does VRChat count as a game? In a very populated instance with 50+ people it can take up to about 20gb of vram. Honestly everything VR takes up huge amounts of vram, it's not possible to run No Man's Sky at max settings with only 8gb vram, and you still might need to dial down some settings with 12gb vram. This is with a Quest 2, I'm guessing headsets with higher resolution will be even more demanding.
  - capableweb 4y ago
    > €5500 compared to €1200 for just double the memory at the price of less performance is not really worth it in my opinion. It's not just double the memory, is faster memory as well. Although, less CUDA cores and also doesn't include latest NVENC encoder/decoder, nor AV1 encoder, so... Tradeoffs I guess :) For some people it will be worth it, for others no.

rejectfinite 4y ago

>Memory size remains the biggest bottleneck For what? AI shit? These are for games broski

lake_vincent 4y ago
Well, yeah, broski. GPUs are just highly optimized matrix multipliers, so they're for any computation involving matrix multiplication...such computations arise in computer graphics, crypto mining, and machine learning (aka AI shit). And yes, memory size is important for some applications. Is that okay?
- rrss 4y ago
  this simplification to “everything needs matrix multiply and GPUs are matrix multipliers” doesn’t really match the actual architecture. the hardware for low-precision matrix multiplications in ML is mostly independent of the hardware that’s typically utilized for vertex and fragment shaders in computer graphics. It really only gets used for graphics if using DLSS. It’s not used at all for (integer) computations in cryptomining. Are they for high-precision scientific simulations because it’s possible to multiply matrices on them? These cards are probably quite shit at double-precision matrix multiplications.
- thomastjeffery 4y ago
  And these ones aren't optimized for machine learning, and that's OK, too.
- rejectfinite 4y ago
  >And yes, memory size is important for some applications. Is that okay? Yes and they make Quadros for those :)
Filligree 4y ago
A 24GB card is not for gaming. It can do games, which is convenient, but no game currently made needs that much.
- 4y ago

arecurrence 4y ago

Today is a gaming product announcement and I'm not sure that games have a need even beyond 12 GB today. I suspect that the 4090 price not budging much is telling as far as what ram amount demand has been focused on for games. I assume they will soon have a professional card announcement that includes 48GB+ cards. Assuming that the high ram cards have improvements similar to this generational leap in the gaming market, they will be in high demand.

rejectfinite 4y ago
>today Sure, but you dont buy a high end GPU and then buy the best one next generation... I mean some do sure but a high end GPU like this you keep for a few years...
- arecurrence 4y ago
  I believe part of the problem is they went with 24 GB gaming cards a couple years ago and the need for that didn't materialize. Hence, cold feet for launching with 48 GB because there still isn't a need for more than 12 GB and 48 GB impinges on their high margin professional market.
  - vanchor3 4y ago
    > I believe part of the problem is they went with 24 GB gaming cards a couple years ago and the need for that didn't materialize. To be fair the ability to easily purchase the cards didn't materialize either until recently. Why develop for what your consumer can't get?

fuzzy2 4y ago

Is it though? I guess it is for machine learning. But do games require that much memory? Remember that GeForce is the consumer line. I would expect it to mainly target the gamer market.

Filligree 4y ago
Realistically speaking, the 3090 targets enthusiasts and hobbyist ML work. There’s not much need for a card of that size for gaming, but there’s definitely room for a card that can do ML and gaming.

wellthisisgreat 4y ago

Yeah, I have the same sentiment. I hope they come out with 4090TI with more memory though

alkonaut 4y ago

If we disregard AI/ML applications for a second, are there really cards that are regularly limiting FPS in games between e.g. 12GB and 16GB?

philliphaydon 4y ago
Depends on the game. 4k ultra settings can definitively chew a lot of memory. If you’re doing 1440p then 12g is more than enough.
- Aardwolf 4y ago
  Why would screen resolution affect memory that much? The textures would be the same, and 4K monitor pixels is megabytes, not gigabytes
  - philliphaydon 4y ago
    you can have different texture resolutions. And more pixels to draw on the screen while trying to push out 100s of frames per second.
    - Aardwolf 4y ago
      At least in the games I play usually texture detail and screen resolution are two independent settings (and if you play windowed it can be any arbitrary resolution), I am mostly a PC gamer though, not sure what console games do.
      - philliphaydon 4y ago
        You can absolutely get away with 12gb, 10gb, even 8gb vram and play 4k high refresh rate. But also doesn't account for poor programming. I remember a reddit post about God of War using 15gb of vram after a while of playing, probably a memory leak. Having available vram just gives yourself headroom for newer games with higher quality textures, there was a game which name eludes me which shipped with 8k resolution textures for 4k gaming so it used like 10gb of vram? Or something along those lines. But it can also be used for pre-caching stuff, like if you were playing WoW and the next zone was already precached so you don't need to load it from the disk. I think that new tech Sony and MS were working on, direct storage access? Will play a bigger and better role than higher vram. I only game at 1440p so I don't care for more than 8gb of vram.

blagie 4y ago

Any clue about how well ML frameworks do with multiple cards? I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.

Filligree 4y ago
For training? The note cards the better, up to a couple hundred. Custom code may be required, but your bottleneck will be the GPU count. For inference? You can’t generally shard an AI across multiple cards, but you can run multiple instances. Depends on what your trying to do. In no case will it be a simple linear speedup, and you certainly can’t add the memory together.
- blagie 4y ago
  I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference; it's a dev machine - Training is a bit of fine-tuning; I'm not building big models from scratch. - Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run. - It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time. I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration. Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models. I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.