5 ms·

I wish US sellers would start doing this, too. Nvidia's continuing refusal to put higher amounts of VRAM into its "affordable" cards combined with AMD's head sc

I wish US sellers would start doing this, too. Nvidia's continuing refusal to put higher amounts of VRAM into its "affordable" cards combined with AMD's head scratching resistance to entering the ML space leaves average consumers who want to run bigger models locally with very few options.

izacus 3y ago
I think it leaves them with an option nVidia wants right? Paying them more for the VRAM.
- cyanydeez 3y ago
  right. it's bizzare watching tech heads confused about price segregation like it's some phantom force and not one of the glorious tools of monopolies and price gouging.
- michaelt 3y ago
  Even that, nvidia doesn't make easy. The RTX 4090 is so huge and power-hungry you can barely fit two into a case, let alone any more than that. And even if you replace your case, and your power supply (and maybe your motherboard and CPU too, gotta have enough PCIe lanes) you'll still only have 48GB of vram. OK, so we can't use an RTX GPU. Good luck even understanding the rest of nvidia's product range. The A40? "The World’s Most Powerful Data Center GPU for Visual Computing". V100? "The most advanced data center GPU ever built". H100? "Unprecedented performance, scalability, and security for every data center". A100? "Unprecedented acceleration at every scale to power the world’s highest-performing elastic data centers". H200? "The world’s most powerful GPU" L4? "The breakthrough universal accelerator". Good luck figuring out who sells them in your country, or if they're in stock. Oh, and remember to read the entire spec in excruciating detail. That $2700 L4 24GB is slower than an RTX 4090. Think you'll launch something in the cloud? With Google Cloud half the cards are only available in some regions - and even if you're in a region that offers a given GPU, maybe there's a shortage and support tells you to just keep requesting GPUs repeatedly until you get lucky.
  - MichaelZuo 3y ago
    Nvidia specifically indicates the VRAM size and whether the product is suitable, so you pick the ones with enough to satisfy your expected needs. What’s there to not understand?
    - beeboobaa 3y ago
      That VRAM isn't everything, apparently.
    - michaelt 3y ago
      * Is a given product their most advanced, or was it merely their most advanced when the marketing copy was written 5 years ago? * Is the product actually available for purchase today, and at what price? * In what form factors - Full height? Half height? Two slots? Three? SXM? Does it have a fan built in? * What is the relative performance? Half the time the specs all quote different numbers. Why does one product quote 'single precision performance' and 'rt core performance' and 'tensor performance' while another quotes the 'tensor cores' and 'shader cores' and another quotes the 'FP64', 'FP32', 'FP16', 'BFLOAT16', 'TF32' and 'INT8'? And don't imagine you're going to get away with ignoring those specs. A $2700 L4 24GB is slower than an RTX 4090, for example, because it's for power-efficient servers or something.
      - MichaelZuo 3y ago
        Most of this information is indicated on their website or online spec sheets? Like I said Nvidia themselves indicate whether it’s suitable. And if your still confused or doubt the accuracy, or don’t want to spend any time reading spec sheets, I imagine the sales channel folks will be able to guarantee it in writing for a fee.
        michaelt 3y ago
        > Most of this information is indicated on their website or online spec sheets? You'd think so, wouldn't you? But nvidia is not that smart. They have decided that the GPUs should be split over at least three different pages, and those pages should be camouflaged. I can only assume the marketing team are judged based on time spent on site or number of pages viewed, rather than on sales made. When you go to the home page, point to products and click on "NVIDIA RTX / Quadro" you might expect to find the RTX 4090 and some Quadro products. You will in fact find neither - the RTX 4090 is under 'GeForce' and the 'Quadro' brand is no longer used. Maybe on the home page you choose "NVIDIA RTX-Powered AI Workstations" - that'll give you a list of their workstation-suitable cards for ML, right? No, that page contains no products at all. The only option is to Find A Partner which links you to a bunch of partners - several of whom do not in fact sell AI Workstations. Other partners sell AI workstations... without GPUs. As far as HP is concerned, $4000 only gets you the base model workstation, with 32GB of RAM and a 1TB hard disk. If you want GPUs with that, you'll need to call the sales team, who might perhaps deign to sell you one. Or perhaps you're at nvidia's home page, and you're looking for data centre grade GPUs? For that product page, simply choose whether you mean the DGX, EGX, IGX, HGX, MGX or OVX platform? But they make it very easy to find the keynote speech by His Excellency Omar Sultan Al Olama on the latest breakthroughs in AI. He does, in fairness, have a very appropriate surname.
OJFord 3y ago
> I wish US sellers would start doing this, too. Isn't that exactly what's happening? From OP: > The GeForce RTX 2080 Ti 22GB has found its way outside China. eBay seller customgpu_official, an upgrade and repair store in Palo Alto, California, sells a similar MSI GeForce RTX 2080 Ti Aero with 22GB for $499. The store is touting the graphics card as a budget alternative for students and startups that want to get their feet wet in AI workloads. The graphics card is allegedly stable in Stable diffusion, large language models (LLMs), and Llama 2. According to the merchant's website, it has sold over 500 units of the GeForce RTX 2080 Ti 22GB. The China stuff is just background story I think. > Nvidia's continuing refusal to put higher amounts of VRAM into its "affordable" cards [...] leaves average consumers who want to run bigger models locally with very few options. That's market segmentation and product differentiation for you. You want that much VRAM you're doing ML; you want to do ML you can pay us ML prices. What 'average consumer' wants to 'run bigger models locally' anyway! If they made the 'affordable' cards with VRAM ranging from say 8GB to 80GB, it would be only gamers buying at the bottom end and only MLers at the top. And they can charge the latter a lot more - so even if they did what you want, you'd end up with Applesque pricing bumps for beefier/more GDDR chips.
- KeplerBoy 3y ago
  That's not how this traditionally worked. A few generations ago board partners were allowed to solder whatever RAM they wanted onto the GPU board. Boards from different OEMs were different in meaningful ways and OEMs had the chance to differentiate themselves from the competition with some actual engineering. More RAM, multiple GPUs per board, AGP to PCIe chips, you name it. Nowadays Nvidia restricts is partners from all that fun and undercuts them with their own models. No wonder EVGA quit this game.
  - HWR_14 3y ago
    Yes, and NViDIA didn't care when it was all selling to gamers. Now it wants crypto bros (obv a little historical there) and ML people who are using it to make money to pay a higher price point. So they do it by restricting the options available at the affordable level. See also, what happens when you want to use group administration features in Windows.
alecco 3y ago
Yeah, plus no NVLink for RTX 40xx cards. And 4060 with only half PCIe lanes.
- wruza 3y ago
  Are PCIe lanes really that important for ML? I've seen reports that x4 isn't any different from x16 for a single card.
  - borissk 3y ago
    If the card onboard memory is not enough, the card can use the PC RAM over the PCIe connection - and then the PCIe bandwidth is very important. But the main reason for less lanes is to cut costs.
    - wruza 3y ago
      Yes, but it starts to crawl on 16x anyway. My firsthand experience with LLMs is that there's little difference between VRAM-to-RAM spills and running on just RAM. Same for LoRA training. When I accidentally disturbed too much VRAM by e.g. parallel upscales, it went from 1.3it/s to 30-40s/it and never recovered. NVidia even added a new setting in their control panel to disable CUDA RAM fallback to work around unintentional slowdowns in lowvram cards, afaiu.

izacus 3y ago

I think it leaves them with an option nVidia wants right? Paying them more for the VRAM.

cyanydeez 3y ago
right. it's bizzare watching tech heads confused about price segregation like it's some phantom force and not one of the glorious tools of monopolies and price gouging.
michaelt 3y ago
Even that, nvidia doesn't make easy. The RTX 4090 is so huge and power-hungry you can barely fit two into a case, let alone any more than that. And even if you replace your case, and your power supply (and maybe your motherboard and CPU too, gotta have enough PCIe lanes) you'll still only have 48GB of vram. OK, so we can't use an RTX GPU. Good luck even understanding the rest of nvidia's product range. The A40? "The World’s Most Powerful Data Center GPU for Visual Computing". V100? "The most advanced data center GPU ever built". H100? "Unprecedented performance, scalability, and security for every data center". A100? "Unprecedented acceleration at every scale to power the world’s highest-performing elastic data centers". H200? "The world’s most powerful GPU" L4? "The breakthrough universal accelerator". Good luck figuring out who sells them in your country, or if they're in stock. Oh, and remember to read the entire spec in excruciating detail. That $2700 L4 24GB is slower than an RTX 4090. Think you'll launch something in the cloud? With Google Cloud half the cards are only available in some regions - and even if you're in a region that offers a given GPU, maybe there's a shortage and support tells you to just keep requesting GPUs repeatedly until you get lucky.
- MichaelZuo 3y ago
  Nvidia specifically indicates the VRAM size and whether the product is suitable, so you pick the ones with enough to satisfy your expected needs. What’s there to not understand?
  - beeboobaa 3y ago
    That VRAM isn't everything, apparently.
  - michaelt 3y ago
    * Is a given product their most advanced, or was it merely their most advanced when the marketing copy was written 5 years ago? * Is the product actually available for purchase today, and at what price? * In what form factors - Full height? Half height? Two slots? Three? SXM? Does it have a fan built in? * What is the relative performance? Half the time the specs all quote different numbers. Why does one product quote 'single precision performance' and 'rt core performance' and 'tensor performance' while another quotes the 'tensor cores' and 'shader cores' and another quotes the 'FP64', 'FP32', 'FP16', 'BFLOAT16', 'TF32' and 'INT8'? And don't imagine you're going to get away with ignoring those specs. A $2700 L4 24GB is slower than an RTX 4090, for example, because it's for power-efficient servers or something.
    - MichaelZuo 3y ago
      Most of this information is indicated on their website or online spec sheets? Like I said Nvidia themselves indicate whether it’s suitable. And if your still confused or doubt the accuracy, or don’t want to spend any time reading spec sheets, I imagine the sales channel folks will be able to guarantee it in writing for a fee.
      - michaelt 3y ago
        > Most of this information is indicated on their website or online spec sheets? You'd think so, wouldn't you? But nvidia is not that smart. They have decided that the GPUs should be split over at least three different pages, and those pages should be camouflaged. I can only assume the marketing team are judged based on time spent on site or number of pages viewed, rather than on sales made. When you go to the home page, point to products and click on "NVIDIA RTX / Quadro" you might expect to find the RTX 4090 and some Quadro products. You will in fact find neither - the RTX 4090 is under 'GeForce' and the 'Quadro' brand is no longer used. Maybe on the home page you choose "NVIDIA RTX-Powered AI Workstations" - that'll give you a list of their workstation-suitable cards for ML, right? No, that page contains no products at all. The only option is to Find A Partner which links you to a bunch of partners - several of whom do not in fact sell AI Workstations. Other partners sell AI workstations... without GPUs. As far as HP is concerned, $4000 only gets you the base model workstation, with 32GB of RAM and a 1TB hard disk. If you want GPUs with that, you'll need to call the sales team, who might perhaps deign to sell you one. Or perhaps you're at nvidia's home page, and you're looking for data centre grade GPUs? For that product page, simply choose whether you mean the DGX, EGX, IGX, HGX, MGX or OVX platform? But they make it very easy to find the keynote speech by His Excellency Omar Sultan Al Olama on the latest breakthroughs in AI. He does, in fairness, have a very appropriate surname.

OJFord 3y ago

> I wish US sellers would start doing this, too. Isn't that exactly what's happening? From OP: > The GeForce RTX 2080 Ti 22GB has found its way outside China. eBay seller customgpu_official, an upgrade and repair store in Palo Alto, California, sells a similar MSI GeForce RTX 2080 Ti Aero with 22GB for $499. The store is touting the graphics card as a budget alternative for students and startups that want to get their feet wet in AI workloads. The graphics card is allegedly stable in Stable diffusion, large language models (LLMs), and Llama 2. According to the merchant's website, it has sold over 500 units of the GeForce RTX 2080 Ti 22GB. The China stuff is just background story I think. > Nvidia's continuing refusal to put higher amounts of VRAM into its "affordable" cards [...] leaves average consumers who want to run bigger models locally with very few options. That's market segmentation and product differentiation for you. You want that much VRAM you're doing ML; you want to do ML you can pay us ML prices. What 'average consumer' wants to 'run bigger models locally' anyway! If they made the 'affordable' cards with VRAM ranging from say 8GB to 80GB, it would be only gamers buying at the bottom end and only MLers at the top. And they can charge the latter a lot more - so even if they did what you want, you'd end up with Applesque pricing bumps for beefier/more GDDR chips.

KeplerBoy 3y ago
That's not how this traditionally worked. A few generations ago board partners were allowed to solder whatever RAM they wanted onto the GPU board. Boards from different OEMs were different in meaningful ways and OEMs had the chance to differentiate themselves from the competition with some actual engineering. More RAM, multiple GPUs per board, AGP to PCIe chips, you name it. Nowadays Nvidia restricts is partners from all that fun and undercuts them with their own models. No wonder EVGA quit this game.
- HWR_14 3y ago
  Yes, and NViDIA didn't care when it was all selling to gamers. Now it wants crypto bros (obv a little historical there) and ML people who are using it to make money to pay a higher price point. So they do it by restricting the options available at the affordable level. See also, what happens when you want to use group administration features in Windows.

alecco 3y ago

Yeah, plus no NVLink for RTX 40xx cards. And 4060 with only half PCIe lanes.

wruza 3y ago
Are PCIe lanes really that important for ML? I've seen reports that x4 isn't any different from x16 for a single card.
- borissk 3y ago
  If the card onboard memory is not enough, the card can use the PC RAM over the PCIe connection - and then the PCIe bandwidth is very important. But the main reason for less lanes is to cut costs.
  - wruza 3y ago
    Yes, but it starts to crawl on 16x anyway. My firsthand experience with LLMs is that there's little difference between VRAM-to-RAM spills and running on just RAM. Same for LoRA training. When I accidentally disturbed too much VRAM by e.g. parallel upscales, it went from 1.3it/s to 30-40s/it and never recovered. NVidia even added a new setting in their control panel to disable CUDA RAM fallback to work around unintentional slowdowns in lowvram cards, afaiu.