3 ms·

400B parameters in a "plug-and-play" form factor for $6k is wild. Meanwhile all the usual "desktop" players are still trying to find a way to make good on thei

400B parameters in a "plug-and-play" form factor for $6k is wild. Meanwhile all the usual "desktop" players are still trying to find a way to make good on their promises to develop their own competitive chips for AI inference and training workloads in the cloud. I'm betting on Nvidia to continue to outperform them. The talent, culture and capabilities gap just feels insurmountable for the next decade at least barring major fumbles from Nvidia.

ekianjo 2y ago
What special sauce do they have to run 400B models fast exactly?
- dragontamer 2y ago
  LPDDR5x. Which is way slower than GDDR6x or GDDR7 let alone HBM. I don't expect these machines to be anywhere near as fast as the hype. 256-bit LPDDR5X is impressive, don't get me wrong. But it's impressive for a CPU platform. It's actually pretty bad for a GPU.
  - ekianjo 2y ago
    Exactly. So where does the "supercomputer" concept comes from exactly?
- zeta0134 2y ago
  From the article: > Huang also revealed ‘Project Digits,’ a new product based on its Grace Blackwell AI-specific architecture that aims to offer at-home AI processing capable of running 200 billion-parameter models locally for a projected retail cost of around $3,000. > There are many exciting things about Project Digits, including the fact that two can be paired to offer 405 billion-parameter model support for ‘just’ $6,000 My experience with running local LLMs is quite limited, but most tools can split the workload between GPUs (or more commonly GPU+CPU) with minimal fuss. It parallelizes fairly well. There may not be any actual secret sauce beyond just having the necessary gobs and gobs of fast memory to load the model into.
  - ekianjo 2y ago
    I don't see any special sauce mentioned to make it run these models fast. If you get a token per second this thing is useless
- deleted 2y ago
  [deleted]
- llm_trw 2y ago
  Quantization and hype.
esperent 2y ago
Is there any info about how many tokens a second you would get with a 400b model? Without that it's like claiming a graphics card can output at 8k, but neglecting to state the frames per second. Suspect, in other words. Outputs at 8k resolution! (+) ...at 5 fps (+)
- mmoskal 2y ago
  I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take. For comparison H100 can read its memory 40 times per second, so if you use it all you can get around 40t/s. Of course in either case you don't have to fill it up, but instead use smaller model or more GPUs.
  - sliken 2y ago
    I've seen similar claims, with nothing in particular to support it. Hopefully Nvidia clarifies this soon. AMD Strix Halo has a 128GB config with 256GB/sec. Apple has a MBP m4 max with 128GB ram for $4,700 with a 546GB/sec interface. Hopefully Nvidia can do better.
llm_nerd 2y ago
nvidia could just release variations of their "gaming" cards with more RAM. There is absolutely nothing stopping them from releasing 64GB or 128GB 5080s and 5090s. But they don't because that would cannibalize the extremely overpriced enterprise offerings. The #1 reason people are forced into the tens of thousands of dollar cards is memory needs. So considering that, ask what niche this device really fills: Is it a new "supercomputer" for the home? Not really, given that it is silicon and memory bandwidth restricted so much that their $600 GPU can beat it soundly on every metric (not surprising when you look at the power and airflow/cooling needs of "real" GPUs) but in scenarios requiring large memory. But while this can hoist those larger models, it is going to be far removed from state of the art. It's neat, but the market for this is being grossly overstated on a lot of these hype advertorials. The large models you'll run on it will be quantized the point of absurdity, not to mention that for 99.9%+ of users, anything short of state of the art is basically useless. It's a neat eGPU of sorts for a Mac or something (they really hype the fact that you use CUDA for this). Still really trying to figure out what value it possibly brings outside of trying to lure a bunch of enthusiasts to blow money on this so they can fiddle with Llama for a week and then realize it's a waste of time.
- throwup238 2y ago
  Would it really cannibalize the enterprise offering? The whole point of that NVIDIA Enterprise license is that the their TOS forbids using anything else in datacenters.
KronisLV 2y ago
Even 200B for 3k USD is really good, 128 GB of memory for that price is surprising! Right now using a two RTX 2080 setup for a pilot project, where it runs ollama and qwen2.5-coder (14B quantized version), a more serious step up from there on a budget would realistically be an RX 7900 XTX (24 GB, though I've had issues with setting up ROCm) for 1k EUR or RTX A5000 (24 GB) for 2.5k EUR in the current market. Honestly, I don't even need the best performance for what I'm trying to do, even an Arc A770 (16 GB) for 350 EUR would be enough to iterate, except that it's not actually supported by ollama and lots of stuff out there: https://github.com/ollama/ollama/blob/main/docs/gpu.md https://github.com/ollama/ollama/blob/main/docs/gpu.md (I know ollama isn't the only solution, but it sucks when the tools you like aren't available)
- redundantly 2y ago
  If performance isn't an issue, why not get a Mac mini?
  - KronisLV 2y ago
    A really good suggestion, I actually used a MacBook Air with an M1 in the first stages and it was fine for prototyping too! Though the local market here is a bit bad. There's a Mac Mini with an M2 Pro and 16 GB for 1.9k EUR, so more expensive than just two GPUs. I'm guessing that the local sellers are trying to profit quite a bit, because on Apple's site, the M4 16 GB version starts at 600 USD (and for 24 GB it's 800 USD and for 32 GB it's 1000 USD). That actually makes it a good option!
- kristianp 2y ago
  What are you doing that requires local over calling an API provider? If you're developing an AI app that makes lots of calls and is designed for a local GPU it makes some sense perhaps?
  - KronisLV 2y ago
    On prem alternatives to ChatGPT and the likes of GitHub Copilot, where you cannot legally send data to cloud services. Underneath everything, ollama can do a lot of the heavy lifting, but it still needs to run somewhere. For decent chat models you probably want a server with either one beefy GPU or a few regular consumer ones (it actually seems to split the load just fine, at least when it comes to Nvidia hardware), whereas for smaller models like autocomplete (where 3B or 1.5B models are enough) you can choose whether to use the same server or run locally.
deleted 2y ago
[deleted]

ekianjo 2y ago

What special sauce do they have to run 400B models fast exactly?

dragontamer 2y ago
LPDDR5x. Which is way slower than GDDR6x or GDDR7 let alone HBM. I don't expect these machines to be anywhere near as fast as the hype. 256-bit LPDDR5X is impressive, don't get me wrong. But it's impressive for a CPU platform. It's actually pretty bad for a GPU.
- ekianjo 2y ago
  Exactly. So where does the "supercomputer" concept comes from exactly?
zeta0134 2y ago
From the article: > Huang also revealed ‘Project Digits,’ a new product based on its Grace Blackwell AI-specific architecture that aims to offer at-home AI processing capable of running 200 billion-parameter models locally for a projected retail cost of around $3,000. > There are many exciting things about Project Digits, including the fact that two can be paired to offer 405 billion-parameter model support for ‘just’ $6,000 My experience with running local LLMs is quite limited, but most tools can split the workload between GPUs (or more commonly GPU+CPU) with minimal fuss. It parallelizes fairly well. There may not be any actual secret sauce beyond just having the necessary gobs and gobs of fast memory to load the model into.
- ekianjo 2y ago
  I don't see any special sauce mentioned to make it run these models fast. If you get a token per second this thing is useless
deleted 2y ago
[deleted]
llm_trw 2y ago
Quantization and hype.

esperent 2y ago

Is there any info about how many tokens a second you would get with a 400b model? Without that it's like claiming a graphics card can output at 8k, but neglecting to state the frames per second. Suspect, in other words. Outputs at 8k resolution! (+) ...at 5 fps (+)

mmoskal 2y ago
I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take. For comparison H100 can read its memory 40 times per second, so if you use it all you can get around 40t/s. Of course in either case you don't have to fill it up, but instead use smaller model or more GPUs.
- sliken 2y ago
  I've seen similar claims, with nothing in particular to support it. Hopefully Nvidia clarifies this soon. AMD Strix Halo has a 128GB config with 256GB/sec. Apple has a MBP m4 max with 128GB ram for $4,700 with a 546GB/sec interface. Hopefully Nvidia can do better.

llm_nerd 2y ago

nvidia could just release variations of their "gaming" cards with more RAM. There is absolutely nothing stopping them from releasing 64GB or 128GB 5080s and 5090s. But they don't because that would cannibalize the extremely overpriced enterprise offerings. The #1 reason people are forced into the tens of thousands of dollar cards is memory needs. So considering that, ask what niche this device really fills: Is it a new "supercomputer" for the home? Not really, given that it is silicon and memory bandwidth restricted so much that their $600 GPU can beat it soundly on every metric (not surprising when you look at the power and airflow/cooling needs of "real" GPUs) but in scenarios requiring large memory. But while this can hoist those larger models, it is going to be far removed from state of the art. It's neat, but the market for this is being grossly overstated on a lot of these hype advertorials. The large models you'll run on it will be quantized the point of absurdity, not to mention that for 99.9%+ of users, anything short of state of the art is basically useless. It's a neat eGPU of sorts for a Mac or something (they really hype the fact that you use CUDA for this). Still really trying to figure out what value it possibly brings outside of trying to lure a bunch of enthusiasts to blow money on this so they can fiddle with Llama for a week and then realize it's a waste of time.

throwup238 2y ago
Would it really cannibalize the enterprise offering? The whole point of that NVIDIA Enterprise license is that the their TOS forbids using anything else in datacenters.

KronisLV 2y ago

Even 200B for 3k USD is really good, 128 GB of memory for that price is surprising! Right now using a two RTX 2080 setup for a pilot project, where it runs ollama and qwen2.5-coder (14B quantized version), a more serious step up from there on a budget would realistically be an RX 7900 XTX (24 GB, though I've had issues with setting up ROCm) for 1k EUR or RTX A5000 (24 GB) for 2.5k EUR in the current market. Honestly, I don't even need the best performance for what I'm trying to do, even an Arc A770 (16 GB) for 350 EUR would be enough to iterate, except that it's not actually supported by ollama and lots of stuff out there: https://github.com/ollama/ollama/blob/main/docs/gpu.md https://github.com/ollama/ollama/blob/main/docs/gpu.md (I know ollama isn't the only solution, but it sucks when the tools you like aren't available)

redundantly 2y ago
If performance isn't an issue, why not get a Mac mini?
- KronisLV 2y ago
  A really good suggestion, I actually used a MacBook Air with an M1 in the first stages and it was fine for prototyping too! Though the local market here is a bit bad. There's a Mac Mini with an M2 Pro and 16 GB for 1.9k EUR, so more expensive than just two GPUs. I'm guessing that the local sellers are trying to profit quite a bit, because on Apple's site, the M4 16 GB version starts at 600 USD (and for 24 GB it's 800 USD and for 32 GB it's 1000 USD). That actually makes it a good option!
kristianp 2y ago
What are you doing that requires local over calling an API provider? If you're developing an AI app that makes lots of calls and is designed for a local GPU it makes some sense perhaps?
- KronisLV 2y ago
  On prem alternatives to ChatGPT and the likes of GitHub Copilot, where you cannot legally send data to cloud services. Underneath everything, ollama can do a lot of the heavy lifting, but it still needs to run somewhere. For decent chat models you probably want a server with either one beefy GPU or a few regular consumer ones (it actually seems to split the load just fine, at least when it comes to Nvidia hardware), whereas for smaller models like autocomplete (where 3B or 1.5B models are enough) you can choose whether to use the same server or run locally.

deleted 2y ago

[deleted]