3 ms·

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the s

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

NaomiLehman 1y ago
because of possible hardware-accelerated matmul in GPU cores?
- christkv 1y ago
  Massive memory bandwidth for the most part. M3 ultra had like 810 Gb/s vs ~300 for the DGX Spark. Also you can get up to 512 GB memory with a 256 GB config as well
- aurareturn 1y ago
  Yes. Matmul in M5 GPU, memory bandwidth, consumer/prosumer friendly OS, and they are just excellent portable laptops.
arresin 1y ago
I wish I could run Linux on them (the m5)
- hoppp 1y ago
  There was an arch linux version that supports apple silicon
  - porphyra 1y ago
    Asahi Linux has long switched from Arch to Fedora (though a janky Arch version still exists). But they don't support anything newer than M2.
spaceywilly 1y ago
What is the value proposition for buying one of these vs renting time on similar hardware from a cloud provider?
- xmichael909 1y ago
  I don't think there is one. Honestly this version 1 is dead on arrival.
jamesblonde 1y ago
Given that most of Nvidia's enterprise software products are all single server designed to run on DGX boxes, like NIMs, this makes sense. I am still amazed at how many companies buy a ton of DGX boxes and then are surprised that Nvidia does not have any Kubernetes native platform for training and inferencing across all the DGX machines. The Run.ai acquisition did not change anything, as you leave all the work to the user to integrate with distributed training frameworks like Ray or scalable inference platforms, like KServe/vLLM.
kirillzubovsky 1y ago
Fascinating that we didn't have to wait too long. Apple announced M5 this morning. Does it compare though?
teleforce 1y ago
If I understand correctly the DGX is for the development, and the AGX Thor is more geared toward local LLM inferencing [1],[2]. [1] (Updated) NVIDIA Jetson AGX Thor Developer Kit to Launch in Mid-August with 2070 TFLOPS AI Performance, Priced at $3499: https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-developer-kit-to-launch-in-mid-august-with-2070-tflops-ai-performance-priced-at-3499/ https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-devel... [2] AAEON Announces BOXER-8741AI with NVIDIA Jetson Thor T5000 Module: https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nvidia-jetson-thor-t5000-module/ https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nv...

NaomiLehman 1y ago

because of possible hardware-accelerated matmul in GPU cores?

christkv 1y ago
Massive memory bandwidth for the most part. M3 ultra had like 810 Gb/s vs ~300 for the DGX Spark. Also you can get up to 512 GB memory with a 256 GB config as well
aurareturn 1y ago
Yes. Matmul in M5 GPU, memory bandwidth, consumer/prosumer friendly OS, and they are just excellent portable laptops.

arresin 1y ago

I wish I could run Linux on them (the m5)

hoppp 1y ago
There was an arch linux version that supports apple silicon
- porphyra 1y ago
  Asahi Linux has long switched from Arch to Fedora (though a janky Arch version still exists). But they don't support anything newer than M2.

spaceywilly 1y ago

What is the value proposition for buying one of these vs renting time on similar hardware from a cloud provider?

xmichael909 1y ago
I don't think there is one. Honestly this version 1 is dead on arrival.

jamesblonde 1y ago

Given that most of Nvidia's enterprise software products are all single server designed to run on DGX boxes, like NIMs, this makes sense. I am still amazed at how many companies buy a ton of DGX boxes and then are surprised that Nvidia does not have any Kubernetes native platform for training and inferencing across all the DGX machines. The Run.ai acquisition did not change anything, as you leave all the work to the user to integrate with distributed training frameworks like Ray or scalable inference platforms, like KServe/vLLM.

kirillzubovsky 1y ago

Fascinating that we didn't have to wait too long. Apple announced M5 this morning. Does it compare though?

teleforce 1y ago

If I understand correctly the DGX is for the development, and the AGX Thor is more geared toward local LLM inferencing [1],[2]. [1] (Updated) NVIDIA Jetson AGX Thor Developer Kit to Launch in Mid-August with 2070 TFLOPS AI Performance, Priced at $3499: https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-developer-kit-to-launch-in-mid-august-with-2070-tflops-ai-performance-priced-at-3499/ https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-devel... [2] AAEON Announces BOXER-8741AI with NVIDIA Jetson Thor T5000 Module: https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nvidia-jetson-thor-t5000-module/ https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nv...