4 ms·
Can these be used for local inference on large models? I'm assuming the 128G of memory is like system memory, not like GPU VRAM.
by probablydan 1y ago
Can these be used for local inference on large models? I'm assuming the 128G of memory is like system memory, not like GPU VRAM.
- deleted 1y ago[deleted]
- bigyabai 1y agoYes, but it is substantially cheaper and usually faster to buy a Jetson Orin chip or build an x86 homelab.
- sgillen 1y agoIt has a unified memory architecture, so the 128G is shared directly between CPU and GPU. Though it's slower than dGPU VRAM.
- CamperBob2 1y agoMemory bandwidth is less than 300 GB/sec, looking at the data sheet. So it won't really be any faster at local inference than a Mac Pro or a decent x86 box. It appears to be an embedded DGX Spark, at the end of the day.