4 ms·
Perhaps they are referring to default GPU allocation that is 75% of the unified memory, but it is trivial to increase it.
by rz2k 1y ago
Perhaps they are referring to default GPU allocation that is 75% of the unified memory, but it is trivial to increase it.
- jauntywundrkind 1y agoThe GPU memory allocation refers to how capacity is alloted, not bandwidth. Sounds like the same 256-bit/quad-channel 8000MHz lpddr5 you can get today with Strix Halo.
- rz2k 1y ago384GB is 75% of 512GB. The M3 Ultra bandwidth is over 800GB/s, though potentially less in practice. Using an M3 Ultra I think the performance is pretty remarkable for inference and concerns about prompt processing being slow in particular are greatly exaggerated. Maybe the advantage of the DGX Spark will be for training or fine tuning.
- vid 1y agoI very consistently see people say prompt processing is slow for larger context sizes ("notoriously slow"), something that is much less of an issue with eg CUDA setups.
- Art9681 1y agoDepends on the model. gpt-oss-120b will easily crunch large prompts in a few seconds. It's remarkable. It's gpt-4-mini at home.