6 ms·
Well, that’s disappointing since the Mac Studio 128GB is $3,499. If Apple happens to launch a Mac Mini with 128GB RAM it would eat Nvidia Sparks’ lunch every da
by nialse 1y ago
Well, that’s disappointing since the Mac Studio 128GB is $3,499. If Apple happens to launch a Mac Mini with 128GB RAM it would eat Nvidia Sparks’ lunch every day.
- moondev 1y agoJust don't try to run a NCCL
- zackangelo 1y agoWouldn't you be able to test nccl if you had 2 of these?
- newman314 1y agoAgreed. I also wonder why they chose to test against a Mac Studio with only 64GB instead of 128GB.
- yvbbrjdr 1y agoHi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.
- ggerganov 1y agoFYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp4096 | 3564.31 ± 9.91 | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 | 53.93 ± 1.71 | | gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp4096 | 1792.32 ± 34.74 | | gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 | 38.54 ± 3.10 |
- yvbbrjdr 1y agoI see! Do you know what's causing the slowdown for ollama? They should be using the same backend..
- __mharrison__ 1y agoCurious to how this compares to running on a Mac.
- pjmlp 1y agoOnly if it runs CUDA, MLX / Metal isn't comparable as ecosystem. People that keep pushing for Apple gear tend to forget Apple has decided what industry considers industry standards, proprietary or not, aren't made available on their hardware. Even if Metal is actually a cool API to program for.
- thom 1y agoIt depends what you're doing. I can get valuable work done with the subset of Torch supported on MPS and I'm grateful for the speed and RAM of modern Mac systems. JAX support is worse but hopefully both continue to develop.
- omneity 1y agoCUDA is equally proprietary and not an industry standard though, unless you were thinking of Vulcan/OpenCL which doesn’t bring much in this situation.
- NewsaHackO 1y agoCUDA isn't the industry standard? What is then?
- pjmlp 1y agoYes it is an industry standard, there is even a technical term for it. It is called De facto standard, which you can check in your favourite dictionary.