4 ms·
Certainly big for China, but if it can't run CUDA then it's not going to help them catch up in the AI race
by OsrsNeedsf2P 1y ago
Certainly big for China, but if it can't run CUDA then it's not going to help them catch up in the AI race
- aswanson 1y agoMaybe they'll make a better abstraction layer than cuda.
- chickenzzzzu 1y agoVulkan already achieves 95% of cuda, with the remaining 5% being scheduling.
- LeoPanthera 1y agoMacs can't run CUDA but everyone is buying those for AI anyway.
- jszymborski 1y agoGP was likely referring to training, not inference.
- wolfgangK 1y agoOnly those who don't care/know about prompt processing speed are buying Macs for LLM inference.
- esseph 1y agoDon't know and don't care are definitely things that I could be, but it also makes sense if they want to keep lookups private.
- com2kid 1y agoEven 40 tokens per second is plenty enough for real time usage. The average person reads at ~4 words per second, 40 tokens per second is going to be 15-20 words per second. Even useful models like gemma3 27b are hitting 22 t/s on 4bit quants. You aren't going to be reformatting gigabytes of PDFs or anything, but for a lot of common use cases, those speeds are fine.
- chermi 1y agoI thought they were already working around cuda?
- sliken 1y agoDunno, training maybe, for inference pytorch and llama seem more important.