4 ms·
Some discussion of this box here: https://news.ycombinator.com/item?id=43425935 https://news.ycombinator.com/item?id=43425935 TLDR, AMD machines based on HX395
by fancyfredbot 2y ago
Some discussion of this box here: https://news.ycombinator.com/item?id=43425935 https://news.ycombinator.com/item?id=43425935
TLDR, AMD machines based on HX395 offer comparable memory bandwidth and size at lower cost ($2000) but lack the high speed networking and compute power (60TF FP16 on AMD and 120TF on NVIDIA)
Apple M4 Max costs 23% more with equivalent memory, has twice the memory bandwidth but again no fast networking and significantly less compute (30TF FP16 on M4 Max).
For inference on large language models the extra memory bandwidth probably means M4 Max is the fastest option unless you use large batches or long contexts. For training large models needing more than 128GB RAM two of these Nvidia boxes is probably fastest.
- a2128 2y agoAnother thing lacking on AMD machines is the software support. Not just that a lot of stuff is CUDA-only still, but AMD themselves struggle to support their own hardware for ROCm. According to their docs ROCm for HX395 is supported on Windows[0] but not Linux[1]. Though further reading indicates the Windows version is also just a subset of Linux ROCm that seems to not support PyTorch or anything. I really don't know what they're doing over there but I hate it. [0] https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat... [1] https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...
- fancyfredbot 2y agoIt's a huge mess. I think people are running LLMs on HX395 using the Vulkan backend in llama.cpp which will work on Linux. No RocM or pytorch though.