3 ms·
What Apple Silicon brings to the table is not simply just performance, but a large amount of unified memory that can be used by the GPU (which are needed for in
by cyber_kinetist 3y ago
What Apple Silicon brings to the table is not simply just performance, but a large amount of unified memory that can be used by the GPU (which are needed for inference of large deep neural networks like LLMs).
A top-of-the-line Mac Studio will give you 192 GB of unified RAM in less than $7000. Meanwhile a H100 from NVIDIA with 80 GB of VRAM will cost you like $30000...
- api 3y agoIf the software were a little better Apple Silicon would be by far the most cost effective rig today for DIY LLM research or training. 192GiB RAM is enough to train or inference Falcon 180B in RAM at 8-bit resolution.