3 ms·
Apple silicon is great for low power desktops and laptops, but they don't actually have groundbreaking performance relative to what we've got in the server spac
by Aurornis 3y ago
Apple silicon is great for low power desktops and laptops, but they don't actually have groundbreaking performance relative to what we've got in the server space. If you dropped the M2 Ultra from the $4000 into a server, it would perform about the same as a $1500 AMD 7950X3D based server (this is a common budget server setup with ECC) in CPU tasks. Stick a common GPU in there and you're running circles around the M2 Silicon in GPU tasks.
The Apple Silicon is great at really low power work, but if you dial desktop or server GPU power limits down they also become quite efficient. The marginal cost of electricity is cheaper than buying more hardware, so nVidia and others run their parts deep into the diminishing returns part of the curve to maximize performance at the expense of power efficiency.
- cyber_kinetist 3y agoWhat Apple Silicon brings to the table is not simply just performance, but a large amount of unified memory that can be used by the GPU (which are needed for inference of large deep neural networks like LLMs). A top-of-the-line Mac Studio will give you 192 GB of unified RAM in less than $7000. Meanwhile a H100 from NVIDIA with 80 GB of VRAM will cost you like $30000...
- api 3y agoIf the software were a little better Apple Silicon would be by far the most cost effective rig today for DIY LLM research or training. 192GiB RAM is enough to train or inference Falcon 180B in RAM at 8-bit resolution.