4 ms·
> it’s openly available under an MIT license and can run on CPUs, including Apple’s M2. Weird comparison? The M2 already runs 7 or 13gb LLama and Mistral model
by instagraham 1y ago
> it’s openly available under an MIT license and can run on CPUs, including Apple’s M2.
Weird comparison? The M2 already runs 7 or 13gb LLama and Mistral models with relative ease.
The M-series and Macbooks are so ubiquitous that perhaps we're forgetting how weak the average CPU (think i3 or i5) can be.
- nine_k 1y agoThe M-series have a built-in GPU and unified RAM accessible to both. Running a model on an M-series chip without using the GPU is, imho, pointless. (That said, it's still a long shot from an H100 with a ton of VRAM, or from a Google TPU.) If a model can be "run on a CPU", it should acceptably run on a general-purpose 8-core CPU, like an i7, or i9, or a Ryzen 7, or even an ARM design like a Snapdragon.