3 ms·
The mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs ar
by pletnes 4y ago
The mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs are capable of running LLMs.
- zamadatix 4y agoThe article and linked GitHub project both say this runs on the CPU. The GitHub project notes the quantization doesn’t work right with other systems for some reason but if you have enough RAM to quantize it you have enough RAM to run it without quantizing it anyways.
- pletnes 4y agoAha. Then I guess performance is the only issue. I guess it should run fine on Intel.