4 ms·
Is nobody running this on a PC?
by seydor 4y ago
Is nobody running this on a PC?
- pletnes 4y agoThe mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs are capable of running LLMs.
- zamadatix 4y agoThe article and linked GitHub project both say this runs on the CPU. The GitHub project notes the quantization doesn’t work right with other systems for some reason but if you have enough RAM to quantize it you have enough RAM to run it without quantizing it anyways.
- pletnes 4y agoAha. Then I guess performance is the only issue. I guess it should run fine on Intel.
- washadjeffmad 4y agoThis is a Mac thread. Virtually everyone is using a PC. I test between a HEDT Linux system and a recent Intel build on Windows 11. Took me around 8 hours to build and deploy 4bit. 7B and 13B worked great, still working on the quantized 30B weights.
- lerela 4y agoThe FP16 7B version runs on my Ubuntu XPS with 32GB memory, ~300ms/token. 13B also works but results aren't really good (the model will loop after a few sentences) so parameters probably need tuning. So far I'm unable to reliably generate outputs in a different language than English, the model will very quickly start to translate (even if it's not asked) or just switch to English.