3 ms·
How’s it looking for a six year old MacBook? Not there yet? Does this still use the gpu?
by bilsbie 4y ago
How’s it looking for a six year old MacBook?
Not there yet?
Does this still use the gpu?
- simonw 4y agoI believe lambda.cpp has been designed for at least an M1 - no idea if there are options for running LLaMA on older hardware.
- astrange 4y agoIt doesn't use CoreML so it should work on Intel machines at some speed. If it used the GPU/ANE and was a true large language model then it would only work on M1 systems because they're unified memory (which nothing except an A100 can match.)
- cromwellian 4y agoI’m wonder how hard it would be to port to OpenXLA which was recently released, that would let you target many CPUs, GPUs, and even TPUs.
- MacsHeadroom 4y ago>which nothing except an A100 can match LLaMA's GPT-3 175B level model, LLaMA-13B, only requires 8GB of VRAM (and no RAM) to run using pre-quantized 4bit weights. So it's hardly a job for an A100. Even the largest model, LLaMA-65B, is only 30GB and inference can be split between two graphics cards with almost no effect on performance.
- pumanoir 4y agoIs there a place to get the prequant'ed model weights?