4 ms·
It doesn't use CoreML so it should work on Intel machines at some speed. If it used the GPU/ANE and was a true large language model then it would only work on
by astrange 4y ago
It doesn't use CoreML so it should work on Intel machines at some speed.
If it used the GPU/ANE and was a true large language model then it would only work on M1 systems because they're unified memory (which nothing except an A100 can match.)
- cromwellian 4y agoI’m wonder how hard it would be to port to OpenXLA which was recently released, that would let you target many CPUs, GPUs, and even TPUs.
- MacsHeadroom 4y ago>which nothing except an A100 can match LLaMA's GPT-3 175B level model, LLaMA-13B, only requires 8GB of VRAM (and no RAM) to run using pre-quantized 4bit weights. So it's hardly a job for an A100. Even the largest model, LLaMA-65B, is only 30GB and inference can be split between two graphics cards with almost no effect on performance.
- pumanoir 4y agoIs there a place to get the prequant'ed model weights?