3 ms·
Yeah, that TVM Vulkan autotuning is incredible. And its not even using the matmul Vulkan extension, I dont think. MLC's 4 bit quantization is "dumb" compared t
by brucethemoose2 3y ago
Yeah, that TVM Vulkan autotuning is incredible. And its not even using the matmul Vulkan extension, I dont think.
MLC's 4 bit quantization is "dumb" compared to llama.cpp, which reduces perplexity (and also explains some of the speed difference), but the biggest missing feature is CPU offloading (which would allow you to run 70B reasonably well on a 4090).
I think the holy grail of local llm inference is llama 70B, run in TVM, split between the GPU and IGP. It feels like we are inches away... All the pieces are there, but there are no front end devs connecting those dots.
- modeless 3y agoWow, using the IGP for the parts that don't fit on the discrete GPU is a great idea.
- brucethemoose2 3y agoYeah. It might be possible in llama.cpp soon, but the vulkan implementation may or may not be fast on the IGP.
- gsuuon 3y agoInterestingly, mlc's web-llm runs much better on my iGPU than dGPU when the model size goes over available dGPU vram. Llama 2 7B runs faster on dGPU, but when I switch to Llama 2 13B suddenly my iGPU outperforms. I think because the iGPU effectively utilizes shared memory?
- brucethemoose2 3y agoMLC has no GPU offloading. I assume the driver itself is spilling over into RAM, and this is essentially a silent malfunction because the driver implementation is extremely slow.