3 ms·
It was like 30s compared to Justine's 45. Apparently I have the right drivers installed, but apparently you need to compile the executable with the right toolki
by j2kun 3y ago
It was like 30s compared to Justine's 45. Apparently I have the right drivers installed, but apparently you need to compile the executable with the right toolkit (?) to make it work in WSL: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2 https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-...
- jart 3y agoYes you need to install CUDA and MSVC for GPU. But here's some good news! We just rolled our own GEMM functions so llamafile doesn't have to depend on cuBLAS anymore. That means llamafile 0.4 (which I'm shipping today) will have GPU on Windows that works out of the box, since not depending on cuBLAS anymore means that I'm able to compile a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82 https://github.com/Mozilla-Ocho/llamafile/pull/82