3 ms·

I was thinking of trying this on my Windows machine with an RTX 4070 but it sounds like the GPU isn't used in WSL. Was your testing really slow when using just

I was thinking of trying this on my Windows machine with an RTX 4070 but it sounds like the GPU isn't used in WSL. Was your testing really slow when using just the CPU?

j2kun 3y ago
It was like 30s compared to Justine's 45. Apparently I have the right drivers installed, but apparently you need to compile the executable with the right toolkit (?) to make it work in WSL: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2 https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-...
- jart 3y ago
  Yes you need to install CUDA and MSVC for GPU. But here's some good news! We just rolled our own GEMM functions so llamafile doesn't have to depend on cuBLAS anymore. That means llamafile 0.4 (which I'm shipping today) will have GPU on Windows that works out of the box, since not depending on cuBLAS anymore means that I'm able to compile a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82 https://github.com/Mozilla-Ocho/llamafile/pull/82