4 ms·

I just tried this and ran into a few hiccups before I got it working (on a Windows desktop with a NVIDIA GeForce RTX 3080 Ti) WSL outputs this error (hidden by

by j2kun 3y ago

I just tried this and ran into a few hiccups before I got it working (on a Windows desktop with a NVIDIA GeForce RTX 3080 Ti) WSL outputs this error (hidden by the one-liner's map to dev/null) > error: APE is running on WIN32 inside WSL. You need to run: sudo sh -c 'echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop' Then zsh hits: `zsh: exec format error: ./llava-v1.5-7b-q4-main.llamafile` so I had to run it in bash. (The title says bash, I know, but it seems weird that it wouldn't work in zsh) It also reports a warning that GPU offloading is not supported, but it's probably a WSL thing (I don't do any GPU programming on my windows machine).

ronsor 3y ago
This is a bug in ZSH... which was fixed months ago. So you must have an old version of ZSH.
- j2kun 3y ago
  Thanks for the tip! I have 5.8.1 but I see I'm one major version behind.
- jart 3y ago
  I fixed that bug in zsh two years ago. It got released in zsh 5.9. https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841bc62b06e37134c6e51ea https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841... I'm going to be answering questions about it until my dying day.
augusto-moura 3y ago
In zsh you can fix it by prepending `sh` before the command: `sh ./llava-v1.5-7b-q4-main.llamafile`, it's a quirk with zsh and APE [1] [1]: https://justine.lol/ape.html https://justine.lol/ape.html
venusenvy47 3y ago
I was thinking of trying this on my Windows machine with an RTX 4070 but it sounds like the GPU isn't used in WSL. Was your testing really slow when using just the CPU?
- j2kun 3y ago
  It was like 30s compared to Justine's 45. Apparently I have the right drivers installed, but apparently you need to compile the executable with the right toolkit (?) to make it work in WSL: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2 https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-...
  - jart 3y ago
    Yes you need to install CUDA and MSVC for GPU. But here's some good news! We just rolled our own GEMM functions so llamafile doesn't have to depend on cuBLAS anymore. That means llamafile 0.4 (which I'm shipping today) will have GPU on Windows that works out of the box, since not depending on cuBLAS anymore means that I'm able to compile a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82 https://github.com/Mozilla-Ocho/llamafile/pull/82

by j2kun 3y ago

ronsor 3y ago

This is a bug in ZSH... which was fixed months ago. So you must have an old version of ZSH.

j2kun 3y ago
Thanks for the tip! I have 5.8.1 but I see I'm one major version behind.
jart 3y ago
I fixed that bug in zsh two years ago. It got released in zsh 5.9. https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841bc62b06e37134c6e51ea https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841... I'm going to be answering questions about it until my dying day.

augusto-moura 3y ago

In zsh you can fix it by prepending `sh` before the command: `sh ./llava-v1.5-7b-q4-main.llamafile`, it's a quirk with zsh and APE [1] [1]: https://justine.lol/ape.html https://justine.lol/ape.html

venusenvy47 3y ago

I was thinking of trying this on my Windows machine with an RTX 4070 but it sounds like the GPU isn't used in WSL. Was your testing really slow when using just the CPU?

j2kun 3y ago
It was like 30s compared to Justine's 45. Apparently I have the right drivers installed, but apparently you need to compile the executable with the right toolkit (?) to make it work in WSL: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2 https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-...
- jart 3y ago
  Yes you need to install CUDA and MSVC for GPU. But here's some good news! We just rolled our own GEMM functions so llamafile doesn't have to depend on cuBLAS anymore. That means llamafile 0.4 (which I'm shipping today) will have GPU on Windows that works out of the box, since not depending on cuBLAS anymore means that I'm able to compile a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82 https://github.com/Mozilla-Ocho/llamafile/pull/82