3 ms·

Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a hug

Fantastic. For those of who who swim in the Microsoft ecosystem, and do not compile Linux apps from code, what Linux dustro would run this without fixing a huge number of dependencies? It seems like someone would have included Llama.cpp in their distro, ready-to-run. Yes, I'm an idiot.

jart 3y ago
llamafile runs on all Linux distros since ~2009. It doesn't have any dependencies. It'd probably even run as the init process too (if you assimilate it). The only thing it needs is the Linux 2.6.18+ kernel application binary interface. If you have an SELinux policy, then you may need to tune things, and on some distros you might have to install APE Loader for binfmt_misc, but that's about it. See the Gotchas in the README. Also goes without saying that llamafile runs on WIN32 too, if that's the world you're most comfortable with. It even runs on BSD distros and MacOS. All in a single file.
- FragenAntworten 3y ago
  It doesn't seem to run on NixOS, though I'm new to Nix and may be missing something. $ ./llava-v1.5-7b-q4-server.llamafile --help ./llava-v1.5-7b-q4-server.llamafile: line 60: /bin/mkdir: No such file or directory Regardless, this (and Cosmopolitan) are amazing work - thank you!
  - jart 3y ago
    The APE shell script needs to run /bin/mkdir in order to map the embedded ELF executable in memory. It should be possible for you to work around this on Linux by installing our binfmt_misc interpreter: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" That way the only file you'll need to whitelist with Nix is /usr/bin/ape. You could also try just vendoring the 8kb ape executable in your Nix project, and simply executing `./ape ./llamafile`.
    - FragenAntworten 3y ago
      `./ape ./llamafile` worked immediately and without problems I can see - thank you!
askiiart 3y ago
> It seems like someone would have included Llama.cpp in their distro, ready-to-run. Assuming you mean installable with a package manager, not preinstalled on a distro, that requires that some maintainer decide it's worthwhile to add it and maintain it. Distros are pretty selective in what they add to their repos, but there's probably a tool for building .deb or .rpm packages of llama.cpp, and probably a repository for it, but as far as I know no distro has llama.cpp in its repos. Or Arch Linux's AUR system is much more open, and it indeed has llama-cpp (4 versions of it!), though it requires a helper, such as yay, if you want to install it and keep it up-to-date as if it were a normal package. So Arch has it installable with a package manager if you use yay to supplement pacman. https://aur.archlinux.org/packages?O=0&K=llama-cpp https://aur.archlinux.org/packages?O=0&K=llama-cpp
lordfrikk 3y ago
On a fresh install of Ubuntu 23.10, I made it executable (`chmod +x llamafile`) and it worked just like that. After installing `nvidia-cuda-toolkit` it also worked with my GPU. So far the most seamless method I've tried.

jart 3y ago

llamafile runs on all Linux distros since ~2009. It doesn't have any dependencies. It'd probably even run as the init process too (if you assimilate it). The only thing it needs is the Linux 2.6.18+ kernel application binary interface. If you have an SELinux policy, then you may need to tune things, and on some distros you might have to install APE Loader for binfmt_misc, but that's about it. See the Gotchas in the README. Also goes without saying that llamafile runs on WIN32 too, if that's the world you're most comfortable with. It even runs on BSD distros and MacOS. All in a single file.

FragenAntworten 3y ago
It doesn't seem to run on NixOS, though I'm new to Nix and may be missing something. $ ./llava-v1.5-7b-q4-server.llamafile --help ./llava-v1.5-7b-q4-server.llamafile: line 60: /bin/mkdir: No such file or directory Regardless, this (and Cosmopolitan) are amazing work - thank you!
- jart 3y ago
  The APE shell script needs to run /bin/mkdir in order to map the embedded ELF executable in memory. It should be possible for you to work around this on Linux by installing our binfmt_misc interpreter: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" That way the only file you'll need to whitelist with Nix is /usr/bin/ape. You could also try just vendoring the 8kb ape executable in your Nix project, and simply executing `./ape ./llamafile`.
  - FragenAntworten 3y ago
    `./ape ./llamafile` worked immediately and without problems I can see - thank you!

askiiart 3y ago

> It seems like someone would have included Llama.cpp in their distro, ready-to-run. Assuming you mean installable with a package manager, not preinstalled on a distro, that requires that some maintainer decide it's worthwhile to add it and maintain it. Distros are pretty selective in what they add to their repos, but there's probably a tool for building .deb or .rpm packages of llama.cpp, and probably a repository for it, but as far as I know no distro has llama.cpp in its repos. Or Arch Linux's AUR system is much more open, and it indeed has llama-cpp (4 versions of it!), though it requires a helper, such as yay, if you want to install it and keep it up-to-date as if it were a normal package. So Arch has it installable with a package manager if you use yay to supplement pacman. https://aur.archlinux.org/packages?O=0&K=llama-cpp https://aur.archlinux.org/packages?O=0&K=llama-cpp

lordfrikk 3y ago

On a fresh install of Ubuntu 23.10, I made it executable (`chmod +x llamafile`) and it worked just like that. After installing `nvidia-cuda-toolkit` it also worked with my GPU. So far the most seamless method I've tried.