6 ms·

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. N

It keeps saying the phrase “model you can run locally”, but despite days of trying, I failed to compile any of the GitHub repos associated with these models. None of the Python dependencies are strongly versioned, and “something” happened to the CUDA compatibility of one of them about a month ago. The original developers “got lucky” but now nobody else can compile this stuff. After years of using only C# and Rust, both of which have sane package managers with semantic versioning, lock files, reproducible builds, and even SHA checksums the Python package ecosystem looks ridiculously immature and even childish. Seriously, can anyone here build a docker image for running these models on CUDA? I think right now it’s borderline impossible, but I’d be happy to be corrected…

KETpXDDzR 4y ago
llama.cpp was easy to setup IMO
- jiggawatts 4y ago
  Can you link to a working Dockerfile? I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!
  - PostOnce 4y ago
    No need to drag a gigabyte of docker stuff into this, just extract the zip file from github and type make into your terminal congratulations, it now works. If you're not a developer, maybe you'll have to type sudo apt install build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
    - vidarh 4y ago
      Did that, got a compiler error within seconds within seconds. Looks like it might need a newer version of gcc than is in the distro on my laptop. Which is why people ask for Docker. If it really will work just with make with build-essential on a new enough distro image, a Dockerfile that documents that would be trivial, and does not at all stop people from just typing make if their setup is new enough.
      - deleted 4y ago
        [deleted]
  - kolinko 4y ago
    but what for?
rch 4y ago
Just use Nixpkgs already.
- microtonal 4y ago
  Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only recourse is using the -bin flavors of PyTorch, etc. which will just download the precompiled upstream versions. Sadly, the result will still be much slower than other distributions. First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine. So, you spend hours fixing derivations so that they build on many core machines and upstream all the diffs. Or YOLO and you disable unit tests altogether. A few hours/days later (depending on your knowledge of Nix), you finally have a built of all packages that you want, you launch whatever you are doing on your CUDA-capable GPU. Turns out that it is 30-50% slower. Finding out why is another multi-day expedition in profiling and tinkering. In the end pyenv (or a Docker container) on a boring distribution doesn't look so bad. (Disclaimer: I initially added the PyTorch/libtorch bin packages to nixpkgs and was co-maintainer of the PyTorch derivation for a while.)
  - UncleEntity 4y ago
    I learned the very hard way not to mess with the python version the system depends on. If you absolutely must then build it separately and link (or use) that exactly like blender does with their binaries. Campbell (one of the core blender devs) used to love to bump the python version as soon as it was released and if you wanted to do any dev work you’d have to run another python environment until the distro version caught up. Being as I liked to use the fedora libs as a sort of sanity check this was a bit of a hassle to say the least.
deleted 4y ago
[deleted]
Taek 4y ago
I have it running locally using the oobabooga webui, setup was moderately annoying but I'm definitely no python expert and I didn't have too much trouble.
int_19h 4y ago
All of these things exist in the Python package ecosystem, and are generally much more common outside of ML/DS stuff. The latter... well, it reminds me of coding in early PHP days. Basically, anything goes so long as it works.
crowwork 4y ago
https://mlc.ai/mlc-llm/ https://mlc.ai/mlc-llm/
- jiggawatts 4y ago
  I love how the response to a complaint about unreproducible builds without any versions being specified is an install script that straight up clones the "current commit" of a Git repo instead of a specific working commit id or tag. Astonishing.
  - Spivak 4y ago
    There's some truth to the "Arch" philosophy for some types of software it's actually more stable to just pull from master.
    - jiggawatts 4y ago
      Not in this case. I’ve tried several repos and they all fail with various version mismatch issues.
valine 4y ago
I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.
- jiggawatts 4y ago
  > the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?
  - UncleEntity 4y ago
    They require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?
    - celdon25 4y ago
      I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"
      - valine 4y ago
        Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?
        jiggawatts 4y ago
kmod 4y ago
I believe the cuda stuff, via Nvidia licensing restrictions, is forced to live outside of these packaging systems (so that you sign a Nvidia eula). Not saying this is a good thing but I think that none of the systems you mentioned would handle this well either
- emikulic 4y ago
  This used to be the case but nowadays you can just pip install e.g. https://libraries.io/pypi/nvidia-cuda-runtime-cu11 https://libraries.io/pypi/nvidia-cuda-runtime-cu11
DustinBrett 4y ago
I had it running before with Dalai (https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai) but have since moved to using the browser based WebGPU method (https://mlc.ai/web-llm/ https://mlc.ai/web-llm/) which uses Vicuna 7B and is quite good.
dagenix 4y ago
There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.
- jiggawatts 4y ago
  Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence. E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68ec9079492a72a35c33d5000da432ce94af71/docker/Dockerfile#L11-L23 https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one, apparently. Game over. Check out "requirements.txt": accelerate==0.18.0 colorama datasets flexgen==0.1.7 gradio==3.25.0 markdown numpy pandas Pillow>=9.5.0 pyyaml requests rwkv==0.7.3 safetensors==0.3.0 sentencepiece tqdm Wow. Less than half of those have any version specified. The rest? "Meh, I don't care, whatever." Then this beauty: git+https://github.com/huggingface/peft I love reaching out to the Internet in the middle of a build pipeline to pull the latest commit of a random repo, because that's so nice and safe, scalable, and cacheable in an artefact repository! The NPM ecosystem gets regularly excoriated for the exact same mistakes, which by now are so well known, so often warned against, so often exploited, so regularly broken that it's getting boring. It's like SQL injection. If you're still doing it in 2023, if your site is still getting hacked because of it, then you absolutely deserve to be labelled immature and even childish.
  - forgingahead 4y ago
    You're right, Python's ecosystem and dependency management is a shitshow and everybody involved should be ashamed of themselves. But of course there are many "you're holding it wrong" commenters here who are in denial of this fact. It's an absolute pity that Python has become the de-facto language for public ML projects.
    - dagenix 4y ago
nl 4y ago
Use the HuggingFace Transformer library. Unlike random github repos they are professionally maintained with proper versioning. Here's the docs: https://huggingface.co/docs/transformers/main/model_doc/llama https://huggingface.co/docs/transformers/main/model_doc/llam...

KETpXDDzR 4y ago

llama.cpp was easy to setup IMO

jiggawatts 4y ago
Can you link to a working Dockerfile? I've heard several people say that it is easy, but then surely it ought to be trivial to set script the build so that it works reliable in a container!
- PostOnce 4y ago
  No need to drag a gigabyte of docker stuff into this, just extract the zip file from github and type make into your terminal congratulations, it now works. If you're not a developer, maybe you'll have to type sudo apt install build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
  - vidarh 4y ago
    Did that, got a compiler error within seconds within seconds. Looks like it might need a newer version of gcc than is in the distro on my laptop. Which is why people ask for Docker. If it really will work just with make with build-essential on a new enough distro image, a Dockerfile that documents that would be trivial, and does not at all stop people from just typing make if their setup is new enough.
    - deleted 4y ago
      [deleted]
- kolinko 4y ago
  but what for?

rch 4y ago

Just use Nixpkgs already.

microtonal 4y ago
Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly. The only recourse is using the -bin flavors of PyTorch, etc. which will just download the precompiled upstream versions. Sadly, the result will still be much slower than other distributions. First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine. So, you spend hours fixing derivations so that they build on many core machines and upstream all the diffs. Or YOLO and you disable unit tests altogether. A few hours/days later (depending on your knowledge of Nix), you finally have a built of all packages that you want, you launch whatever you are doing on your CUDA-capable GPU. Turns out that it is 30-50% slower. Finding out why is another multi-day expedition in profiling and tinkering. In the end pyenv (or a Docker container) on a boring distribution doesn't look so bad. (Disclaimer: I initially added the PyTorch/libtorch bin packages to nixpkgs and was co-maintainer of the PyTorch derivation for a while.)
- UncleEntity 4y ago
  I learned the very hard way not to mess with the python version the system depends on. If you absolutely must then build it separately and link (or use) that exactly like blender does with their binaries. Campbell (one of the core blender devs) used to love to bump the python version as soon as it was released and if you wanted to do any dev work you’d have to run another python environment until the distro version caught up. Being as I liked to use the fedora libs as a sort of sanity check this was a bit of a hassle to say the least.

deleted 4y ago

[deleted]

Taek 4y ago

I have it running locally using the oobabooga webui, setup was moderately annoying but I'm definitely no python expert and I didn't have too much trouble.

int_19h 4y ago

All of these things exist in the Python package ecosystem, and are generally much more common outside of ML/DS stuff. The latter... well, it reminds me of coding in early PHP days. Basically, anything goes so long as it works.

crowwork 4y ago

https://mlc.ai/mlc-llm/ https://mlc.ai/mlc-llm/

jiggawatts 4y ago
I love how the response to a complaint about unreproducible builds without any versions being specified is an install script that straight up clones the "current commit" of a Git repo instead of a specific working commit id or tag. Astonishing.
- Spivak 4y ago
  There's some truth to the "Arch" philosophy for some types of software it's actually more stable to just pull from master.
  - jiggawatts 4y ago
    Not in this case. I’ve tried several repos and they all fail with various version mismatch issues.

valine 4y ago

I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.

jiggawatts 4y ago
> the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?
- UncleEntity 4y ago
  They require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?
  - celdon25 4y ago
    I remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"
    - valine 4y ago
      Most of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?
      - jiggawatts 4y ago

kmod 4y ago

I believe the cuda stuff, via Nvidia licensing restrictions, is forced to live outside of these packaging systems (so that you sign a Nvidia eula). Not saying this is a good thing but I think that none of the systems you mentioned would handle this well either

emikulic 4y ago
This used to be the case but nowadays you can just pip install e.g. https://libraries.io/pypi/nvidia-cuda-runtime-cu11 https://libraries.io/pypi/nvidia-cuda-runtime-cu11

DustinBrett 4y ago

I had it running before with Dalai (https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai) but have since moved to using the browser based WebGPU method (https://mlc.ai/web-llm/ https://mlc.ai/web-llm/) which uses Vicuna 7B and is quite good.

dagenix 4y ago

There is a lot that can be improved with python packaging, but calling it "childish" is itself a pretty immature comment.

jiggawatts 4y ago
Is it? Literally every example I've seen so far is completely unversioned and mere weeks after being written simply doesn't work as a direct consequence. E.g: https://github.com/oobabooga/text-generation-webui/blob/ee68ec9079492a72a35c33d5000da432ce94af71/docker/Dockerfile#L11-L23 https://github.com/oobabooga/text-generation-webui/blob/ee68... Take this line: pip3 install torch torchvision torchaudio Which version of torch is this? The latest. FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04 Which version of CUDA is this? An incompatible one, apparently. Game over. Check out "requirements.txt": accelerate==0.18.0 colorama datasets flexgen==0.1.7 gradio==3.25.0 markdown numpy pandas Pillow>=9.5.0 pyyaml requests rwkv==0.7.3 safetensors==0.3.0 sentencepiece tqdm Wow. Less than half of those have any version specified. The rest? "Meh, I don't care, whatever." Then this beauty: git+https://github.com/huggingface/peft I love reaching out to the Internet in the middle of a build pipeline to pull the latest commit of a random repo, because that's so nice and safe, scalable, and cacheable in an artefact repository! The NPM ecosystem gets regularly excoriated for the exact same mistakes, which by now are so well known, so often warned against, so often exploited, so regularly broken that it's getting boring. It's like SQL injection. If you're still doing it in 2023, if your site is still getting hacked because of it, then you absolutely deserve to be labelled immature and even childish.
- forgingahead 4y ago
  You're right, Python's ecosystem and dependency management is a shitshow and everybody involved should be ashamed of themselves. But of course there are many "you're holding it wrong" commenters here who are in denial of this fact. It's an absolute pity that Python has become the de-facto language for public ML projects.
  - dagenix 4y ago

nl 4y ago

Use the HuggingFace Transformer library. Unlike random github repos they are professionally maintained with proper versioning. Here's the docs: https://huggingface.co/docs/transformers/main/model_doc/llama https://huggingface.co/docs/transformers/main/model_doc/llam...