5 ms·
I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant. None of th
by valine 3y ago
I’ve got 4 different llama models running locally with CUDA and can freely switch between them, including LLaVA which is a multimodal LLaMA variant.
None of them are particularly difficult to get running, the trick is to search the project’s github issue tracker. 99% of the time your problem will be in there with steps to fix it.
- jiggawatts 3y ago> the trick is to search the project’s github issue tracker. What ever happened to the crazy notion of Dockerfiles that simply build successfully? Isn’t half the point of containerisation that it papers over the madness of the Python module ecosystem?
- UncleEntity 3y agoThey require someone to take time out of their busy schedule to build? Complaining that people won’t work for you for free is a bit much, don’t you think?
- celdon25 3y agoI remember the days when lowering the barrier to entry was considered to be a safe investment since it would pay for itself through increased project interest (and more contributions by the community). Now it is apparently seen as "working for free for ungrateful people"
- valine 3y agoMost of the repos for these new LLaMA derivatives are brand new, and put together by people who specialize in ML research not software engineering. I’m just happy to have access to their code. I’m sure most of these teams would be happy to merge a PR that lowered the barrier to entry, why don’t you get on that?
- jiggawatts 3y ago> specialize in ML research not software engineering. That has nothing to do with Python tooling being bad. A safe assumption is that Python package managers are being developed by developers, who have no excuse. If a C++ codebase developed by scientists had null pointer exceptions in it, then I could excuse things. But if the C++ compiler itself introduced unforced null pointer errors, then it absolutely deserves criticism. It shouldn't be possible for a ML researcher to use Conda or whatever package manager in a way that despite using a formally specified "requirements.txt", it won't build a week later because of how loose the specification of module versions is allowed to be. The Python attitude and more specifically Conda is at fault here, not the ML researching trying to get his job done.
- qaq 3y agoPoetry is fairly reasonable package manager
- alchemist1e9 3y agoOne point you have I absolutely agree with: > Conda is at fault here Conda is so so bad. But trying to explain why to people who have fallen into it’s trap is difficult. People don’t realize the packages are not signed on enough information to reproduce them. The optimizer to find matching versions to make an environment that satisfies your constraints is really bad idea. As an experienced C++ developer unfortunately and fortunately I’ve concluded the most “correct” solution is to use nixpkgs.
- dventimi 3y agoWhat's a good alternative? No, really. Conda has caused me so much pain.
- alchemist1e9 3y agoAs I mentioned I’ve settled on Nix with nixpkgs however it’s got a steep learning curve and really isn’t appropriate for anyone but fairly experienced unix hackers. It’s a problem.
- UncleEntity 3y agoThere’s a big difference between making a project easy to use and requiring they package it all up in someone’s preferred container format which can’t even legally include all the dependencies. Facebook seems to be pretty hands off (as is expected since the code is open source) unless you distribute the model weights and then they drop the dmca banhammer. So, yeah, simply complaining with no effort to understand the problem is kind of ungrateful.
- avereveard 3y agoThe problem with that is gpu access from the docker image and iirc Nvidia doesn't have a Windows passtrough cuda driver
- MakeUsersWant 3y agoCould you publish a set of known-good versions (pip freeze, OS version, etc)?