3 ms·
Because it's impossible to get started as someone who has no background in ML. I tried earlier this year and also failed. 1. Obtain the Llama models. Apparentl
by lopkeny12ko 3y ago
Because it's impossible to get started as someone who has no background in ML. I tried earlier this year and also failed.
1. Obtain the Llama models. Apparently you have to sign up for access and get a download link? I don't want to do that, found some public download links instead. Ok, now I have a few hundred GBs of model files.
2. Compile llama.cpp. Missing dependencies, took an hour to figure out how to resolve.
3. Quantize the model? What does that even mean?
4. Install pytorch. Run the command in the README. Python exception.
5. Install NVIDIA helper libraries. Doesn't work. Try installing the AMD helpers to run on CPU instead. No instructions for how to do this. Eventually figured it out.
6. Try running pytorch again. Same exception.
I gave up after a full day of trying to make this work. The Docker image is for people like me.
- yieldcrv 3y agogot it thanks for the explanation! helps the command line actually work
- imp0cat 3y agoYeah it's great, you can get this running in no time (even without gpu, just to play with it): docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama docker exec -it ollama ollama run llama2 And that's all!
- stefandesu 3y agoIt's nice to have this possibility on Linux now. It wasn't much more complicated on macOS before: brew install ollama brew services start ollama ollama run llama2 Works great on my M1 MacBook Air, although it's definitely not as good as ChatGPT.
- politelemon 3y agoAnd with GPU just do this for the first command docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama You'll need to have the NVidia container toolkit installed.
- alexchantavy 3y agoThank you! For docker-compose, do deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] Docs: https://docs.docker.com/compose/gpu-support/ https://docs.docker.com/compose/gpu-support/
- brandonasuncion 3y agoBtw, the quantized models are on Huggingface, so step 1 and steps 3-6 can be avoided. (The link is 3/4ths into the llama.cpp README)
- pdntspa 3y agoOh no, gotta RTFM Why is everyone so afraid to get their hands dirty?
- averageRoyalty 3y agoThis isn't really fair. There are many manuals with rapidly changing or incomplete information, or information only months old that is no longer accurate. This is a fast moving space and can definitely be confusing.
- pdntspa 3y agoRegardless, people can do their research. They just don't. Yes this is a fast moving space, personally it's exhausting to keep up on. But rather than pushing people to improve themselves or LEARN we cater to spoiled lazy child syndrome. Put in the work, reap the rewards. No shortcuts!
- averageRoyalty 3y agoI agree in principle, however you appear to be arguing in bad faith. The person you responded outlined their process (to a level you can look at and see they clearly made an effort) and stated they put a full day in. This is not "spoiled lazy child syndrome", although there are for sure those out there who fit that mold.