3 ms·
Remove "and Python 3.11" from title. Python used only for converting model to llama.cpp project format, 3.10 or whatever is fine. Additionally, llama.cpp works
by diimdeep 4y ago
Remove "and Python 3.11" from title. Python used only for converting model to llama.cpp project format, 3.10 or whatever is fine.
Additionally, llama.cpp works fine with 10 y.o hardware that supports AVX2.
I'm running llama.cpp right now on an ancient Intel i5 2013 MacBook with only 2 cores and 8 GB RAM - 7B 4bit model loads in 8 seconds to 4.2 GB RAM and gives 600 ms per token.
btw: anyone knows how to disable swap per process in macOS ? even though there is enough free RAM, sometimes macOS decides to use swap instead.
- metadat 4y agoCan you provide a link to what guide or steps you followed to get this up and running? I have a physical Linux machine with 300+ GB of RAM, would love to try out llama on it but I'm not sure where to get started for how to get it working with such a configuration. Edit: Thank you, @diimdeep!
- deleted 4y ago[deleted]
- diimdeep 4y agoSure. You can get models with magnet link from here https://github.com/shawwn/llama-dl/ https://github.com/shawwn/llama-dl/ To get running, just follow these steps https://github.com/ggerganov/llama.cpp/#usage https://github.com/ggerganov/llama.cpp/#usage
- HervalFreire 4y agoIs it legal to post that here?
- CharlesW 4y ago> Remove "and Python 3.11" from title. Python used only for converting model to llama.cpp project format, 3.10 or whatever is fine. As @rnosov notes elsewhere in the thread, this post has a workaround for the PyTorch issue with Python 3.11, which is why the "and Python 3.11" qualification is there.
- ahoho 4y agoDo you know if there's there a good reason to favor 3.11 over 3.10 for this use case?
- CharlesW 4y agoI'm a Python neophyte, but I've read that Python 3.11 is 10-60% faster than 3.10, so that may be a consideration.
- simonw 4y agoIn this particular case that doesn't matter, because the only time you run Python is for a one-off conversion against the model files. That takes at most a minute to run, but once converted you'll never need to run it again. Actual llama.cpp model inference uses compiled C++ code with no Python involved at all.
- e12e 4y agohttps://endoflife.date/python https://endoflife.date/python ?
- codetrotter 4y agoThe real question is. Which python3 version does current macOS ship with? Well, on my macOS Ventura 13.2.1 install, /usr/bin/python3 is Python 3.9.6, which may be too old? But also, my custom installed python3 via homebrew is not 3.11 either. My /opt/homebrew/bin/python3 is Python 3.10.9 MacBook Pro M1
- datadeft 4y agobrew install python@3.11
- zitterbewegung 4y agoI am able to get 65B on a MacBook Pro 14.2 with 64gb of ram. https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...