4 ms·
GPU acceleration is pretty easy with llama.cpp. You just run make with an extra flag and then an argument or two at runtime.
by bugglebeetle 3y ago
GPU acceleration is pretty easy with llama.cpp. You just run make with an extra flag and then an argument or two at runtime.
- simonw 3y agoTherein lies the problem. I want to write instructions for people to use my software that don't expect them to know how to run make, or even to have a C compiler installed.
- bugglebeetle 3y agoThen you more or less need a GUI like OpenAI built with ChatGPT so you control the whole environment. Even setting up LLM in Homebrew required me to do the whole install twice because of some arcane error.
- simonw 3y agoI think I can fix that by shipping a bottle release - "brew install simonw/llm/llm" currently attempts to compile Pydantic's Rust extensions, which means it runs incredibly slowly. I built a bottle for it which installs much faster, but you currently have to download the file from https://static.simonwillison.net/static/2023/llm--0.5.arm64_ventura.bottle.1.tar.gz https://static.simonwillison.net/static/2023/llm--0.5.arm64_... - I've not yet figured out how to get that to install when the user runs "brew install simonw/llm/llm" - issue here: https://github.com/simonw/llm/issues/102 https://github.com/simonw/llm/issues/102
- simonw 3y agoAdrien Brault on Twitter gave me this recipe, which worked perfectly: https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2167b5d129 https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2...