4 ms·
So when will I be able to "sudo apt-get install llm" ?
by dvh 2y ago
So when will I be able to "sudo apt-get install llm" ?
- jsanders9 2y agoOllama is close...
- mysterhawk 2y agoYou can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
- mysterhawk 2y agoAnd everything runs way faster on cpu like that
- mysterhawk 2y agoI could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
- mysterhawk 2y agoWith this model: https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram
- mysterhawk 2y agohttps://justine.lol/matmul/ https://justine.lol/matmul/ https://www.youtube.com/watch?v=-mRi-B3t6fA https://www.youtube.com/watch?v=-mRi-B3t6fA Checkout these articles
- yjftsjthsd-h 2y agoI'm not aware of any Debian family distro that packages it, but NixOS has at least ollama and llama-cpp in its repos. Honestly even if the more stable distributions did have these things packaged, I would hesitate to use the packaged versions because all of this stuff is still so quickly moving that you'd be on an old version and it would hurt. Edit: Arch has ollama in official repos too. OpenSUSE has https://software.opensuse.org/package/ollama https://software.opensuse.org/package/ollama .
- paxys 2y agoYou already can with ollama
- o11c 2y agoRealistically, you probably want to wait until Vulkan support trickles out. That way, you aren't at the whim of the various evil hardware drivers (everybody's suck), and the AI can give you a disappointingly confused answer much faster than running the LLM on a CPU can.