3 ms·
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
by mysterhawk 2y ago
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
- mysterhawk 2y agoAnd everything runs way faster on cpu like that
- mysterhawk 2y agoI could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
- mysterhawk 2y agoWith this model: https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram
- mysterhawk 2y agohttps://justine.lol/matmul/ https://justine.lol/matmul/ https://www.youtube.com/watch?v=-mRi-B3t6fA https://www.youtube.com/watch?v=-mRi-B3t6fA Checkout these articles