2 ms·
I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
by mysterhawk 2y ago
I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
- mysterhawk 2y agoWith this model: https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram