3 ms·
Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D https://github.com/geerlingguy/ollama-benchmark?tab=read
by geerlingguy 2y ago
Or attach a 12 or 16 GB GPU to a single Pi 5 directly, and get 20+ tokens/s on an even larger model :D
https://github.com/geerlingguy/ollama-benchmark?tab=readme-ov-file#deepseek https://github.com/geerlingguy/ollama-benchmark?tab=readme-o...
- littlestymaar 2y agoReading the beginning of your comment I was like “ah yes I saw Jeff Geerling do that on a video”. Then I saw you github link and your HN handle and I was like “Wait, it is Jeff Geerling!”. :D
- ziml77 2y agoHaha I had nearly the same thing happen. First I was like "that sounds like something Jeff Geerling would do". Then I saw the github link and was like "ah yeah Jeff Geerling did do it" and then I saw the username and was like "oh it's Jeff Geerling!"
- replete 2y agoThanks for sharing. Pi5 + cheap AMD GPU = convenient modest LLM api server? ...if you find the right magic rocm incantations I guess Double thanks for the 3rd party mac mini SSD tip - eagerly awaiting delivery!
- geerlingguy 2y agollama.cpp runs great with Vulkan, so no ROCm magic required!