3 ms·
try ninfer once you get your 5090 https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer
by vardalab 1mo ago
try ninfer once you get your 5090
https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer
- SamInTheShell 1mo agoThanks for sharing, I'm definitely trying it out after I get through my project milestones for the 5090. The README claims 700tok/s for Qwen 3.8 27b, that would be amazing, I'm only expecting an increase from my ~50tok/s on my Radeon to 200tok/s on the 5090.
- SamInTheShell 28d agoJust wanted to share, I had a SaaS AI drop me a script to bench ninfer against llama-cpp and it is impressive. The place that it's doing better at than llama-cpp seems to really be late in the context window. Initial results boiled down as follows. # lmstudio-community/qwen3.8-27b@q4_k_m decode falloff 104.3 tok/s @ 12,683 -> 55.8 tok/s @ 240,755 (53% retained) prefill falloff 3,274 tok/s -> 1,059 tok/s (32% retained) # qwen3_8_27b_nvfp4.ninfer decode falloff 173.3 tok/s @ 11,867 -> 139.3 tok/s @ 225,710 (80% retained) prefill falloff 8,726 tok/s -> 2,816 tok/s (32% retained) I should still have room for more performance on the table. I've not even touched the overclock settings on the GPU. This is a really cool project, I'm going to have to get into what those 3 guys are doing... assuming it can be done with what I got.