3 ms·
The article mentions Q4, Q5, Q8, and NVFP4. It's total AI slop though, tough read. In my testing I got 150 tokens/sec with a single 5090 RTX.
by metadat 2mo ago
The article mentions Q4, Q5, Q8, and NVFP4. It's total AI slop though, tough read.
In my testing I got 150 tokens/sec with a single 5090 RTX.
- Foobar8568 2mo agoWhich model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.
- iv42 2mo agoIf you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).
- Foobar8568 2mo agoLooks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures. https://github.com/Neroued/ninfer/blob/master/docs/performance.md https://github.com/Neroued/ninfer/blob/master/docs/performan... Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract And on my "own" "quick" benchmark, it's slower than vllm.
- iv42 2mo agoI don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_many_tokenssecond_output_are_you_getting_with/ https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...
- pich 2mo ago[dead]
- sleepyeldrazi 2mo agoyeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).
- pich 2mo agoA 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth