ds4: prefill: 150.18 t/s, generation: 48.47 t/s together with https://github.com/antirez/ds4 https://github.com/antirez/ds4 + RTX Pro 6000, local provider :) I

ds4: prefill: 150.18 t/s, generation: 48.47 t/s together with https://github.com/antirez/ds4 https://github.com/antirez/ds4 + RTX Pro 6000, local provider :) I guess you could rent one too, get similar speed. https://gist.github.com/embedding-shapes/eceefa1b2699dca448f9af22e995cdeb https://gist.github.com/embedding-shapes/eceefa1b2699dca448f... The weights I've tried is the "DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix" ones which fits just about within 96GB VRAM. With some tuning, I've managed to get it to get up to ~60 t/s, I'm sure there is other things to do there too :)