4 ms·
I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.
by asabla 1y ago
I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.
- modeless 1y agoCool, what software?
- asabla 1y agoInitial testing has only been done with ollama. Plan on testing out llama.cpp and vllm when there is enough time
- steinvakt2 1y agoAnd flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?
- PeterStuer 1y agoI don't think the 4090 has native 4bit support, which will probably have a significant impact.
- diggan 1y ago> And flash attention doesn't work on 5090 yet, right? Flash attention works with GPT-OSS + llama.cpp (tested on 1d72c8418) and other Blackwell card (RTX Pro 6000) so I think it should work on 5090 as well, it's the same architecture after all.