4 ms·
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of
by anon291 1mo ago
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
- kennywinker 1mo agoFair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.