3 ms·
Performance for text generation is memory-limited, so lack of native fp8 support does not matter. You have more than enough compute left over to do the math in
by threeducks 2y ago
Performance for text generation is memory-limited, so lack of native fp8 support does not matter. You have more than enough compute left over to do the math in whichever floating point format you fancy.