4 ms·
The hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inf
by Tuna-Fish 3y ago
The hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inference library. So for now at least the market disagrees with you on the relative merits of 8 vs 16 bit processing.
In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.