4 ms·
4-bit quantized 30B-A3B MoE models can run at something like 21 tokens/sec on a several year old AMD CPU.
by yencabulator 4mo ago
4-bit quantized 30B-A3B MoE models can run at something like 21 tokens/sec on a several year old AMD CPU.
loading thread…