CPU inference is “OK” on quantized models smaller than 10B Gives at least a couple tokens per sec on fast modern CPUs. So it really depends on how fast the sp

CPU inference is “OK” on quantized models smaller than 10B Gives at least a couple tokens per sec on fast modern CPUs. So it really depends on how fast the speed needs to be and the expected loads, compute cost and budget etc etc