4 ms·

I think this is before quantization (fp16). If quantized to 3 or 4 bit precision, you should be able to run it.