I'm not sure how you manage to be so wrong about something this simple. If you do a single inference at a time, you do GEMV, which spends most of the time load

I'm not sure how you manage to be so wrong about something this simple. If you do a single inference at a time, you do GEMV, which spends most of the time loading parameters and then performs one multiplication and one add per parameter. If you do batching, then you get to do GEMM, which means you load the parameter once and perform multiple calculations per parameter. This is faster even for a purely sequential matrix multiplication implementation. CPUs tend to have both SIMD and multiple cores these days. This means that your computational resources exceed the available memory bandwidth by far. What you suggested in the second "paragraph" is just letting someone else do the batching but with a lower precision data type. You're starting to contradict your first point.