4 ms·All of this is for batch size 1.by p12tic 1y agoAll of this is for batch size 1.qeternity 1y agoI know. That was my point. Throughput doesn't scale on CPU as well as it does on GPU.p12tic 1y agoWe both agree. Batch size 1 is only relevant to people who want to run models on their own private machines. Which is the case of OP.