4 ms·
because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications
by huac 3y ago
because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.
- mark_l_watson 3y agoOr, inference that can fluidly use available CPUs, GPUs, and neural units on M1/M2/M3.
- wmf 3y agoNot if a 96 GB GPU costs $30K?
- huac 3y agoThe OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference. (to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)
- qeternity 3y ago> Equivalent inference on a 4090 ($1600) is up to 400 tok/s What? Who's doing 400 t/s @ 7b on a 4090?
- huac 3y agoI am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;). batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.