3 ms·
Horrific comparison point. LLM inference is way more expensive locally for single users than running batch inference at scale in a datacenter on actual GPUs/TPU
by msp26 8mo ago
Horrific comparison point. LLM inference is way more expensive locally for single users than running batch inference at scale in a datacenter on actual GPUs/TPUs.
- AlexandrB 8mo agoHow is that horrific? It sets an upper bound on the cost, which turns out to be not very high.