3 ms·
"and $1 cost per GPU hour." "We estimate the cost per query to be 0.36 cents." "<2000ms latency" "50% hardware utilization rates" Assuming all queries take
by GreedClarifies 4y ago
"and $1 cost per GPU hour."
"We estimate the cost per query to be 0.36 cents."
"<2000ms latency"
"50% hardware utilization rates"
Assuming all queries take 2s this implies that ( 1 * 3600 /
( 2 / 0.5 ) ) * 0.36 = 324 GPUs would be used for each query. This seems implausibly high.
- dylan522p 4y agoYour math is completely wrong dude. It's 2000ms per token not for the whole query. Hardware utilization rates and MFU are not the same thing, you forgot the latter. You are pretending its perfectly parelelized on 1 GPU too. I use 8x GPU box throughput.
- hervature 4y ago0.36 cents or $0.0036. Your math should be 3.24 GPU/query.
- GreedClarifies 4y agoYou’re 100% right, my bad! And that puts the estimate entirely in the realm of the possible. Whoops!