4 ms·

Well, let's do some order-of-magnitude calculations: A single 1kW B200 GPU will set you back $50k, and as NVIDIA claims[0] can do 125 tokens per second with LLa

Well, let's do some order-of-magnitude calculations: A single 1kW B200 GPU will set you back $50k, and as NVIDIA claims[0] can do 125 tokens per second with LLama4. Let's imagine you can use it for 36 months, at a DC, cooling and electricity price of 20 cents per kWh. That's $4.3E-6 per token for the card and $4E-7 per token for DC and power, together $4.7E-6 per token. Let's say you are a power user, so your queries and responses are complex and numerous, say 1000 tokens per query+response and 1 query every 10 minutes of an 8h workday. That's 48k tokens per workday, at 20 workdays per month that's 960k tokens per month. So the cost (not sales price!) for those 960k tokens (roughly 1M) a month should be $4.5 Now you can go over the numbers again and think about where they might be wrong: Maybe a typical query is more than 1000 tokens. Maybe power users issue more queries. You might very well multiply by a factor of 10 here. Nvidia getting more greedy for new GPUs? Add 50%. Data center and power cost too conservative, network and storage also important? Add 50%. 3 years of use for a GPU too long, because the field is very quickly adapting ever larger models? Add 50%. Usage factor not 100%, but lower, say a more realistic 50%? Double the cost. Llama4 not good enough, need a more advanced model? May produce a lot less tokens per GPU-hour, but numbers are hard to come by. With that, it's easy to imagine that one might still loose money at $200 per month. To compare, Azure sells OpenAI models in 1M token batches that can easily be compared to the above monthly cost. https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick/ https://developer.nvidia.com/blog/blackwell-breaks-the-1000-... https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/pricing/details/cognitive-...

piva00 1y ago
There's also the training costs, which are a massive capital investment, that need to be spread over all subscribers. Depends on training schedule as well, for each training window the costs of it need to be spread over subscribers during that window. It's good it scales down with a higher number of paying subscriptions (each pays a smaller share of training costs).
supermatt 1y ago
> NVIDIA claims[0] can do 125 tokens per second The claim is per user. With batching, it is MUCH higher (72x)
- thyristan 1y ago
  I'm not so sure there. The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack[0]. The phrasing "and it reaches 72,000 TPS/server at our highest throughput configuration" also hints at something being fishy here. They carefully use the phrases "node" earlier, and "server" later, without getting specific by what they mean a "server" to be. Also, for that 72000 figure, there is no mention of batching at all. The very short article [2] linked in [0] which is supposed to be the independent source of those numbers also doesn't specify any details to that effect. In general, I've learned to treat Nvidia numbers very carefully. They are well-known for misrepresenting apples-to-orange-elefants figures such as comparing FP16, FP8 or FP4 FLOPS, thereby grossly overstating the performance advantages of their new architectures[3]. [0] https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick/ https://developer.nvidia.com/blog/blackwell-breaks-the-1000-... [1] > NVIDIA DGX™ GB200 is purpose-built for training and inferencing trillion-parameter generative AI models. Designed as a rack-scale solution, each liquid-cooled rack features 36 NVIDIA GB200 Grace Blackwell Superchips—–36 NVIDIA Grace CPUs and 72 Blackwell GPUs https://www.nvidia.com/en-eu/data-center/dgx-gb200/ https://www.nvidia.com/en-eu/data-center/dgx-gb200/ [2] https://www.linkedin.com/feed/update/urn:li:activity:7331470006850883584/ https://www.linkedin.com/feed/update/urn:li:activity:7331470... [3] https://dev.to/maximsaplin/nvidias-1000x-performance-boost-claim-verified-j7f https://dev.to/maximsaplin/nvidias-1000x-performance-boost-c...
  - supermatt 1y ago
    > The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack Sure, but in the article it is already mentioned as 1000 TPS/User for an 8 GPU node, and the rack contains 9 nodes - i.e. 9x more GPUs, not 72x - so the 72000 TPS/Server simply being a multiple of 72 seems like a red herring. But yeah, I agree that 72x seems high - although only 9x seems low given vLLM showing over 20x speedups with continuous batching. I guess there are a lot of variables.
jjmarr 1y ago
You're not factoring the input tokens into the equation, which is 90% of the price with a tool like Cline. My queries are like 30,000 tokens input for 50 tokens output
- thyristan 1y ago
  I didn't really distinguish between those token types, since I didn't want to complicate the calculation too much. But yes, you are right, there is a large difference between input and output tokens.
  - jjmarr 1y ago
    Input management is the biggest difference between monthly plans like Copilot and pay-per-use like Cline. Monthly plans hide the input from the user and try to minimize it while pay-per-use plans visibly use the full context window. Mentally modelling the pricing as being determined by output doesn't match reality in my experience. It's also fundamentally different economics since input is gated by VRAM capacity while output is gated by compute.

piva00 1y ago

There's also the training costs, which are a massive capital investment, that need to be spread over all subscribers. Depends on training schedule as well, for each training window the costs of it need to be spread over subscribers during that window. It's good it scales down with a higher number of paying subscriptions (each pays a smaller share of training costs).

supermatt 1y ago

> NVIDIA claims[0] can do 125 tokens per second The claim is per user. With batching, it is MUCH higher (72x)

thyristan 1y ago
I'm not so sure there. The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack[0]. The phrasing "and it reaches 72,000 TPS/server at our highest throughput configuration" also hints at something being fishy here. They carefully use the phrases "node" earlier, and "server" later, without getting specific by what they mean a "server" to be. Also, for that 72000 figure, there is no mention of batching at all. The very short article [2] linked in [0] which is supposed to be the independent source of those numbers also doesn't specify any details to that effect. In general, I've learned to treat Nvidia numbers very carefully. They are well-known for misrepresenting apples-to-orange-elefants figures such as comparing FP16, FP8 or FP4 FLOPS, thereby grossly overstating the performance advantages of their new architectures[3]. [0] https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick/ https://developer.nvidia.com/blog/blackwell-breaks-the-1000-... [1] > NVIDIA DGX™ GB200 is purpose-built for training and inferencing trillion-parameter generative AI models. Designed as a rack-scale solution, each liquid-cooled rack features 36 NVIDIA GB200 Grace Blackwell Superchips—–36 NVIDIA Grace CPUs and 72 Blackwell GPUs https://www.nvidia.com/en-eu/data-center/dgx-gb200/ https://www.nvidia.com/en-eu/data-center/dgx-gb200/ [2] https://www.linkedin.com/feed/update/urn:li:activity:7331470006850883584/ https://www.linkedin.com/feed/update/urn:li:activity:7331470... [3] https://dev.to/maximsaplin/nvidias-1000x-performance-boost-claim-verified-j7f https://dev.to/maximsaplin/nvidias-1000x-performance-boost-c...
- supermatt 1y ago
  > The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack Sure, but in the article it is already mentioned as 1000 TPS/User for an 8 GPU node, and the rack contains 9 nodes - i.e. 9x more GPUs, not 72x - so the 72000 TPS/Server simply being a multiple of 72 seems like a red herring. But yeah, I agree that 72x seems high - although only 9x seems low given vLLM showing over 20x speedups with continuous batching. I guess there are a lot of variables.

jjmarr 1y ago

You're not factoring the input tokens into the equation, which is 90% of the price with a tool like Cline. My queries are like 30,000 tokens input for 50 tokens output

thyristan 1y ago
I didn't really distinguish between those token types, since I didn't want to complicate the calculation too much. But yes, you are right, there is a large difference between input and output tokens.
- jjmarr 1y ago
  Input management is the biggest difference between monthly plans like Copilot and pay-per-use like Cline. Monthly plans hide the input from the user and try to minimize it while pay-per-use plans visibly use the full context window. Mentally modelling the pricing as being determined by output doesn't match reality in my experience. It's also fundamentally different economics since input is gated by VRAM capacity while output is gated by compute.