6 ms·

How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?

by jstummbillig 11mo ago

ericskiff 11mo ago
I believe this was a statement on cost per token to us as consumers of the service
- moffkalast 11mo ago
  Training cost-effectiveness doesn't matter for open models since someone else ate the cost. In this case, Chinese taxpayers.
  - KvanteKat 11mo ago
    Deepseek is a private corporation funded by a hedge fund (High-Flyer). I doubt much public money was spent by the Chinese state on this. Like with LLMs in the US, the people paying for it so far are mainly investors who are betting on a return in the long to medium term.
    - boringg 11mo ago
      Do you actually believe what you just wrote or are you trolling? One version at least has a foot planted in reality. The other one well...
rowanG077 11mo ago
Well consumers care about the cost to them, and those we know. And deepseek is destroying everything in that department.
- eru 11mo ago
  Yes. Though we don't know for sure whether that's because they actually have lower costs, or whether it's just the Chinese taxpayer being forced to serve us a treat.
  - chronogram 11mo ago
    Third party providers are still cheap though. The closed models are the ones where you can't see the real cost to running them.
    - eru 11mo ago
      Oh, I was mostly talking about the Chinese taxpayer footing the training bill. You are right that we can directly observe the cost of inference for open models.
      - rescbr 11mo ago
        Not sure the Chinese taxpayer is footing the bill though - of course, it might not be net zero, there might be secondary effects, etc. A few days ago I read an article saying the Chinese utilities have a pricing structure that favors high-tech industries (say, an AI data center), making the difference by charging more the energy-intensive but less sophisticated industries (an aluminium smelter, for example). Admittedly, there are some advantages when you do central and long-term economic planning.
tedivm 11mo ago
If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing where you are able to, which will save you money on both methods) and you have a basic idea of how much it would cost per token. Then you compare that to the cost of something like GPT5, which is a bit simpler because the cost per (million) token is something you can grab off of a website. You'd be surprised how much money running something like DeepSeek (or if you prefer a more established company, Qwen3) will save you over the cloud systems. That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on.
- qeternity 11mo ago
  > DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on. Uh, Deepseek will not (unless you are referring to one of their older R1 finetuned variants). But any flagship Deepseek model will require 16x A100/H100+ with NVL in FP8.
- Muromec 11mo ago
  >That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on. What's cheap nowdays? I'm out of the loop. Does anything ever run on integrated AMD that is Ryzen AI that comes in framework motherboards? Is under 1k americans cheap?
  - GTP 11mo ago
    Not really in the loop either, but when Deepseek R1 was released, I sumbled upon this YouTube channel [1] that made local AI PC builds in the 1000-2000$ range. But he doesn't always use GPUs, maybe the cheaper builds were CPU plus a lot of RAM, I don't remember. [1] https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx
deaux 11mo ago
We can judge on inference cost because we do know what those are for open-weights models as there are a dozen independent providers that host these models and price them according to respective inference cost. We can't judge on training cost, that's true.
stingraycharles 11mo ago
You can use tokens/sec on something like AWS Bedrock (which hosts both open and closed models) as a proxy for “costs per token” for the closed providers.
mzl 11mo ago
Apart from measuring prices from venture-backed providers which might or might not correlate with cost-effectiveness, I think the measures of intelligence per watt and intelligence per joule from https://arxiv.org/abs/2511.07885 https://arxiv.org/abs/2511.07885 is very interesting.
badmonster 11mo ago
Good point. Could usage patterns + inference costs give us proxy metrics? What would be a fair baseline?

How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?

ericskiff 11mo ago

I believe this was a statement on cost per token to us as consumers of the service

moffkalast 11mo ago
Training cost-effectiveness doesn't matter for open models since someone else ate the cost. In this case, Chinese taxpayers.
- KvanteKat 11mo ago
  Deepseek is a private corporation funded by a hedge fund (High-Flyer). I doubt much public money was spent by the Chinese state on this. Like with LLMs in the US, the people paying for it so far are mainly investors who are betting on a return in the long to medium term.
  - boringg 11mo ago
    Do you actually believe what you just wrote or are you trolling? One version at least has a foot planted in reality. The other one well...

rowanG077 11mo ago

Well consumers care about the cost to them, and those we know. And deepseek is destroying everything in that department.

eru 11mo ago
Yes. Though we don't know for sure whether that's because they actually have lower costs, or whether it's just the Chinese taxpayer being forced to serve us a treat.
- chronogram 11mo ago
  Third party providers are still cheap though. The closed models are the ones where you can't see the real cost to running them.
  - eru 11mo ago
    Oh, I was mostly talking about the Chinese taxpayer footing the training bill. You are right that we can directly observe the cost of inference for open models.
    - rescbr 11mo ago
      Not sure the Chinese taxpayer is footing the bill though - of course, it might not be net zero, there might be secondary effects, etc. A few days ago I read an article saying the Chinese utilities have a pricing structure that favors high-tech industries (say, an AI data center), making the difference by charging more the energy-intensive but less sophisticated industries (an aluminium smelter, for example). Admittedly, there are some advantages when you do central and long-term economic planning.

tedivm 11mo ago

If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing where you are able to, which will save you money on both methods) and you have a basic idea of how much it would cost per token. Then you compare that to the cost of something like GPT5, which is a bit simpler because the cost per (million) token is something you can grab off of a website. You'd be surprised how much money running something like DeepSeek (or if you prefer a more established company, Qwen3) will save you over the cloud systems. That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on.

qeternity 11mo ago
> DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on. Uh, Deepseek will not (unless you are referring to one of their older R1 finetuned variants). But any flagship Deepseek model will require 16x A100/H100+ with NVL in FP8.
Muromec 11mo ago
>That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on. What's cheap nowdays? I'm out of the loop. Does anything ever run on integrated AMD that is Ryzen AI that comes in framework motherboards? Is under 1k americans cheap?
- GTP 11mo ago
  Not really in the loop either, but when Deepseek R1 was released, I sumbled upon this YouTube channel [1] that made local AI PC builds in the 1000-2000$ range. But he doesn't always use GPUs, maybe the cheaper builds were CPU plus a lot of RAM, I don't remember. [1] https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx

deaux 11mo ago

We can judge on inference cost because we do know what those are for open-weights models as there are a dozen independent providers that host these models and price them according to respective inference cost. We can't judge on training cost, that's true.

stingraycharles 11mo ago

You can use tokens/sec on something like AWS Bedrock (which hosts both open and closed models) as a proxy for “costs per token” for the closed providers.

mzl 11mo ago

Apart from measuring prices from venture-backed providers which might or might not correlate with cost-effectiveness, I think the measures of intelligence per watt and intelligence per joule from https://arxiv.org/abs/2511.07885 https://arxiv.org/abs/2511.07885 is very interesting.

badmonster 11mo ago

Good point. Could usage patterns + inference costs give us proxy metrics? What would be a fair baseline?