4 ms·

Do these model releases really matter to cost if the hardware is still so very expensive and Nvidia still has a defacto monopoly? I can't buy x8 H100s to run a

by janice1999 1y ago

Do these model releases really matter to cost if the hardware is still so very expensive and Nvidia still has a defacto monopoly? I can't buy x8 H100s to run a model and whatever company I buy AI access from has to pay for them somehow.

fzzzy 1y ago
You only need 64 gb of cpu ram to run gpt-oss, or one h100.
- claytonjy 1y ago
  you can’t really buy H100s except in multiples of 8. If you want fewer, you must rent. Even then, hyperscalers tend to be a bit inflexible there; GCP only recently added support for smaller shapes, and they can’t yet be reserved, only on-demand or spot iirc.
- janice1999 1y ago
  I assume you're talking that's a quantised 20B model on a several thousand dollar Mac? That's really impressive and huge progress but is that indicative of companies serving thousands of users? They still have to buy Nvidia at the end of the day.
amluto 1y ago
I find it unlikely that the margins on inference hardware will remain anywhere near as high as they are right now. Inference at scale can be complex, but the complexity is manageable. You can do fancy batched inference, or you can make a single pass over the relevant weights for each inference step. With more models using MoE, the latter is more tractable, and the actual tensor/FMA units that do the bulk of the math are simple enough that any respectable silicon vendor can make them.
- janice1999 1y ago
  Is there a viable non-Nvidia vendor for inference at scale? AMD? Or is in-house hardware like Google and Amazon?
  - kridsdale1 1y ago
    Yes to all of the above.
    - amluto 1y ago
      And it will likely become even more true. There’s no way that a handful of highly-motivated companies will spend hundreds of billions annually on very high margin Nvidia hardware without investing at least a few percent of that on developing cheaper alternatives.
    - dingnuts 1y ago
      Interesting! Care to share literally any details about their capex and build out so we can understand the amount of compute that's being made available or is the burden of evidence on people who are trying to remain grounded?
      - amluto 1y ago
        Google reported an estimated 2025 AI CapEx of around $85 billion. I don’t know how much is inference vs training (or shared), and Google is quite proud of using a whole bunch of their own chips. Much of the data on how much money is spent where is public. In any event, one can make some generalizations about the companies involved. Nvidia makes excellent hardware that everyone wants and charges large enough markups that their margins are around 90%. AMD is chasing the big buyers to sell their products. Google spends a lot and is a mature company, and they seem uninterested in selling chips that compete with Nvidia, but they certainly care about revenue and profit. OpenAI, Anthropic, etc and, perhaps oddly, Meta don’t seem to care too much about profit, but they certainly spend enough money that it would help them to get more bang for their buck. Alibaba, etc buy whatever Nvidia gear they can get, but they have a lot of incentive to find a domestic supplier, and Huawei seems quite interested in becoming that supplier. And there are plenty of US startups (Cerebras and others) going after the inference market.
willy_k 1y ago
Yes they do, if the model size / vram requirement keeps shrinking for a given performance target, like has been happening, then it gets cheaper to run X level of model.
siva7 1y ago
The news is that this won't be necessarily for the majority of private and workforce. They run on your own machine.
skybrian 1y ago
Assuming we continue to see real competition at running open source models and there isn’t a supply bottleneck, it will make it hard to sell access at much more than cost. So, prices might go up compared to companies selling service at a loss, but there’s a limit. Maybe someone knows which providers are selling access roughly at cost and what their prices are?

fzzzy 1y ago

You only need 64 gb of cpu ram to run gpt-oss, or one h100.

claytonjy 1y ago
you can’t really buy H100s except in multiples of 8. If you want fewer, you must rent. Even then, hyperscalers tend to be a bit inflexible there; GCP only recently added support for smaller shapes, and they can’t yet be reserved, only on-demand or spot iirc.
janice1999 1y ago
I assume you're talking that's a quantised 20B model on a several thousand dollar Mac? That's really impressive and huge progress but is that indicative of companies serving thousands of users? They still have to buy Nvidia at the end of the day.

amluto 1y ago

I find it unlikely that the margins on inference hardware will remain anywhere near as high as they are right now. Inference at scale can be complex, but the complexity is manageable. You can do fancy batched inference, or you can make a single pass over the relevant weights for each inference step. With more models using MoE, the latter is more tractable, and the actual tensor/FMA units that do the bulk of the math are simple enough that any respectable silicon vendor can make them.

janice1999 1y ago
Is there a viable non-Nvidia vendor for inference at scale? AMD? Or is in-house hardware like Google and Amazon?
- kridsdale1 1y ago
  Yes to all of the above.
  - amluto 1y ago
    And it will likely become even more true. There’s no way that a handful of highly-motivated companies will spend hundreds of billions annually on very high margin Nvidia hardware without investing at least a few percent of that on developing cheaper alternatives.
  - dingnuts 1y ago
    Interesting! Care to share literally any details about their capex and build out so we can understand the amount of compute that's being made available or is the burden of evidence on people who are trying to remain grounded?
    - amluto 1y ago
      Google reported an estimated 2025 AI CapEx of around $85 billion. I don’t know how much is inference vs training (or shared), and Google is quite proud of using a whole bunch of their own chips. Much of the data on how much money is spent where is public. In any event, one can make some generalizations about the companies involved. Nvidia makes excellent hardware that everyone wants and charges large enough markups that their margins are around 90%. AMD is chasing the big buyers to sell their products. Google spends a lot and is a mature company, and they seem uninterested in selling chips that compete with Nvidia, but they certainly care about revenue and profit. OpenAI, Anthropic, etc and, perhaps oddly, Meta don’t seem to care too much about profit, but they certainly spend enough money that it would help them to get more bang for their buck. Alibaba, etc buy whatever Nvidia gear they can get, but they have a lot of incentive to find a domestic supplier, and Huawei seems quite interested in becoming that supplier. And there are plenty of US startups (Cerebras and others) going after the inference market.

willy_k 1y ago

Yes they do, if the model size / vram requirement keeps shrinking for a given performance target, like has been happening, then it gets cheaper to run X level of model.

siva7 1y ago

The news is that this won't be necessarily for the majority of private and workforce. They run on your own machine.

skybrian 1y ago

Assuming we continue to see real competition at running open source models and there isn’t a supply bottleneck, it will make it hard to sell access at much more than cost. So, prices might go up compared to companies selling service at a loss, but there’s a limit. Maybe someone knows which providers are selling access roughly at cost and what their prices are?