4 ms·

I find it unlikely that the margins on inference hardware will remain anywhere near as high as they are right now. Inference at scale can be complex, but the c

by amluto 1y ago

I find it unlikely that the margins on inference hardware will remain anywhere near as high as they are right now. Inference at scale can be complex, but the complexity is manageable. You can do fancy batched inference, or you can make a single pass over the relevant weights for each inference step. With more models using MoE, the latter is more tractable, and the actual tensor/FMA units that do the bulk of the math are simple enough that any respectable silicon vendor can make them.

janice1999 1y ago
Is there a viable non-Nvidia vendor for inference at scale? AMD? Or is in-house hardware like Google and Amazon?
- kridsdale1 1y ago
  Yes to all of the above.
  - amluto 1y ago
    And it will likely become even more true. There’s no way that a handful of highly-motivated companies will spend hundreds of billions annually on very high margin Nvidia hardware without investing at least a few percent of that on developing cheaper alternatives.
  - dingnuts 1y ago
    Interesting! Care to share literally any details about their capex and build out so we can understand the amount of compute that's being made available or is the burden of evidence on people who are trying to remain grounded?
    - amluto 1y ago
      Google reported an estimated 2025 AI CapEx of around $85 billion. I don’t know how much is inference vs training (or shared), and Google is quite proud of using a whole bunch of their own chips. Much of the data on how much money is spent where is public. In any event, one can make some generalizations about the companies involved. Nvidia makes excellent hardware that everyone wants and charges large enough markups that their margins are around 90%. AMD is chasing the big buyers to sell their products. Google spends a lot and is a mature company, and they seem uninterested in selling chips that compete with Nvidia, but they certainly care about revenue and profit. OpenAI, Anthropic, etc and, perhaps oddly, Meta don’t seem to care too much about profit, but they certainly spend enough money that it would help them to get more bang for their buck. Alibaba, etc buy whatever Nvidia gear they can get, but they have a lot of incentive to find a domestic supplier, and Huawei seems quite interested in becoming that supplier. And there are plenty of US startups (Cerebras and others) going after the inference market.

by amluto 1y ago

janice1999 1y ago

Is there a viable non-Nvidia vendor for inference at scale? AMD? Or is in-house hardware like Google and Amazon?

kridsdale1 1y ago
Yes to all of the above.
- amluto 1y ago
  And it will likely become even more true. There’s no way that a handful of highly-motivated companies will spend hundreds of billions annually on very high margin Nvidia hardware without investing at least a few percent of that on developing cheaper alternatives.
- dingnuts 1y ago
  Interesting! Care to share literally any details about their capex and build out so we can understand the amount of compute that's being made available or is the burden of evidence on people who are trying to remain grounded?
  - amluto 1y ago
    Google reported an estimated 2025 AI CapEx of around $85 billion. I don’t know how much is inference vs training (or shared), and Google is quite proud of using a whole bunch of their own chips. Much of the data on how much money is spent where is public. In any event, one can make some generalizations about the companies involved. Nvidia makes excellent hardware that everyone wants and charges large enough markups that their margins are around 90%. AMD is chasing the big buyers to sell their products. Google spends a lot and is a mature company, and they seem uninterested in selling chips that compete with Nvidia, but they certainly care about revenue and profit. OpenAI, Anthropic, etc and, perhaps oddly, Meta don’t seem to care too much about profit, but they certainly spend enough money that it would help them to get more bang for their buck. Alibaba, etc buy whatever Nvidia gear they can get, but they have a lot of incentive to find a domestic supplier, and Huawei seems quite interested in becoming that supplier. And there are plenty of US startups (Cerebras and others) going after the inference market.