7 ms·

Companies are getting desperate to show AI adoption as right now the numbers just don’t add up. All compute companies say they don't have enough compute to mee

Companies are getting desperate to show AI adoption as right now the numbers just don’t add up. All compute companies say they don't have enough compute to meet demands. Why do you think there isn't enough AI adoption to justify the investment?

cmiles8 8mo ago
“Demand” is mostly their training of models, which they’ve yet to demonstrate is a profitable business. Just because you’re struggling to get raw materials for your business doesn’t make it a good business. Without strong enterprise adoption ASAP (which is what’s seriously suffering) things are going to hit the fan real quick.
- lancebeet 8mo ago
  This will sound snarky, so forgive me, but I honestly don't know the answer. Is this actually true? Is there a reliable source containing statistics on LLM compute usage that includes training vs inference for the whole market?
  - concats 8mo ago
    The revenue numbers are public for the major AI companies. That's probably the best estimate for "inference for the whole market" we have, since most of that inference is billed in either API usage or subscriptions, and it won't include any in-house usage such as training.
  - seanmcdirmid 8mo ago
    I don’t understand why people don’t just use Gemini or some other AI web search to get an answer to these kinds of questions quickly (I excluded the sources, you can get them if you ask the same question). > While AI training is often the most intense and expensive process for a single model, the majority of total AI compute usage (approximately 90%) is used for inference. > Here is the breakdown of why this is the case: > Inference as High-Volume > Activity: Inference occurs every time a user interacts with an AI model (e.g., asking ChatGPT a question, using image recognition, or generating code). While a model is trained once (or updated infrequently), it runs millions or billions of inferences continuously. > Cost Scaling: Training is a massive, one-time upfront cost, while inference is an ongoing, daily operational cost. As the number of AI users grows, the demand for inference compute scales faster than the need for training new, large models. > The Shift to Efficiency: While early AI hype focused on the immense compute needed for training, the industry has shifted toward making inference cheaper and faster through specialized hardware and techniques like optimization, quantization, and small language models (SLMs).
    - ajam1507 8mo ago
      Gemini is not a reliable source. You posted the only part of the AI response that isn't useful in verifying whether it is true.
      - seanmcdirmid 8mo ago
        Sure, I guess. I asked Gemini to give me some markdown of citations and the claims made that address the question: https://share.google/aimode/v3Y9P3rYIx1oj9VI2 https://share.google/aimode/v3Y9P3rYIx1oj9VI2 And I finally figured out how to get links to answers instead of just inlining the content as before. Anyways, there it is. We live in a time where questions like "Does inference or training use more compute?" can be answered quickly by just pasting it into a search box.
- aurareturn 8mo ago
  Do you have source?
  - tempodox 8mo ago
    Just ask the slop generator of your choice. Making sense of all that smoke and mirrors without AI support has become impossible.
- couchdb_ouchdb 8mo ago
  With respect, I don't think you've used the latest models and have not seen Anthropic's enterprise revenue hockey-stick like number. They are so busy outfitting fortune-500, you can't even get someone in sales to respond to emails. I've been waiting for months and so have others.
- seanmcdirmid 8mo ago
  Most of the compute is actually used for inference (90% if Gemini is to be trusted).
duskdozer 8mo ago
"enough compute" will be when there is no more hardware for use outside of their walled garden, at which point they can control what they want

cmiles8 8mo ago

“Demand” is mostly their training of models, which they’ve yet to demonstrate is a profitable business. Just because you’re struggling to get raw materials for your business doesn’t make it a good business. Without strong enterprise adoption ASAP (which is what’s seriously suffering) things are going to hit the fan real quick.

lancebeet 8mo ago
This will sound snarky, so forgive me, but I honestly don't know the answer. Is this actually true? Is there a reliable source containing statistics on LLM compute usage that includes training vs inference for the whole market?
- concats 8mo ago
  The revenue numbers are public for the major AI companies. That's probably the best estimate for "inference for the whole market" we have, since most of that inference is billed in either API usage or subscriptions, and it won't include any in-house usage such as training.
- seanmcdirmid 8mo ago
  I don’t understand why people don’t just use Gemini or some other AI web search to get an answer to these kinds of questions quickly (I excluded the sources, you can get them if you ask the same question). > While AI training is often the most intense and expensive process for a single model, the majority of total AI compute usage (approximately 90%) is used for inference. > Here is the breakdown of why this is the case: > Inference as High-Volume > Activity: Inference occurs every time a user interacts with an AI model (e.g., asking ChatGPT a question, using image recognition, or generating code). While a model is trained once (or updated infrequently), it runs millions or billions of inferences continuously. > Cost Scaling: Training is a massive, one-time upfront cost, while inference is an ongoing, daily operational cost. As the number of AI users grows, the demand for inference compute scales faster than the need for training new, large models. > The Shift to Efficiency: While early AI hype focused on the immense compute needed for training, the industry has shifted toward making inference cheaper and faster through specialized hardware and techniques like optimization, quantization, and small language models (SLMs).
  - ajam1507 8mo ago
    Gemini is not a reliable source. You posted the only part of the AI response that isn't useful in verifying whether it is true.
    - seanmcdirmid 8mo ago
      Sure, I guess. I asked Gemini to give me some markdown of citations and the claims made that address the question: https://share.google/aimode/v3Y9P3rYIx1oj9VI2 https://share.google/aimode/v3Y9P3rYIx1oj9VI2 And I finally figured out how to get links to answers instead of just inlining the content as before. Anyways, there it is. We live in a time where questions like "Does inference or training use more compute?" can be answered quickly by just pasting it into a search box.
aurareturn 8mo ago
Do you have source?
- tempodox 8mo ago
  Just ask the slop generator of your choice. Making sense of all that smoke and mirrors without AI support has become impossible.
couchdb_ouchdb 8mo ago
With respect, I don't think you've used the latest models and have not seen Anthropic's enterprise revenue hockey-stick like number. They are so busy outfitting fortune-500, you can't even get someone in sales to respond to emails. I've been waiting for months and so have others.
seanmcdirmid 8mo ago
Most of the compute is actually used for inference (90% if Gemini is to be trusted).

duskdozer 8mo ago

"enough compute" will be when there is no more hardware for use outside of their walled garden, at which point they can control what they want