5 ms·

> “The intelligence of an AI model roughly equals the log of the resources used to train and run it,” [Sam Altman] Taking that at face value, it means we would

by layer8 1y ago

> “The intelligence of an AI model roughly equals the log of the resources used to train and run it,” [Sam Altman] Taking that at face value, it means we would have to invest exponential resources just to get linear improvements. That’s not exactly an optimistic outlook.

noir_lord 1y ago
"A deep incisive point, it seems like you want to turn the entire mass of the solar system into computronium to run ChatGPT27".
- sph 1y ago
  Grey goo scenario, but the goo are NVIDIA cards used to train LLMs.
  - an0malous 1y ago
    We probably already live in a simulation where an LLM is trying to compute how many “r”s are on “strawberry”
    - Angostura 1y ago
      If so, expect seahorses to never have existed by next week
      - deleted 1y ago
        [deleted]
deleted 1y ago
[deleted]
Neywiny 1y ago
And in his interviews he talks about the near vertical progress they're making. Of course nobody agrees but if those two are true, that's doubly exponential to keep pace.
PeterStuer 1y ago
Even today's frontier models, without any further improvement, have incredible commercial potential. And even when improvements have diminisching returns in terms of result, the market shifts towards improved models in a supralinear fashion. So a linear improvement resulting from exponential investment might still net you exponential commercial return. Also, the LLM space is a red queen environment. Stop investing and you are done. All that said. IMHO Short to medium term breakthrouhgs will come from hybrid AI systems, the LLM being the universal putty between all users and systems.
- otabdeveloper4 1y ago
  > have incredible commercial potential That assertion is unsupported and unproven. Also, if a commercial use for LLMs is ever found, it will be in the local, personal computing market, not the "cloud".
  - diamond559 1y ago
    It's actually completely been disproven, Open"AI" is burning over ten billion a quarter in net losses. It has no commercial value.
    - shakow 1y ago
      To play the Devil's advocate, OpenAI actually has hundreds of millions in income, even though they are spending far more in training new models.
      - rhetocj23 1y ago
        The issue is, as it stands, LLMs are going to become a commodity. Could that happen and OAI is still a valuable firm? Maybe. Do I personally think OpenAI has the product prowess to pull it off? Nah, I think they are overly-concentrated with technologists (same problem Google has). Apple is still king when it comes to figuring out the ideal form of the product the buyer desires. Apple can take a future OSS model and produce the winning product. That truth would be very bitter for many to swallow. Cook maintaining good relations with China could be the thing that makes Apple topple everyone in the long-run.
locknitpicker 1y ago
> Taking that at face value, it means we would have to invest exponential resources just to get linear improvements. Not necessarily. Approaches such as mixture of experts help lower training costs by covering domains with specialized models.
- hiddencost 1y ago
  I encourage you to rethink your identity. You are way out of your depth on this, and posting nonsensical things as fact.
- diamond559 1y ago
  haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!!
  - locknitpicker 1y ago
    > haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!! I understand it's very easy to post ignorant messages in internet forums, but the answer to your question is yes, "they have done it" and it does result in cheaper training costs. See models such as DeepSeek-MoE or Mixtral. https://github.com/deepseek-ai/DeepSeek-MoE https://github.com/deepseek-ai/DeepSeek-MoE https://mistral.ai/news/mixtral-of-experts https://mistral.ai/news/mixtral-of-experts
tim333 1y ago
I think he means resources as in compute cycles and the like. Those tend to increase exponentially for the same number of dollars in a Moore's law type way to intelligence should increase in an approximately linear way, something similar to a few IQ points per year. You can see a similar effect in computer chess ELO scores over time, with the odd blip up see https://www.reddit.com/r/dataisbeautiful/comments/1iovlb0/oc_how_do_humans_compare_to_the_best_chess_engines/ https://www.reddit.com/r/dataisbeautiful/comments/1iovlb0/oc... (1985 - 2023) and https://wiki.aiimpacts.org/speed_of_ai_transition/range_of_human_performance/time_for_ai_to_cross_the_human_performance_range_in_chess https://wiki.aiimpacts.org/speed_of_ai_transition/range_of_h... (1960 - 2000)
darthvaden 1y ago
Make openai a non corporate entity that belongs to the UN. It can't be profit driven. Fuck the economic pundits. This tech is beyond humans it needs to be done right. "All tech have been Enshitified to satisfy investor greed, sufficiently advanced tech can be called magic and these fools will enshitify magic also"
- darthvaden 1y ago
  Create a cern

by layer8 1y ago

noir_lord 1y ago

"A deep incisive point, it seems like you want to turn the entire mass of the solar system into computronium to run ChatGPT27".

sph 1y ago
Grey goo scenario, but the goo are NVIDIA cards used to train LLMs.
- an0malous 1y ago
  We probably already live in a simulation where an LLM is trying to compute how many “r”s are on “strawberry”
  - Angostura 1y ago
    If so, expect seahorses to never have existed by next week
    - deleted 1y ago
      [deleted]

deleted 1y ago

[deleted]

Neywiny 1y ago

And in his interviews he talks about the near vertical progress they're making. Of course nobody agrees but if those two are true, that's doubly exponential to keep pace.

PeterStuer 1y ago

Even today's frontier models, without any further improvement, have incredible commercial potential. And even when improvements have diminisching returns in terms of result, the market shifts towards improved models in a supralinear fashion. So a linear improvement resulting from exponential investment might still net you exponential commercial return. Also, the LLM space is a red queen environment. Stop investing and you are done. All that said. IMHO Short to medium term breakthrouhgs will come from hybrid AI systems, the LLM being the universal putty between all users and systems.

otabdeveloper4 1y ago
> have incredible commercial potential That assertion is unsupported and unproven. Also, if a commercial use for LLMs is ever found, it will be in the local, personal computing market, not the "cloud".
- diamond559 1y ago
  It's actually completely been disproven, Open"AI" is burning over ten billion a quarter in net losses. It has no commercial value.
  - shakow 1y ago
    To play the Devil's advocate, OpenAI actually has hundreds of millions in income, even though they are spending far more in training new models.
    - rhetocj23 1y ago
      The issue is, as it stands, LLMs are going to become a commodity. Could that happen and OAI is still a valuable firm? Maybe. Do I personally think OpenAI has the product prowess to pull it off? Nah, I think they are overly-concentrated with technologists (same problem Google has). Apple is still king when it comes to figuring out the ideal form of the product the buyer desires. Apple can take a future OSS model and produce the winning product. That truth would be very bitter for many to swallow. Cook maintaining good relations with China could be the thing that makes Apple topple everyone in the long-run.

locknitpicker 1y ago

> Taking that at face value, it means we would have to invest exponential resources just to get linear improvements. Not necessarily. Approaches such as mixture of experts help lower training costs by covering domains with specialized models.

hiddencost 1y ago
I encourage you to rethink your identity. You are way out of your depth on this, and posting nonsensical things as fact.
diamond559 1y ago
haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!!
- locknitpicker 1y ago
  > haha good one, so why haven't they done this yet? What are they waiting for? Let's see these super advanced "experts" with "specialized models"!! I understand it's very easy to post ignorant messages in internet forums, but the answer to your question is yes, "they have done it" and it does result in cheaper training costs. See models such as DeepSeek-MoE or Mixtral. https://github.com/deepseek-ai/DeepSeek-MoE https://github.com/deepseek-ai/DeepSeek-MoE https://mistral.ai/news/mixtral-of-experts https://mistral.ai/news/mixtral-of-experts

tim333 1y ago

I think he means resources as in compute cycles and the like. Those tend to increase exponentially for the same number of dollars in a Moore's law type way to intelligence should increase in an approximately linear way, something similar to a few IQ points per year. You can see a similar effect in computer chess ELO scores over time, with the odd blip up see https://www.reddit.com/r/dataisbeautiful/comments/1iovlb0/oc_how_do_humans_compare_to_the_best_chess_engines/ https://www.reddit.com/r/dataisbeautiful/comments/1iovlb0/oc... (1985 - 2023) and https://wiki.aiimpacts.org/speed_of_ai_transition/range_of_human_performance/time_for_ai_to_cross_the_human_performance_range_in_chess https://wiki.aiimpacts.org/speed_of_ai_transition/range_of_h... (1960 - 2000)

darthvaden 1y ago

Make openai a non corporate entity that belongs to the UN. It can't be profit driven. Fuck the economic pundits. This tech is beyond humans it needs to be done right. "All tech have been Enshitified to satisfy investor greed, sufficiently advanced tech can be called magic and these fools will enshitify magic also"

darthvaden 1y ago
Create a cern