13 ms·

Yeah, make the network deeper. When all you have is a hammer... It makes a lot of sense that a transformation layer that makes the tokens more semantically rel

Yeah, make the network deeper. When all you have is a hammer... It makes a lot of sense that a transformation layer that makes the tokens more semantically relevant will help optimize the entire network after it and increase the effective size of your context window. And one of the main immediate obstacle stopping those models from being intelligent is context window size. On the other hand, the current models already cost something on the line of the median country GDP to train, and they are nowhere close to that in value. The saying that "if brute force didn't solve your problem, you didn't apply enough force" is intended to be listened as a joke.

jagraff 1y ago
I think the median country GDP is something like $100 Billion https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) Models are expensive, but they're not that expensive.
- telotortium 1y ago
  LLM model training costs arise primarily from commodity costs (GPUs and other compute as well as electricity), not locally-provided services, so PPP is not the right statistic to use here. You should use nominal GDP for this instead. According to Wikipedia[0], the median country's nominal GDP (Cyprus) is more like $39B. Still much larger than training costs, but much lower than your PPP GDP number. [0] https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...
- amelius 1y ago
  Maybe it checks out if you don't use 1 year as your timeframe for GDP but the number of days required for training.
- kordlessagain 1y ago
  The median country GDP is approximately $48.8 billion, which corresponds to Uganda at position 90 with $48.769 billion. The largest economy (US) has a GDP of $27.7 trillion. The smallest economy (Tuvalu) has a GDP of $62.3 million. The 48 billion number represents the middle point where half of all countries have larger GDPs and half have smaller GDPs.
  - hoseja 1y ago
    Well then you have to agree that $48.8 billion IS "something like $100 billion".
- Nicook 1y ago
  does anyone even have good estimates for model training?
- marcosdumay 1y ago
  $100 billion is the best estimate around of how much OpenAI took in investment to build ChatGPT.
  - programjames 1y ago
    Maybe for an Australian billion? But in American English it would be $100 million.
    - ash_091 1y ago
      In both Australian and American English a billion is 1,000,000,000 (one thousand million).
  - Eisenstein 1y ago
    A top of the line consumer desktop, the Mac Pro, costs $7000. The commonly acknowledged first non-mechanical computer, the ENIAC, cost $400,000, which adjusted for inflation is $6,594,153 (see note). Will AI models follow the same pricing trajectory? Probably not but they no longer cost even close to $100 billion. Note: 1946 CPI = 19.5, 2025 CPI = 321.465 which makes for an increase of 16.49.
    - ayewo 1y ago
      It seems all you did was use this formula: CPI{2025} / CPI{1946} * Price{1946} = Price{2025} to obtain the price adjusted for inflation? That is the only way I was able to arrive at the same number you got: $6,594,153.846. TIL.
    - marcosdumay 1y ago
      > Will AI models follow the same pricing trajectory? If the article is correct, and this is the best way to make them, their price will explode.
      - mr_toad 1y ago
        Step 1: Spend hundreds of billions on AI Step 2: ? Step 3: Profit. It’s not enough to have the biggest model, or the best model per dollar spent, you still need to figure out how to make money with it. It’s not clear than vastly increased expenditure will produce a good ROI.
whiplash451 1y ago
I get your point but do we have evidence behind “ something on the line of the median country GDP to train”? Is this really true?
- robrenaud 1y ago
  It's not even close.
helloplanets 1y ago
> the current models already cost something on the line of the median country GDP to train This is just blatantly false. > According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute. https://hai.stanford.edu/ai-index/2024-ai-index-report https://hai.stanford.edu/ai-index/2024-ai-index-report No need to even open up the actual report to find that. Just scroll down the page to read the 'key takeaways'.
hiddencost 1y ago
I mean brute force is working great. Acceleration is large, cost per unit intelligence are dropping much faster than Moore's law. We've been doing this kind of scaling since the 80s across various measures and are quite good at it.

jagraff 1y ago

I think the median country GDP is something like $100 Billion https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) Models are expensive, but they're not that expensive.

telotortium 1y ago
LLM model training costs arise primarily from commodity costs (GPUs and other compute as well as electricity), not locally-provided services, so PPP is not the right statistic to use here. You should use nominal GDP for this instead. According to Wikipedia[0], the median country's nominal GDP (Cyprus) is more like $39B. Still much larger than training costs, but much lower than your PPP GDP number. [0] https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...
amelius 1y ago
Maybe it checks out if you don't use 1 year as your timeframe for GDP but the number of days required for training.
kordlessagain 1y ago
The median country GDP is approximately $48.8 billion, which corresponds to Uganda at position 90 with $48.769 billion. The largest economy (US) has a GDP of $27.7 trillion. The smallest economy (Tuvalu) has a GDP of $62.3 million. The 48 billion number represents the middle point where half of all countries have larger GDPs and half have smaller GDPs.
- hoseja 1y ago
  Well then you have to agree that $48.8 billion IS "something like $100 billion".
Nicook 1y ago
does anyone even have good estimates for model training?
marcosdumay 1y ago
$100 billion is the best estimate around of how much OpenAI took in investment to build ChatGPT.
- programjames 1y ago
  Maybe for an Australian billion? But in American English it would be $100 million.
  - ash_091 1y ago
    In both Australian and American English a billion is 1,000,000,000 (one thousand million).
- Eisenstein 1y ago
  A top of the line consumer desktop, the Mac Pro, costs $7000. The commonly acknowledged first non-mechanical computer, the ENIAC, cost $400,000, which adjusted for inflation is $6,594,153 (see note). Will AI models follow the same pricing trajectory? Probably not but they no longer cost even close to $100 billion. Note: 1946 CPI = 19.5, 2025 CPI = 321.465 which makes for an increase of 16.49.
  - ayewo 1y ago
    It seems all you did was use this formula: CPI{2025} / CPI{1946} * Price{1946} = Price{2025} to obtain the price adjusted for inflation? That is the only way I was able to arrive at the same number you got: $6,594,153.846. TIL.
  - marcosdumay 1y ago
    > Will AI models follow the same pricing trajectory? If the article is correct, and this is the best way to make them, their price will explode.
    - mr_toad 1y ago
      Step 1: Spend hundreds of billions on AI Step 2: ? Step 3: Profit. It’s not enough to have the biggest model, or the best model per dollar spent, you still need to figure out how to make money with it. It’s not clear than vastly increased expenditure will produce a good ROI.

whiplash451 1y ago

I get your point but do we have evidence behind “ something on the line of the median country GDP to train”? Is this really true?

robrenaud 1y ago
It's not even close.

helloplanets 1y ago

> the current models already cost something on the line of the median country GDP to train This is just blatantly false. > According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute. https://hai.stanford.edu/ai-index/2024-ai-index-report https://hai.stanford.edu/ai-index/2024-ai-index-report No need to even open up the actual report to find that. Just scroll down the page to read the 'key takeaways'.

hiddencost 1y ago

I mean brute force is working great. Acceleration is large, cost per unit intelligence are dropping much faster than Moore's law. We've been doing this kind of scaling since the 80s across various measures and are quite good at it.