5 ms·

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term su

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the market. OpenAI and Anthropic are raising the costs of inference (by reducing how much inference users get via subscriptions). And we haven’t seen Oracle / CoreWeave struggle to pay their debts yet, but they will be selling assets once they get close to that point.

uejfiweun 3mo ago
What makes you think the supply crunch is short-term?
- msdz 3mo ago
  Unless the raw materials have an inherent limit on mining/production due to the amount present on the planet, why should or would companies not ramp up to eventually meet demand? Edit: Okay, this doesn’t mean that that’s actually possible in the short-term, so I think you’re right. But that means as the silver lining, in the medium term horizon there’ll be enough supply again? :’)
  - JumpCrisscross 3mo ago
    > why should or would companies not ramp up to eventually meet demand? Memory is a cyclical market that has historically rewarded conservatism [1]. Counterpoint: there is enough demand from enough capital-rich customers that they may be willing to shoulder the capital risk. [1] https://www.ldeepai.com/tech-hub/dram-industry-consolidation-three-survivors-memory-scale/#:~:text=The%20DRAM%20industry,memory%20inevitably%20consolidates https://www.ldeepai.com/tech-hub/dram-industry-consolidation... Sorry for the slop link, it has a good chart from a solid source
    - msdz 3mo ago
      Thanks for the link (and underlying thoughts), I really hadn’t considered that. So essentially, due to technological progress and other factors inducing price collapses (or at least cycles), you can’t start stockpiling insane amounts of finished-product semiconductor, which means you can’t scale production at current technology levels to infinity either?
    - AussieWog93 3mo ago
      I heard that China was spinning up DDR5 (but not HBM?) production in the next couple of years, with the hope of outcompeting Korea and Taiwan in the mid to long term.
      - HPsquared 3mo ago
        It does seem like an opportunity on a silver platter for Chinese newcomers. Huge demand at the moment.
        Danox 3mo ago
        In two to four years, the Chinese will have at least half the memory market worldwide, and once in, they will continue forward and not look back. I also believe there would be one or two tech companies that will get into memory by taking it in-house to make sure that they won’t have this problem again in the future.
    - djfergus 3mo ago
      [dead]
    - techpression 3mo ago
      Don't forget price fixing [1] which we are seeing clear indicators of happening right now too. [1] https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal
    - wongarsu 3mo ago
      For existing producers expanding capacity would be a risky move. But it's the perfect time for any newcomers to enter the market. Low yields and worse product don't matter as much right now, and by the time the market cools down you have everything dialed in and can compete on even ground
- IsTom 3mo ago
  If demand doesn't fall down or current manufacturers supply go up, somebody (presumably in China) will spin up fabs. Apple wanted to use blacklisted Chinese RAM already.
  - byzantinegene 3mo ago
    I am hopeful but I am not confident China has the capability to do it.
    - anonym29 3mo ago
      They're already doing it https://www.techspot.com/news/112502-memory-prices-tipped-fall-china-starts-flooding-market.html https://www.techspot.com/news/112502-memory-prices-tipped-fa...
    - coldtea 3mo ago
      As if it requires some unique genes or brains? If a place can do it, another place, with a huge track record on manufacturing and lately expanding all kinds of tech, can.
      - pfannkuchen 3mo ago
        It might help to not have much in the way of environmental, safety or labor regulations. Whether or not you feel like those are good overall (I do), they do actually also slow things down.
        coldtea 3mo ago
        >It might help to not have much in the way of environmental, safety or labor regulations. Yes, like how it helped western industry early on. Or, well into the 70s for the most part.
  - not-a-llm 3mo ago
    where is this cheap Chinese RAM? I'd like to buy some
- t-3 3mo ago
  If the increased demand is not short term, production capacity will eventually increase. In the meantime, the logistics disruptions and industrial material shortages and energy inflation will disappear as soon as the wars disrupting them stop, which should bring prices down.
  - not-a-llm 3mo ago
    you assume demand will remain flat what if demand keeps rising faster than production capacity is deployed?
    - t-3 3mo ago
      If demand and prices keep rising without production capacity being built fast enough, there will likely eventually be a rush leading to overinvestment and price crashes, but there are too many other factors involved; state investment for security, international politics and trade relations, the possibility of an AI bubble burst, etc.
  - MomsAVoxell 3mo ago
    There are wars coming. The prices are not going down. We are in a bubble which will be burst the moment the world starts retaliating against the US' 20+ year history of supporting genocide and committing war crimes unabated. Buy the AI toys while you still can.
- GeekyBear 3mo ago
  DRAM doesn't require cutting edge Fabs, and Chinese manufacturers have finally caught up to the DDR5 standard. > Multiple motherboard and PC component makers move forward with Chinese-made memory validation https://www.pcgamer.com/hardware/memory/multiple-motherboard-and-pc-component-makers-move-forward-with-chinese-made-memory-validation/ https://www.pcgamer.com/hardware/memory/multiple-motherboard... The only thing standing in the way of a major Chinese DDR5 ramp up is money.
- thephyber 3mo ago
  The underlying demand for massive RAM builds is coming from AI hyperscalers. There are lots of signals that the sector has been overinvested and that corporate customers are pulling back on spending as the cost of the APIs is revealed. Once the hyperscalers start struggling to bay their debts (it will happen, just a question of time), there will be a supply glut. So the only question is: do we share the same definition of “short term”.
throw1234567891 3mo ago
512GB M3 Ultra is out of stock, not coming back, and there’s nothing like it on the consumer market. That’s the reason they go for so much.
- brandensilva 3mo ago
  But there will be and his point is they cut off supply to make room for the new M5 ultra which I hope has 768GB or more of memory.
  - throw1234567891 3mo ago
    Let’s see, I hope!
  - throw1234567891 3mo ago
    By the way, they cut the supply because they run out of RAM. They still sell the 96 GB model. Everything else is gone, no stock, they cannot manufacture them. Even minis are 48GB max!
- jazzyjackson 3mo ago
  Yea but how many of them are selling at that price? Dozens?
- 3abiton 3mo ago
  Funnily the current high end Mac Studio are not suited for current LLMs. M3 Ultra is "quite an old" chip for AI, despite its bandwidth. The issue for running local models (especially LLMs), you need few things to align really well: 1) compute power (affecting PP) 2) VRAM capacity (affecting model size you can load) 3) Bandwidth (somewhat affecting decoding speed). The issue with the M3 chip is the compute performance, as it doesn't fit well the transformer architecture. This changed with the M5 (apple baked their own matmul into the chip), which would significantly speed up PP (and video/image generation btw), making the M5 Ultra significantly faster than the M3 Ultra and in practice much more usable. You can try to load Kimi or GLM on M3 Ultra, but it's not usable. Now the M5 Ultra is not out yet, but undoubtedly it will be a superior offering, and shilling 15k on 512GB version is actually reasonable (if it's every priced remotely around that tag).

uejfiweun 3mo ago

What makes you think the supply crunch is short-term?

msdz 3mo ago
Unless the raw materials have an inherent limit on mining/production due to the amount present on the planet, why should or would companies not ramp up to eventually meet demand? Edit: Okay, this doesn’t mean that that’s actually possible in the short-term, so I think you’re right. But that means as the silver lining, in the medium term horizon there’ll be enough supply again? :’)
- JumpCrisscross 3mo ago
  > why should or would companies not ramp up to eventually meet demand? Memory is a cyclical market that has historically rewarded conservatism [1]. Counterpoint: there is enough demand from enough capital-rich customers that they may be willing to shoulder the capital risk. [1] https://www.ldeepai.com/tech-hub/dram-industry-consolidation-three-survivors-memory-scale/#:~:text=The%20DRAM%20industry,memory%20inevitably%20consolidates https://www.ldeepai.com/tech-hub/dram-industry-consolidation... Sorry for the slop link, it has a good chart from a solid source
  - msdz 3mo ago
    Thanks for the link (and underlying thoughts), I really hadn’t considered that. So essentially, due to technological progress and other factors inducing price collapses (or at least cycles), you can’t start stockpiling insane amounts of finished-product semiconductor, which means you can’t scale production at current technology levels to infinity either?
  - AussieWog93 3mo ago
    I heard that China was spinning up DDR5 (but not HBM?) production in the next couple of years, with the hope of outcompeting Korea and Taiwan in the mid to long term.
    - HPsquared 3mo ago
      It does seem like an opportunity on a silver platter for Chinese newcomers. Huge demand at the moment.
      - Danox 3mo ago
        In two to four years, the Chinese will have at least half the memory market worldwide, and once in, they will continue forward and not look back. I also believe there would be one or two tech companies that will get into memory by taking it in-house to make sure that they won’t have this problem again in the future.
  - djfergus 3mo ago
    [dead]
  - techpression 3mo ago
    Don't forget price fixing [1] which we are seeing clear indicators of happening right now too. [1] https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal
  - wongarsu 3mo ago
    For existing producers expanding capacity would be a risky move. But it's the perfect time for any newcomers to enter the market. Low yields and worse product don't matter as much right now, and by the time the market cools down you have everything dialed in and can compete on even ground
IsTom 3mo ago
If demand doesn't fall down or current manufacturers supply go up, somebody (presumably in China) will spin up fabs. Apple wanted to use blacklisted Chinese RAM already.
- byzantinegene 3mo ago
  I am hopeful but I am not confident China has the capability to do it.
  - anonym29 3mo ago
    They're already doing it https://www.techspot.com/news/112502-memory-prices-tipped-fall-china-starts-flooding-market.html https://www.techspot.com/news/112502-memory-prices-tipped-fa...
  - coldtea 3mo ago
    As if it requires some unique genes or brains? If a place can do it, another place, with a huge track record on manufacturing and lately expanding all kinds of tech, can.
    - pfannkuchen 3mo ago
      It might help to not have much in the way of environmental, safety or labor regulations. Whether or not you feel like those are good overall (I do), they do actually also slow things down.
      - coldtea 3mo ago
        >It might help to not have much in the way of environmental, safety or labor regulations. Yes, like how it helped western industry early on. Or, well into the 70s for the most part.
- not-a-llm 3mo ago
  where is this cheap Chinese RAM? I'd like to buy some
t-3 3mo ago
If the increased demand is not short term, production capacity will eventually increase. In the meantime, the logistics disruptions and industrial material shortages and energy inflation will disappear as soon as the wars disrupting them stop, which should bring prices down.
- not-a-llm 3mo ago
  you assume demand will remain flat what if demand keeps rising faster than production capacity is deployed?
  - t-3 3mo ago
    If demand and prices keep rising without production capacity being built fast enough, there will likely eventually be a rush leading to overinvestment and price crashes, but there are too many other factors involved; state investment for security, international politics and trade relations, the possibility of an AI bubble burst, etc.
- MomsAVoxell 3mo ago
  There are wars coming. The prices are not going down. We are in a bubble which will be burst the moment the world starts retaliating against the US' 20+ year history of supporting genocide and committing war crimes unabated. Buy the AI toys while you still can.
GeekyBear 3mo ago
DRAM doesn't require cutting edge Fabs, and Chinese manufacturers have finally caught up to the DDR5 standard. > Multiple motherboard and PC component makers move forward with Chinese-made memory validation https://www.pcgamer.com/hardware/memory/multiple-motherboard-and-pc-component-makers-move-forward-with-chinese-made-memory-validation/ https://www.pcgamer.com/hardware/memory/multiple-motherboard... The only thing standing in the way of a major Chinese DDR5 ramp up is money.
thephyber 3mo ago
The underlying demand for massive RAM builds is coming from AI hyperscalers. There are lots of signals that the sector has been overinvested and that corporate customers are pulling back on spending as the cost of the APIs is revealed. Once the hyperscalers start struggling to bay their debts (it will happen, just a question of time), there will be a supply glut. So the only question is: do we share the same definition of “short term”.

throw1234567891 3mo ago

512GB M3 Ultra is out of stock, not coming back, and there’s nothing like it on the consumer market. That’s the reason they go for so much.

brandensilva 3mo ago
But there will be and his point is they cut off supply to make room for the new M5 ultra which I hope has 768GB or more of memory.
- throw1234567891 3mo ago
  Let’s see, I hope!
- throw1234567891 3mo ago
  By the way, they cut the supply because they run out of RAM. They still sell the 96 GB model. Everything else is gone, no stock, they cannot manufacture them. Even minis are 48GB max!
jazzyjackson 3mo ago
Yea but how many of them are selling at that price? Dozens?
3abiton 3mo ago
Funnily the current high end Mac Studio are not suited for current LLMs. M3 Ultra is "quite an old" chip for AI, despite its bandwidth. The issue for running local models (especially LLMs), you need few things to align really well: 1) compute power (affecting PP) 2) VRAM capacity (affecting model size you can load) 3) Bandwidth (somewhat affecting decoding speed). The issue with the M3 chip is the compute performance, as it doesn't fit well the transformer architecture. This changed with the M5 (apple baked their own matmul into the chip), which would significantly speed up PP (and video/image generation btw), making the M5 Ultra significantly faster than the M3 Ultra and in practice much more usable. You can try to load Kimi or GLM on M3 Ultra, but it's not usable. Now the M5 Ultra is not out yet, but undoubtedly it will be a superior offering, and shilling 15k on 512GB version is actually reasonable (if it's every priced remotely around that tag).