4 ms·

Hazardous waste? My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models. Maybe eac

by XCSme 2mo ago

Hazardous waste? My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models. Maybe each one of this chip is a layer, so you can stack as many layers as you want.

bronco21016 2mo ago
HBM modules that we can just attach to our GPU? Kinda like CPU RAM.
ethbr1 2mo ago
You're not thinking through the way economics works. If, hypothetically, the thing you wish existed, why wouldn't it be valuable to high-utilization cloud providers? And if useful to them, why would it be sold at a lower price than the (extremely high) price those cloud providers can afford to profitably pay? The only solution to the hardware crunch that eases the cost/access barrier is if there's such a glut of excess supply of newly-commoditized inference hardware that the price craters. And you've yet to propose a feasible manufacturing scenario through which that would be possible.
- XCSme 2mo ago
  Well, before AI the RAM prices were cheapest ever, even if cloud providers were still in demand for hosting and cloud computing. I hope the supply will increase at some point, I doubt cloud providers will be able to buy everything, especially if the competition will be high and the price of AI goes down, so there won't be an easy way to make profits by serving models.
  - ethbr1 2mo ago
    > so there won't be an easy way to make profits by serving models That's the only way I see things going, which ultimately comes down to the market losing faith in AI as a profit engine. Which is probably going to take one of (a) OpenAI / Anthropic IPO failing or newly public financials looking bleak or (b) a high-enough profile large AI company failing (unfortunately most of the large enough to shock the market ones have diversified and durable cash generation engines).
    - seunosewa 2mo ago
      Unfortunately?
dragonwriter 2mo ago
So the solution to the chip supply problem is “pretend there is no chip supply problem and everyone gets the chips they need, but for some reason does some assembly task at home”? If we could just do the first part and get reality to conform, the rest would be superfluous, but I don't think we can do that.
- XCSme 2mo ago
  This is not a solution for now, just a moonshot solution for maybe 5-10 years from now. There's no immediate fix for the RAM supply issue, but as long as it's manufacturing issue, not a raw materials issue, time always solves it.
  - dragonwriter 2mo ago
    Its not a solution ever, because it takes as its premise that the problem is solved and then goes on to suggest something else on top of it (the purpose of which is unclear.)

by XCSme 2mo ago

bronco21016 2mo ago

HBM modules that we can just attach to our GPU? Kinda like CPU RAM.

ethbr1 2mo ago

You're not thinking through the way economics works. If, hypothetically, the thing you wish existed, why wouldn't it be valuable to high-utilization cloud providers? And if useful to them, why would it be sold at a lower price than the (extremely high) price those cloud providers can afford to profitably pay? The only solution to the hardware crunch that eases the cost/access barrier is if there's such a glut of excess supply of newly-commoditized inference hardware that the price craters. And you've yet to propose a feasible manufacturing scenario through which that would be possible.

XCSme 2mo ago
Well, before AI the RAM prices were cheapest ever, even if cloud providers were still in demand for hosting and cloud computing. I hope the supply will increase at some point, I doubt cloud providers will be able to buy everything, especially if the competition will be high and the price of AI goes down, so there won't be an easy way to make profits by serving models.
- ethbr1 2mo ago
  > so there won't be an easy way to make profits by serving models That's the only way I see things going, which ultimately comes down to the market losing faith in AI as a profit engine. Which is probably going to take one of (a) OpenAI / Anthropic IPO failing or newly public financials looking bleak or (b) a high-enough profile large AI company failing (unfortunately most of the large enough to shock the market ones have diversified and durable cash generation engines).
  - seunosewa 2mo ago
    Unfortunately?

dragonwriter 2mo ago

So the solution to the chip supply problem is “pretend there is no chip supply problem and everyone gets the chips they need, but for some reason does some assembly task at home”? If we could just do the first part and get reality to conform, the rest would be superfluous, but I don't think we can do that.

XCSme 2mo ago
This is not a solution for now, just a moonshot solution for maybe 5-10 years from now. There's no immediate fix for the RAM supply issue, but as long as it's manufacturing issue, not a raw materials issue, time always solves it.
- dragonwriter 2mo ago
  Its not a solution ever, because it takes as its premise that the problem is solved and then goes on to suggest something else on top of it (the purpose of which is unclear.)