4 ms·

Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories. There still isn't

by hypeatei 11mo ago

Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories. There still isn't a clear path to profitability for any of these AI products and the capital expenditure has been enormous.

PaulKeeble 11mo ago
Its a bit of a shame these AI GPUs don't actually have displayport/hdmi output ports because they would make for nice cheap and powerful gaming GPUs with a lot of VRAM, they would potentially be really good graphics cards. Will just have to settle for insanely cheap second hand DDR5 and NVMe drives I guess.
- sowbug 11mo ago
  I wouldn't mind my own offline Gemini or ChatGPT 5. But even if the hardware and model were free, I don't know how I'd afford the electricity.
  - jdprgm 11mo ago
    A single machine for personal inference on models of this size isn't going to idle at some point so high that electricity becomes a problem and for personal use it's not like it would be under load often and if for some reason you are able to keep it under heavy load presumably it's doing something valuable enough to easily justify the electricity.
  - mitthrowaway2 11mo ago
    If you can't afford the electricity to afford to run the model on free hardware, you'd certainly never be able to afford the subscription to the same product as a service! But anyway, the trick is to run it in the winter and keep your house warm.
    - sowbug 11mo ago
      I think you're underestimating economies of scale, and today's willingness of large corporations to provide cutting-edge services at a loss.
      - mitthrowaway2 11mo ago
        I don't think I am. I don't think economies of scale on hardware will drive costs below free, and while subscription providers might be willing to offer services below the cost of the hardware that runs them, I don't think they'll offer services below the cost of the electricity that runs them. And while data centers might sign favorable contracts, I don't think they are getting electricity that far below retail.
        sowbug 11mo ago
        Here in California, as a residential customer, I pay close to 50 cents for even the very first kWh I consume each month, and there is no bulk discount. Meanwhile, Google et al. surely pay less than 10% that rate. A single Gemini query probably involves dozens of distributed machines that skilled SREs keep afloat, but let's say very optimistically that if I had a copy of the model, I could run it on a home workstation with six 80GB GPUs, consuming 4,200 watts when active, and 500 watts when idle. My utilization might be 10% on a day when I'm especially curious, whereas Google probably manages 95% utilization of its inference machines. I could go on, but I think you get the point: the dollar cost for me to run a hypothetical version of Gemini at home far exceeds the price Google pays to deliver the same service to me. I'm effectively arbitraging the price of electricity by subscribing to an LLM service, because I could never run it for anywhere near the same price in my house -- and to provide the service at the speed and reliability of Gemini, I doubt I could even fit the equipment in my house-sized house and still have room for me to live in it. Anyway, I stand by my original comment - it's neither easy nor cheap to run a frontier model at home.
- GuB-42 11mo ago
  AI GPUs suck for gaming, I have seen a video from a guy playing Red Dead Redemption 2 on a H100 at a whooping 8 FPS! That's after some hacks, because otherwise it wouldn't run at all. AI GPUs are stripped away of most things display-related to make room for more compute cores. So in theory, they could "work", but there are bottlenecks making that compute power irrelevant for gaming, even if they had a display output.
  - 1718627440 11mo ago
    So there aren't actually GPUs, but more like some other architecture for CPUs.
cesarb 11mo ago
> Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories. Their inventories are not what consumers use. Consumer DDR5 motherboards normally take UDIMMs. Server DDR5 motherboards normally take RDIMMs. They're mechanically incompatible, and the voltages are different. And the memory for GPUs is normally soldered directly to the board (and of the GDDRn family, instead of the DDRn or LPDDRn families used by most CPUs). As for GPUs, they're also different. Most consumer GPUs are PCIe x16 cards with DP and HDMI ports; most hyperscaler GPUs are going to have more exotic form factors like OAM, and not have any DP or HDMI ports (since they have no need for graphics output). So no, unfortunately hyperscalers dumping their inventories would be of little use to consumers. We'll have to wait for the factories to switch their production to consumer-targeted products. Edit: even their NVMe drives are going to have different form factors like E1.S and different connectors like U.2, making them hard for normal consumers to use.
- arjie 11mo ago
  I imagine the cost is primarily in the actual DRAM chips on the DIMM. So availability of RDIMMs on the market will affect DRAM prices anyway. These days lots of motherboards come with Oculink, etc. and you can get a U.2 PCIe card for rather cheap. I put together a small server with mostly commodity parts.
- nine_k 11mo ago
  I bet that friendly Chinese entrepreneurs will sell inexpensive E1.S to m.2 adapters, and maybe even PCIe riser cards for putting an OAM and a bunch of fans, and maybe even an HDMI output. Good hardware won't be wasted, given some demand.
  - bilegeek 11mo ago
    Inexpensive, probably not. E1.S isn't just a different connector, it's a completely different protocol than PCIe.
    - snuxoll 11mo ago
benjojo12 11mo ago
The problem is that it is not entirely clear that the hyperscalers are buying DDR5, instead it seems that supplies are being diverted so that more HBM/GDDR wafers can be produced. HBM/GDDR is not necessarily as useful to the average person as DDR4/DDR5
TacticalCoder 11mo ago
[dead]
phyzix5761 11mo ago
I see it a bit differently. In marketing, companies like AppLovin with the Axon Engine and Zeta Global with Athena are already showing strong profitability, both in earnings and free cash flow. They’re also delivering noticeably higher returns on ad spend compared to pre-AI tools for their customers. This is the area I’m researching most closely, so I can only speak for marketing, but I’d love to hear from others seeing similar results in their industries.
deleted 11mo ago
[deleted]

PaulKeeble 11mo ago

Its a bit of a shame these AI GPUs don't actually have displayport/hdmi output ports because they would make for nice cheap and powerful gaming GPUs with a lot of VRAM, they would potentially be really good graphics cards. Will just have to settle for insanely cheap second hand DDR5 and NVMe drives I guess.

sowbug 11mo ago
I wouldn't mind my own offline Gemini or ChatGPT 5. But even if the hardware and model were free, I don't know how I'd afford the electricity.
- jdprgm 11mo ago
  A single machine for personal inference on models of this size isn't going to idle at some point so high that electricity becomes a problem and for personal use it's not like it would be under load often and if for some reason you are able to keep it under heavy load presumably it's doing something valuable enough to easily justify the electricity.
- mitthrowaway2 11mo ago
  If you can't afford the electricity to afford to run the model on free hardware, you'd certainly never be able to afford the subscription to the same product as a service! But anyway, the trick is to run it in the winter and keep your house warm.
  - sowbug 11mo ago
    I think you're underestimating economies of scale, and today's willingness of large corporations to provide cutting-edge services at a loss.
    - mitthrowaway2 11mo ago
      I don't think I am. I don't think economies of scale on hardware will drive costs below free, and while subscription providers might be willing to offer services below the cost of the hardware that runs them, I don't think they'll offer services below the cost of the electricity that runs them. And while data centers might sign favorable contracts, I don't think they are getting electricity that far below retail.
      - sowbug 11mo ago
        Here in California, as a residential customer, I pay close to 50 cents for even the very first kWh I consume each month, and there is no bulk discount. Meanwhile, Google et al. surely pay less than 10% that rate. A single Gemini query probably involves dozens of distributed machines that skilled SREs keep afloat, but let's say very optimistically that if I had a copy of the model, I could run it on a home workstation with six 80GB GPUs, consuming 4,200 watts when active, and 500 watts when idle. My utilization might be 10% on a day when I'm especially curious, whereas Google probably manages 95% utilization of its inference machines. I could go on, but I think you get the point: the dollar cost for me to run a hypothetical version of Gemini at home far exceeds the price Google pays to deliver the same service to me. I'm effectively arbitraging the price of electricity by subscribing to an LLM service, because I could never run it for anywhere near the same price in my house -- and to provide the service at the speed and reliability of Gemini, I doubt I could even fit the equipment in my house-sized house and still have room for me to live in it. Anyway, I stand by my original comment - it's neither easy nor cheap to run a frontier model at home.
GuB-42 11mo ago
AI GPUs suck for gaming, I have seen a video from a guy playing Red Dead Redemption 2 on a H100 at a whooping 8 FPS! That's after some hacks, because otherwise it wouldn't run at all. AI GPUs are stripped away of most things display-related to make room for more compute cores. So in theory, they could "work", but there are bottlenecks making that compute power irrelevant for gaming, even if they had a display output.
- 1718627440 11mo ago
  So there aren't actually GPUs, but more like some other architecture for CPUs.

cesarb 11mo ago

> Well, patience as a consumer might pay off in the next year or so when the music stops and hyperscalers are forced to dump their inventories. Their inventories are not what consumers use. Consumer DDR5 motherboards normally take UDIMMs. Server DDR5 motherboards normally take RDIMMs. They're mechanically incompatible, and the voltages are different. And the memory for GPUs is normally soldered directly to the board (and of the GDDRn family, instead of the DDRn or LPDDRn families used by most CPUs). As for GPUs, they're also different. Most consumer GPUs are PCIe x16 cards with DP and HDMI ports; most hyperscaler GPUs are going to have more exotic form factors like OAM, and not have any DP or HDMI ports (since they have no need for graphics output). So no, unfortunately hyperscalers dumping their inventories would be of little use to consumers. We'll have to wait for the factories to switch their production to consumer-targeted products. Edit: even their NVMe drives are going to have different form factors like E1.S and different connectors like U.2, making them hard for normal consumers to use.

arjie 11mo ago
I imagine the cost is primarily in the actual DRAM chips on the DIMM. So availability of RDIMMs on the market will affect DRAM prices anyway. These days lots of motherboards come with Oculink, etc. and you can get a U.2 PCIe card for rather cheap. I put together a small server with mostly commodity parts.
nine_k 11mo ago
I bet that friendly Chinese entrepreneurs will sell inexpensive E1.S to m.2 adapters, and maybe even PCIe riser cards for putting an OAM and a bunch of fans, and maybe even an HDMI output. Good hardware won't be wasted, given some demand.
- bilegeek 11mo ago
  Inexpensive, probably not. E1.S isn't just a different connector, it's a completely different protocol than PCIe.
  - snuxoll 11mo ago

benjojo12 11mo ago

The problem is that it is not entirely clear that the hyperscalers are buying DDR5, instead it seems that supplies are being diverted so that more HBM/GDDR wafers can be produced. HBM/GDDR is not necessarily as useful to the average person as DDR4/DDR5

TacticalCoder 11mo ago

[dead]

phyzix5761 11mo ago

I see it a bit differently. In marketing, companies like AppLovin with the Axon Engine and Zeta Global with Athena are already showing strong profitability, both in earnings and free cash flow. They’re also delivering noticeably higher returns on ad spend compared to pre-AI tools for their customers. This is the area I’m researching most closely, so I can only speak for marketing, but I’d love to hear from others seeing similar results in their industries.

deleted 11mo ago

[deleted]