2 ms·
If local models are good enough, doesn't that increase demand for DRAM as everyone buys DRAM for their poorly utilized local machines? Surely it is a more effi
by fleventynine 5mo ago
If local models are good enough, doesn't that increase demand for DRAM as everyone buys
DRAM for their poorly utilized local machines?
Surely it is a more efficient use of DRAM to run inference on shared hardware with large batch sizes and more utilization.
- szatkus 5mo agoLuckily very few people can configure and are interested in local models. But your nearby datacenter running Chinese open-weight models is also good enough.
- fleventynine 5mo agoMy point is that dram demand is mostly orthogonal to whether everyone is using open weight models or secret weight models. Heavy demand for local models (whether secret or open weight) will require even more aggregate DRAM than for shared. Demand will only go down if people reduce their use of these AI tools. Given how much folks here complain about quotas, I'm very skeptical that will happen willingly.
- zozbot234 5mo agoOpen weight models allow for repurposing existing hardware locally, and there's a lot of it around - far more than the amount of new RAM being supplied. So they add some short-term downward pressure to the price. (But not very much, since these datacenter builds are long-term investments that are targeted at eventually running far larger models.)
- fleventynine 5mo agoIf regular people can repurpose old hardware, so can shared providers, who can extract more value from the hardware and thus afford to pay more. In a constrained market, supply and demand favors folks who can most efficiently extract rent. Local models only make sense in a world with abundant compute and energy.