3 ms·
The problem is even if an OSS had the resources (massive data centers the size of NYC packed with top end custom GPU kits) to produce the weights, you need enor
by fnordpiglet 7mo ago
The problem is even if an OSS had the resources (massive data centers the size of NYC packed with top end custom GPU kits) to produce the weights, you need enormous VRAM laden farms of GPUs to do inference on a model like Opus 4.6. Unless the very math of frontier LLMs changes, don’t expect frontier OSS on par to be practical.
- palmotea 7mo ago> you need enormous VRAM laden farms of GPUs to do inference on a model like Opus 4.6. It's probably a trade secret, but what's the actual per-user resource requirement to run the model?
- deleted 7mo ago[deleted]
- supern0va 7mo agoThere's already an ecosystem of essentially undifferentiated infrastructure providers that sell cheap inference of open weights models that have pretty tight margins. If the open weights models are good, there are people looking to sell commodity access to it, much like a cloud provider selling you compute.
- lukeschlather 7mo agoI feel like you're overstating the resources required by a couple orders of magnitude. You do need a GPU farm to do training, but probably only $100M, maybe $1B of GPUs. And yes, that's a lot of GPUs, but they will fit in a single datacenter, and even in dollar terms, there are many individual buildings in NYC that are cheaper.
- fnordpiglet 7mo agoI refer you to the data centers under construction roughly the size of Manhattan to do next generation model training. Granted they’re also to house inference, but my statement wasn’t hyperbole, it’s based on actual reality. To accommodate the next generation of frontier training it’s infeasible for any but the most wealthy organizations on earth to participate. OSS weights are toys. (Mind you i like toys)