3 ms·

Honestly I think the opposite. All these giant tech companies can afford to burn money with ever bigger models and ever more compute and I think that is actuall

Honestly I think the opposite. All these giant tech companies can afford to burn money with ever bigger models and ever more compute and I think that is actually getting in their way. I wager that some scrappy resource constrained startup or research institute will find a way to produce results that are similar to those generated by these ever massive LLM projects only at a fraction of the cost. And I think they’ll do that by pruning the shit out of the model. You don’t need to waste model space on ancient Roman history or the entire canon for the marvel cinematic universe on a model designed to refactor code. You need a model that is fluent in English and “code”. I think the future will be tightly focused models that can run on inexpensive hardware. And unlike today where only the richest companies on the planet can afford training, anybody with enough inclination will be able to train them. (And you can go on a huge tangent why such a thing is absolutely crucial to a free society) I dunno. My point is, there is little incentive for these huge companies to “think small”. They have virtually unlimited budgets and so all operate under the idea that more is better. That isn’t gonna be “the answer”… they are all gonna get instantly blindsided by some group who does more with significantly less. These small scrappy models and the institutes and companies behind them will eventually replace the old guard. It’s a tale as old as time.

clayhacks 2y ago
Deepseek just released their frontier model that they trained on 2k GPUs for <$6M. Way cheaper than a lot of the big labs. If the big labs can replicate some of their optimisations we might see some big gains. And I would hope more small labs could then even further shrink the footprint and costs
- cruffle_duffle 2y ago
  I don’t think this stuff will be truly revolutionary until I can train it at home or perhaps as a group (SETI at home anybody?) Six million is a start but this tech won’t truly be democratized until it costs $1000. Obviously I’m being a little cheeky but my real point is… the idea that this technology is in the control of massive technology companies is dystopian as fuck. Where is the RMS of the LLM space? Who is shouting from every rooftop how dangerous it is to grant so much power and control over information to a handful of massive tech companies, all whom have long histories of caving into various government demands. It’s scary as fuck.
  - lodovic 2y ago
    This is just a tech race. we'll get affordable 64 gb gpus in a few years, businesses want to run their own models.

clayhacks 2y ago

Deepseek just released their frontier model that they trained on 2k GPUs for <$6M. Way cheaper than a lot of the big labs. If the big labs can replicate some of their optimisations we might see some big gains. And I would hope more small labs could then even further shrink the footprint and costs

cruffle_duffle 2y ago
I don’t think this stuff will be truly revolutionary until I can train it at home or perhaps as a group (SETI at home anybody?) Six million is a start but this tech won’t truly be democratized until it costs $1000. Obviously I’m being a little cheeky but my real point is… the idea that this technology is in the control of massive technology companies is dystopian as fuck. Where is the RMS of the LLM space? Who is shouting from every rooftop how dangerous it is to grant so much power and control over information to a handful of massive tech companies, all whom have long histories of caving into various government demands. It’s scary as fuck.
- lodovic 2y ago
  This is just a tech race. we'll get affordable 64 gb gpus in a few years, businesses want to run their own models.