4 ms·
> Working in this space (however you want to say it) costs a lot of money. Not necessarily? OpenAI deliberately chooses to scale their models vertically, which
by smoldesu 3y ago
> Working in this space (however you want to say it) costs a lot of money.
Not necessarily? OpenAI deliberately chooses to scale their models vertically, which incumbents like Google and Meta had largely avoided. Their "big innovation" was GPT-4, an LLM of unprecedented scale that brute-forced it's way to the top. The Open Source 72b+ models of today generally give it a run for it's money, the party trick is over.
Nitpicking the definitions is important, because for OpenAI it's unclear what comes next. Consumers don't know what to expect, portions of the board appear to be in open rebellion, and our goal of AGI remains yet-undefined. I simply don't trust OpenAI (or Elon, for that matter) to do the right thing here.
- qarl 3y ago> Not necessarily? Well then, I expect we'll see a bunch of small projects that beat the big players any day now. I won't hold my breath.
- edgyquant 3y agoAgain you haven’t a clue of the point here. They were created as a research lab and instead after taking a paper created at Google they went closed source and are simply scaling that instead of working to create new and improved models that can actually run on reasonable hardware. Small transformers being able to beat the same models but scaled up is unrelated to anything being discussed and you just seem like a fanboy at this point
- smoldesu 3y agoYou're basically describing what happened to GPT-2 when T5-flan came out. Not to mention, the incumbent model at the time (BERT) was extremely competitive for it's smaller size. Pretty much nobody was talking about OpenAI's models back then because they were too large to be realistically useful. So yeah, I do actually anticipate smaller projects that devalue turtle-tower model scaling.
- qarl 3y agoCool. Send me a note when that happens.
- deleted 3y ago[deleted]