3 ms·
Evidence that LLMs are reaching a point of diminishing returns
- hprotagonist 3y agoNature loves a sigmoid.
- kromem 3y agoDoes anyone have a link to a prediction Gary Marcus made more than 24 months ago that turned out correct?
- cratermoon 3y agoDoes anyone have a link to a prediction Sam Altman made more than 24 months ago that turned out correct?
- kromem 3y ago> In the next five years, computer programs that can think will read legal documents and give medical advice. - https://moores.samaltman.com/ https://moores.samaltman.com/ (March 2021)
- AshamedCaptain 3y agoWow that is pure genious, he didn't say anything about the advice being wrong, or the legal documents comprising fictional cases.
- iLoveOncall 3y agoWebMD exists since 1996, it's easy to predict something that already exists.
- kromem 3y agoThe prediction wasn't the existence of the product, but what was feeding into the content of the product (WebMD doesn't exist spontaneously). Also relevant: > Six patients 65 years or older (2 women and 4 men) were included in the analysis. The accuracy of the primary diagnoses made by GPT-4, clinicians, and Isabel DDx Companion was 4 of 6 patients (66.7%), 2 of 6 patients (33.3%), and 0 patients, respectively. If including differential diagnoses, the accuracy was 5 of 6 (83.3%) for GPT-4, 3 of 6 (50.0%) for clinicians, and 2 of 6 (33.3%) for Isabel DDx Companion (Table). - Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/)
- pyb 3y agoThey can do this, badly. Is it a successful prediction?
- cratermoon 3y agoWell, we don't have any computers programs that can think, so I'd call that a miss.
- andy99 3y agoBest comment I've seen about this: https://news.ycombinator.com/item?id=39627249 https://news.ycombinator.com/item?id=39627249
- kromem 3y agoI was genuinely asking, and not getting any answers I decided to research it for myself. I didn't find anything particularly compelling, but I did dig up this gem of an article about how deep learning was hitting a wall and "we are still a long way from machines that can genuinely understand human language" exactly four days before GPT-4 released: https://nautil.us/deep-learning-is-hitting-a-wall-238440/ https://nautil.us/deep-learning-is-hitting-a-wall-238440/
- notamy 3y ago> Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns. Whatever doubling there might have been, has perhaps come to an end. Maybe I’m misunderstanding how OpenAI names their models, but was there supposed to be a huge jump from GPT-4 to GPT-4 Turbo? Just from the names it sounds like it’s expected to be in the same realm of model performance, just faster.
- buildbot 3y agoYour understanding is correct, Gary Marcus is intentionally misrepresenting what GPT-4 turbo is to make a story. It’s clearly the cost optimized version of gpt-4. That’s why it’s much cheaper per token.
- timrobinson333 3y agoI think the point made in the article is that they wanted to build gpt 5 but they renamed it to 4 turbo because the improvements over 4 weren't what they were expecting
- notamy 3y agoRight, but... is there any actual proof of that? I didn’t see any in the article.
- aduffy 3y agoFrom October: https://www.theinformation.com/articles/openai-dropped-work-on-new-arrakis-ai-model-in-rare-setback https://www.theinformation.com/articles/openai-dropped-work-...
- buildbot 3y agoIf you read the article, the central claim is that they failed at making a more efficient model, not a larger, better, GPT-5 model.
- thorum 3y ago> And here’s the thing – we all know that GPT-3 was vastly better than GPT-2. And we all know that GPT-4 (released thirteen months ago) was vastly better than GPT-3. But what has happened since? Other models have gradually caught up to GPT-4 while OpenAI has declined to discuss or release anything regarding their internal research and work on GPT-5. Given how far ahead they were with GPT-4, it seems odd to assume no further progress has been made and just not published yet. The author’s assumption is based on the claim that GPT-4 Turbo “was a failed attempt at GPT-5” but I don’t see any good reason to believe this unless you take the claim that GPT-5 is impossible as a given and reason backwards.
- kromem 3y agoAlso, part of the issue with the argument is that smaller firms like Anthropic now have models that exceed OpenAI's offering. If we were actually at a point of diminishing returns, they'd only be catching up with similar levels of investment, not bypassing at lower levels. And when digging into the details of the differences between something like Opus and GPT-4, it becomes more readily apparent why OpenAI's product line up is struggling. Their current issues (to the extent they exist in practice) are related to OpenAI specific policies in how they are approaching developing their models in the fine tuning stages, not to inherent limitations in the underlying tech.
- ShamelessC 3y agoOh wow so the game Connections is apparently a robust indicator of LLM capabilities? Give me a break. The thesis here may be correct but Gary Marcus’ hasn’t effectively argued that it is. Is there even a mention of the scaling law’s paper?
- Etheryte 3y agoIs it just me or does this look like an advanced form of blogspam? The author seems to take one idea and just throw out a dozen variations of it over a few days to see if anything sticks. A few titles from the past few days: - Evidence that LLMs are reaching a point of diminishing returns — and what that might mean - Superhuman AGI is not nigh - $10 million says we won’t see human-superior AGI by the end of 2025 Etc, you get the idea.
- buildbot 3y agoUnfortunately this specific blog spammer has convinced many a person they are the One True God of the correct path to AGI.
- Hnaomyiph 3y agoOroborus issues were seen from 100k miles away. LLMs were bound to run into issues immediately based on training data. If I weren’t substantially poor I’d put all my savings into puts of LL ‘S like ChatGPT, and others. I cannot risk burning substantials of my own time to offset the burnage those themselves are creating. Good luck y’all. I’d say I’m hope I’m right but you’re on a ship sunk by glacial ice that is hoping for a lifeboat off. Good fucking luck, you have no time to pivot.
- phillipcarter 3y agoThat article sure was a lot of words for "the rest of the market is catching up to OpenAI". The rest is just a negative critical light on some of the woes of early adopters who likely aren't using the tech effectively. That's kind of problematic because (a) it doesn't emphasize how the tech takes skill and iteration to use right, and (b) it doesn't balance that out with examples of organizations who have seen immediate value from this tech.
- TheAceOfHearts 3y agoI thought this was to be expected. If you invest 1 billion USD in training a new model using existing tools and techniques it probably won't be as big of a leap in capabilities as previous generations. In particular, aren't we running out of content with which to train these models? There's obviously some key details that are still missing in the AI space precluding us from achieving things like "can learn how to drive in a couple hours". I don't want to pile on the hate, but this creator has a fairly negative reputation on Twitter. He comes off as a bit of an asshole on social media sometimes. Making it difficult to disentangle the author from the topics he brings up when the author is so inflammatory.
- belter 3y agoPeople read the guidelines...Flagging is not for stuff you disagree with...
- deleted 3y ago[deleted]