3 ms·
Could all pop today if GPT5 doesn’t benchmark hack hard on some new made up task.
by m_ke 1y ago
Could all pop today if GPT5 doesn’t benchmark hack hard on some new made up task.
- mewpmewp2 1y agoI don't expect GPT-5 to be anything special, it seems OpenAI hasn't been able to keep its lead, but even current level of LLMs to me justifies the market valuations. Of course I might eat my words saying that OpenAI is behind, but we'll see.
- apwell23 1y ago> I don't expect GPT-5 to be anything special because ?
- input_sh 1y agoBecause everything past GPT 3.5 has been pretty unremarkable? Doubt anyone in the world would be able to tell a difference in a blind test between 4.0, 4o, 4.5 and 4.1.
- apwell23 1y ago> Doubt anyone in the world would be able to tell a difference in a blind test between 4.0, 4o, 4.5 and 4.1. But this isn't 4.6 . its 5. I can tell difference between 3 and 4.
- dwater 1y agoThat's a very Spinal Tap argument for why it will be more than just an incremental improvement.
- falcor84 1y agoI would absolutely take you on a blind test between 4.0 and 4.5 - the improvement is significant. And while I do want your money, we can just look at LMArena which does blind testing to arrive at an ELO-based score and shows 4.0 to have a score of 1318 while 4.5 has a 1438 - it's over twice likely to be judged better on an arbitrary prompt, and the difference is more significant on coding and reasoning tasks.
- Workaccount2 1y agoWell word on the street is that the OSS models released this week were Meta-Style benchmaxxed and their real world performance is incredibly underwhelming.
- falcor84 1y agoI don't see how it would "all pop" - same as with the internet bubble, even if the massive valuations disappear, it seems clear to me that the technology is already massively disruptive and will continue growing its impact on the economy even if we never reach AGI.
- m_ke 1y agoExactly like the internet bubble. I've been working in Deep Learning since 2014 and am very bullish on the technology but the trillions of dollars required for the next round of scaling will not be there if GPT-5 is not on the exponential growth curve that sama has been painting for the last few years. Just like the dot com bubble we'll need to wash out a ton of "unicorn" companies selling $1s for $0.50 before we see the long term gains.
- falcor84 1y ago> Exactly like the internet bubble. So is this just about a bit of investor money lost? Because the internet obviously didn't decline at all after 2000, and even the investors who lost a lot but stayed in the game likely recouped their money relatively quickly. As I see it, the lesson from the dot-com bust is that we should stay in the game. And as for GPT-5 being on the exponential growth curve - according to METR, it's well above it: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
- 0xdde 1y agoI wouldn't say "well above" when the curve falls well within the error bars. I wonder how different the plot would look if they reported the median as their point estimate rather than mean.