3 ms·

> Not much explanation yet why GPT-5 warrants a major version bump Exactly. Too many videos - too little real data / benchmarks on the page. Will wait for vibe

> Not much explanation yet why GPT-5 warrants a major version bump Exactly. Too many videos - too little real data / benchmarks on the page. Will wait for vibe check from simonw and others

collinmanderson 1y ago
> Will wait for vibe check from simonw https://openai.com/gpt-5/?video=1108156668 https://openai.com/gpt-5/?video=1108156668 2:40 "I do like how the pelican's feet are on the pedals." "That's a rare detail that most of the other models I've tried this on have missed." 4:12 "The bicycle was flawless." 5:30 Re generating documentation: "It nailed it. It gave me the exact information I needed. It gave me full architectural overview. It was clearly very good at consuming a quarter million tokens of rust." "My trust issues are beginning to fall away" Edit: ohh he has blog post now: https://news.ycombinator.com/item?id=44828264 https://news.ycombinator.com/item?id=44828264
- dimitri-vs 1y ago
  This effectively kills this benchmark.
  - tuesdaynight 1y ago
    Honestly, I have mixed feelings about him appearing there. His blog posts are a nice way to be updated about what's going on, and he deserves the recognition, but he's now part of their marketing content. I hope that doesn't make him afraid of speaking his mind when talking about OpenAI's models. I still trust his opinions, though.
    - croemer 1y ago
      Yeah, even if he wasn't paid to appear there, this seems a bit too close.
  - layer8 1y ago
    The pelican is still a mess.
- bardak 1y ago
  I feel like we need to move on from using the same test on models since as time goes on the information about these specific test is out there in the training data and while i am not saying that it's happened in this case there is nothing stopping model developers from adding extra data for theses tests directly in the training data to make their models seem better than they are
- laurent_du 1y ago
  Damn Theo is really a handsome dude.

> Not much explanation yet why GPT-5 warrants a major version bump Exactly. Too many videos - too little real data / benchmarks on the page. Will wait for vibe check from simonw and others

collinmanderson 1y ago

> Will wait for vibe check from simonw https://openai.com/gpt-5/?video=1108156668 https://openai.com/gpt-5/?video=1108156668 2:40 "I do like how the pelican's feet are on the pedals." "That's a rare detail that most of the other models I've tried this on have missed." 4:12 "The bicycle was flawless." 5:30 Re generating documentation: "It nailed it. It gave me the exact information I needed. It gave me full architectural overview. It was clearly very good at consuming a quarter million tokens of rust." "My trust issues are beginning to fall away" Edit: ohh he has blog post now: https://news.ycombinator.com/item?id=44828264 https://news.ycombinator.com/item?id=44828264

dimitri-vs 1y ago
This effectively kills this benchmark.
- tuesdaynight 1y ago
  Honestly, I have mixed feelings about him appearing there. His blog posts are a nice way to be updated about what's going on, and he deserves the recognition, but he's now part of their marketing content. I hope that doesn't make him afraid of speaking his mind when talking about OpenAI's models. I still trust his opinions, though.
  - croemer 1y ago
    Yeah, even if he wasn't paid to appear there, this seems a bit too close.
- layer8 1y ago
  The pelican is still a mess.
bardak 1y ago
I feel like we need to move on from using the same test on models since as time goes on the information about these specific test is out there in the training data and while i am not saying that it's happened in this case there is nothing stopping model developers from adding extra data for theses tests directly in the training data to make their models seem better than they are
laurent_du 1y ago
Damn Theo is really a handsome dude.