4 ms·
> I wonder how gpt-4 does. The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to
by TacticalCoder 4y ago
> I wonder how gpt-4 does.
The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to train the next versions and are going to be part of the dataset.
So GPT-4 or GPT-5 may get the answer right, but that still wouldn't mean anything.
- fastball 4y agoFairly certain that is not what is going on here. GPT-4 seems genuinely better at reasoning and harder to trick from my testing.
- M4v3R 4y agoNot the case for GPT-4 though, as it’s knowledge cutoff is the same as GPT-3s, that’s why it’s easy to compare the two on the same problems and see what’s the difference.
- versteegen 4y agoNot so fast! Yes, the GPT-4 paper says > GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021 [footnote: The pre-training and post-training data contain a small amount of more recent data], and does not learn from its experience. But note the more recent data. We know that InstructGPT (GPT-3.5) was RL trained on examples of previous queries to GPT3, such as those trick questions. We could assume everything (after filtering e.g. for benchmark contamination) ever sent to OpenAI is in that post-training set. This is indeed a very small amount of data compared to the trillion-plus tokens of older data it was surely trained on. We also know that when ARC did their evaluations of GPT-4, OpenAI hadn't finished fine-tuning yet, so they've certainly been continuing to do so recently. See also my other comment https://news.ycombinator.com/item?id=35300668 https://news.ycombinator.com/item?id=35300668
- ResearchCode 4y agoI suspect you're right. Part of the supervised "learning" is hard-coding answers to gotchas posted on Twitter.