5 ms·
> never seen before frontiermath problems How do you know that?
by fldskfjdslkfj 2y ago
> never seen before frontiermath problems
How do you know that?
- whimsicalism 2y agoBecause that is the whole conceit of how frontiermath is constructed
- fldskfjdslkfj 2y agoDidn't they run a bunch of models on the problem set? I doubt they are hosting all those models on their own infrastructure.
- whimsicalism 2y ago1. OpenAI has confirmed it’s not in their train (unlike putnam where they have never made any such claims) 2. They don't train on API calls 3. It is funny to me that HN finds it easier to believe theories about stealing data from APIs rather than an improvement in capabilities. It would be nice if symmetric scrutiny were applied to optimistic and pessimistic claims about LLMs, but I certainly don’t feel that is the case here.
- fldskfjdslkfj 2y agoEasier to believe or not, thinking that it's not a reasonable possibility is also funny.
- whimsicalism 2y agoDo you also think they somehow stole the codeforces problems before they were even written or you are willing to believe the #175 global rank there?
- fldskfjdslkfj 2y agoI dont think codeforce claims to contain novel unpublished problems. But i'm not saying it's what they did, just that it's a possibility that should be considered till/if it is debunked.
- whimsicalism 2y agofrankly i’m not sure what standard you would possibly consider a debunking codeforces constantly adds new problems that’s like the entire point of the contest, no?
- sudosysgen 2y agoOpenAI never actually directly participated in the contests. OpenAI says they took (unspecified) problems and "simulated" the score it would have had. If they solved recent contests in a realistic contest simulation I would expect them to give the actual solutions and success rates as well, like they did for IOI problems, so I'm actually confused as to why they didn't.
- whimsicalism 2y agovery good clarification, thanks - they should absolutely release more details to provide more clarity and ideally just participate live? i suspect that the model takes a while for individual problems so time might be a constraint there
- sudosysgen 2y agoYes, it would also probably be rate limited or timed out since they're doing 50+ submissions per problem
- jcranmer 2y agoThe modern state of training is to try to use everything they can get their hands on. Even if there are privileged channels that are guaranteed not to be used as training data, mentioning the problems on ancillary channels (say emailing another colleague to discuss the problem) can still create a risk of leakage because nobody making the decision to include the data is aware that stuff that should be excluded is in that data set. And as we've seen from decades of cybersecurity, people are absolute shit at the necessary operational security to avoid mentioning stuff on ancillary channels! Given that performance is known to drop considerably on these kinds of tests when novel problems are tried, and given the ease with which these problems could leak into the training set somehow, it's not unreasonable to be suspicious of a sudden jump in performance as merely a sign that the problems made it into the training set rather than being true performance improvements in LLMs.
- whimsicalism 2y agoOkay, then what about elite level codeforces performance? Those problems weren’t even constructed until after the model was made. The real problem with all of these theories is most of these benchmarks were constructed after their training dataset cutoff points. A sudden performance improvement on a new model release is not suspicious. Any model release that is much better than a previous one is going to be a “sudden jump in performance.” Also, OpenAI is not reading your emails - certainly not with a less than one month lead time.
- ImPostingOnHN 2y agoCan you give an example of one of these problems that 'wasn't even constructed until after the model was made'? I'd like to see if it's truly novel and unique, the first problem of its type ever construed by mankind, or if it's similar to existing problems.
- whimsicalism 2y agoSorry, I thought the whole point of this thread was that models can’t handle problems when they are “slightly varied”. Mottes and baileys all over the place today.
- s1mplicissimus 2y ago> 1. OpenAI has confirmed it’s not in their train (unlike putnam where they have never made any such claims) Companies claim lots of things when it's in their best financial interest to spread that message. Unfortunately history has shown that in public communications, financial interest almost always trumps truth (pick whichever $gate you are aware of for convenience, i'll go with Dieselgate for a specific example). > It is funny to me that HN finds it easier to believe theories about stealing data from APIs rather than an improvement in capabilities. It would be nice if symmetric scrutiny were applied to optimistic and pessimistic claims about LLMs, but I certainly don’t feel that is the case here. What I see is generic unsubstantiated claims of artificial intelligence on one side and specific, reproducible examples that dismantle that claim on the other. I wonder how your epistemology works that leads you to accept marketing claims without evidence
- ttul 2y agoOpenAI’s credibility is central to its business: overstating capabilities risks public blowback, loss of trust, and regulatory scrutiny. As a result, it is unlikely that OpenAI would knowingly lie about its models. They have much stronger incentives to be as accurate as possible—maintaining their reputation and trust from users, researchers, and investors—than to overstate capabilities for a short-term gain that would undermine their long-term position. From a game-theoretic standpoint, repeated interactions with the public (research community, regulators, and customers) create strong disincentives for OpenAI to lie. In a single-shot scenario, overstating model performance might yield short-term gains—heightened buzz or investment—but repeated play changes the calculus: 1. Reputation as “collateral” OpenAI’s future deals, collaborations, and community acceptance rely on maintaining credibility. In a repeated game, players who defect (by lying) face future punishment: loss of trust, diminished legitimacy, and skepticism of future claims. 2. Long-term payoff maximization If OpenAI is caught making inflated claims, the fallout undermines the brand and reduces willingness to engage in future transactions. Therefore, even if there is a short-term payoff, the long-term expected value of accuracy trumps the momentary benefit of deceit. 3. Strong incentives for verification Independent researchers, open-source projects, and competitor labs can test or replicate claims. The availability of external scrutiny acts as a built-in enforcement mechanism, making dishonest “moves” too risky. Thus, within the repeated game framework, OpenAI maximizes its overall returns by preserving its credibility rather than lying about capabilities for a short-lived advantage.
- Spooky23 2y agoFrankly you need to read what they say explicitly and not infer what they mean by your reckoning. They are the system to beat and their competitors are either too small or too risk averse. They ingest millions of data sources. Among them is the training data needed to answer the benchmark questions.
- cma 2y agoThe gpt 4 paper said they only excluded benchmarks by exact text matches. That means discussion of them probably doesn't get excluded.
- rrr_oh_man 2y ago> Because that is the whole conceit Freudian typo?