Why are you being disingenuous? Simply having access to the eval in question is already enough for your synthetics guys to match the distribution, and of course

Why are you being disingenuous? Simply having access to the eval in question is already enough for your synthetics guys to match the distribution, and of course you don't contaminate directly on train, that would be stupid, and you would get caught, but if it does inform the reward, the result is the same. You _should_ quit, but you wouldn't because you'd already convinced yourself you're doing RL God's work, not sleight of hand. > If an MMLU question asks about an abstract algebra proof, is it cheating to have trained on papers about abstract algebra? This kind of disingenuous bullshit is exactly why people call you cheaters. > Generally, I don’t think anyone here is cheating and I think we’re relatively diligent with our evals. You guys should follow Apple's cult guidelines: never stand out. Think different