> Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to

> Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/ It's because people do keep asking these models math problems and then, when they get them right, citing it as evidence that they can actually do mathematical reasoning. Since it's hard to determine what the models know, it's hard to determine when they're just spitting out something they were specifically trained on.