"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that." I was recently list

"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that." I was recently listening to BBC Radio 4's episode on the Poincare Conjecture[1] and the guests on the program were discussing how the problem that looked deceptively simple eluded the great mathematicians of the time (including Poincare himself) for nearly a century and how Grigori Perelman cleverly came up with the proof. It took other mathematicians working in groups years after Perelman's publication to understand and validate his proof. The mathematicians on the program were speaking of highly of his proofs and admiring the originality of his work. This made me think of one neat experiment where if we cut-off a frontier model's training data 2002 or anytime before Perelman posted his proofs on arXiv and check if it can come up with the solution by itself. That would surely be a great signal to see if these LLMs aren't just solving interesting puzzles and that they can came up with something truly novel. P.S I highly recommend Misha Green's "Perfect Rigor" for anyone interested in the history of the problem and the genius behind the proofs of the conjecture - Perleman. I found it an entertaining read and could digest its description of the problem as a layperson (with undergrad level math). [1] https://www.bbc.co.uk/programmes/p0038x8l https://www.bbc.co.uk/programmes/p0038x8l