I'm currently pursuing PhD in theoretical condensed matter physics. I tried submitting questions to Humanity Last Exam [1], and it was not too hard to think of

I'm currently pursuing PhD in theoretical condensed matter physics. I tried submitting questions to Humanity Last Exam [1], and it was not too hard to think of a problem that none of top llms (Claude, gpt, Gemini + both o1 models) got right. What was surprising for me is how small my bag of tricks was. I could think of 5-6 questions in my direct area of expertise with simple numerical answer that were hard for llms, and another maybe 5 that they were able to solve. But basically that was all my expertise. Of course there is stuff that can't be checked with simple numerical answer (quite a lot in my case), and there are probably additional questions that would require more effort from me to give a correct answer. But all in all, I suddenly felt I'm a one-trick pony, and that's given that my PhD is relatively diverse. [1] https://agi.safe.ai/submit https://agi.safe.ai/submit