I think you've got it backwards. Lots of people who could solve your real world problems with a computer might not have a background in maths or physics and so

I think you've got it backwards. Lots of people who could solve your real world problems with a computer might not have a background in maths or physics and so may not be able to solve your puzzles. They may just feel anxious about appearing stupid in front an interviewer, and annoyed they weren't allowed any downtime in the day. This is about empathetically understanding the stress levels between both of you would have been miles apart. Unless the candidate had brought up the subject somehow it'd have been nicer to just try to relax them and get to know them informally by asking about them - or better yet letting them do their own thing for lunch if they wanted.