That isn't it, though. Feynman explains magnetism perfectly well in his lectures, he's just saying that he can't give a decent math-free answer in a few minutes

That isn't it, though. Feynman explains magnetism perfectly well in his lectures, he's just saying that he can't give a decent math-free answer in a few minutes to an interviewer with no science background. Any answer that short would not functionally be any better than "magnets work because of magic".