I mean Feynman's argument really seems to show that it should hold for sufficiently large N. I think one could reasonably argue that it's a statistical accident

I mean Feynman's argument really seems to show that it should hold for sufficiently large N. I think one could reasonably argue that it's a statistical accident that FLT holds for N >= 3 as opposed to say, N >= 100. I think even Wiles' proof only works for N sufficiently large (at least N >= 5). Small N of course were handled by earlier results.