This strikes me as a really weird hypothesis. The idea of some over-arching "fitness factor" existing outside of the huge variety of genetic traits that apply

This strikes me as a really weird hypothesis. The idea of some over-arching "fitness factor" existing outside of the huge variety of genetic traits that apply to aspects of physical and mental ability sounds like comic-book science.