In general I agree. For any gene, you'd expect those alleles which give their holder an advantage in the environment they inhabit to be passed on to their des

In general I agree. For any gene, you'd expect those alleles which give their holder an advantage in the environment they inhabit to be passed on to their descendants, including those which influence brain development. But if their environment ever changes, or they're put into a different environment, they might find themselves at a disadvantage compared with those native to their new environment. Environment is important as it determines which traits and which genetic variants are selected for, and genetics is important as it determines which environment people will flourish in. So if you only use a single number to measure intelligence, it it will only be a good predictor for a single environment.