One of the theory on face recognition is that we learn to recognized people by "storing" the difference between an average face and the people we are looking at

One of the theory on face recognition is that we learn to recognized people by "storing" the difference between an average face and the people we are looking at. And we build that average face from the sum of all people we met. https://en.wikipedia.org/wiki/Cross-race_effect https://en.wikipedia.org/wiki/Cross-race_effect So indeed the different training set has a big impact.