my guess is that the brain does something like PCA, but then over-emphasizes the unusual components (probably because that's what makes each face distinct). in

my guess is that the brain does something like PCA, but then over-emphasizes the unusual components (probably because that's what makes each face distinct). in other words, the brain (either hard-wired or learnt) has an "average" face that it uses as a reference. when it has to handle a new face it doesn't "store" the whole face, but only the difference between the new and the average. so what much of your brain deals with is not "the face" but "differences of that particular face from the average". and somehow (see other replies) this illusion is able to connect that up to your conscious mind without the intermediate step that pads it back out by "adding in" the average.