I don't think you actually demonstrate this statement: > if the line between "strong/less strong" candidates are so blurred It is hard to tell prospectively w

I don't think you actually demonstrate this statement: > if the line between "strong/less strong" candidates are so blurred It is hard to tell prospectively whether someone will be a good programmer. It is not too difficult to tell retrospectively. (Any research that tries to measure competence/success indicators must be using something as its dependent variable, right?) The line you quote is specifically about the fact that resumes could be meaningful even if her competence assignments were not. Resumes seem to fail at that test as well. The biggest threat to validity would probably be that different people will be good/bad developers in different environments, and that this effect is very strong. Then each reviewer could be good at selecting for success in their own company, but they would only agree with each or with aline to the extent that their environment was similar to that of other companies or the ones that aline placed those engineers at. I am pretty comfortable rejecting this theory out of hand myself.