I've always thought that surveys of team members and managers could be a pretty cheap and effective way to do this (in a research setting, where you aren't basi

I've always thought that surveys of team members and managers could be a pretty cheap and effective way to do this (in a research setting, where you aren't basing pay or contract renewal on it). It may not give you much in an absolute sense, but should give you a good idea of the relative ranking of programmers. Of course software engineering research in general has always been a little unimpressive because what you really need is several teams completing projects, some using method A and others using method B. This could cost millions, and would only settle the bet between methods A and B. Lacking this, we're left with the publically available research that relies on unconvincing proxies for success and very small samples. It's better than nothing but far short of what we ought to have given how important software is.