Methodological error: if the same people are rating both nerdiness and kickassery, they could be subtly influenced to make their ratings correlate (negative or

Methodological error: if the same people are rating both nerdiness and kickassery, they could be subtly influenced to make their ratings correlate (negative or positive). Better would be to randomly assign judges to rate either nerdiness or kickassery, not both. Also, it would be good to ask, rather than "how nerdy is this band", instead "which of these two bands is nerdier?" This is usually an easier answer (or, if it's really difficult, that tells you their values are close). It would also give you a hint on whether or not one's subjective measure of nerdiness (or kickassery) is even internally self-consistent. If you often get A > B, B > C, but C > A, then the whole rating exercise is based on a flawed premise. Clearly worth spending some hours making a web page to ask people these questions, then come back to HN and do a "Show HN" and ask for more data from the symphonic metal fans here.