The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are r

The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are running an experiment and using competing hypothesis...in theory, you can identify which metrics are best for that test... but if you had much more data, you'll probably end up using every data point past the first 5 as confirmation bias. (I'm no expert though and haven't read the book in about 3 years.)