The obvious answer is (2) don't run a study at all (assuming the methodology is truly flawed). Or (3) run the study but take the results with a grain of salt an

The obvious answer is (2) don't run a study at all (assuming the methodology is truly flawed). Or (3) run the study but take the results with a grain of salt and don't draw broad conclusions from it. And (4) we can comment on hacker news about how "wouldn't it be nice if they tested this or that because that would have convinced me, personally." I think that we're kind of saying the same thing. It's an thought-provoking study and I agree that it is a useful step. But I also agree that further research is required to really draw a conclusion.