5 ms·

The whole thing is written from the assumption that investors discriminate against women and nonwhites, and that the purpose of research is to confirm this pre-

by sltkr 6y ago

The whole thing is written from the assumption that investors discriminate against women and nonwhites, and that the purpose of research is to confirm this pre-existing belief, rather than find out the objective truth. > [..] future research on investor biases should focus on networks and in-person interactions. Read: the results contradicted our hypothesis, so rather than update our hypothesis, let's shift the focus of research in hopes that we find evidence to support it.

maccard 6y ago
This is exactly how science should work. You can't perform a study, look at the data and pick a result that suits your bias. You're supposed to suggest a hypothesis, and conduct your experiment. If the results contradict your hypothesis, it doesn't mean the opposite of your hypothesis is true, simply that your hypothrsis was false, and that's ok. It's also more impressive that they have simply published their results, rather than burying them and running another experiment.
- mc32 6y ago
  So what if they had found bias, do they say “look we found bias!” Or do they do additional studies to make sure it’s not an aberration or improper methods?
  - sokoloff 6y ago
    It seems ideal if they did both of those things. Publish and make recommendations for future research, exactly as this paper does.
  - maccard 6y ago
    Both. They publish and say "yes, we found bias on X, and our next study will verify via Y"
- bluecalm 6y ago
  So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it? "Have you seen the fire sir?" "I have only seen smoke" "But have you seen the fire?" "It's irrelevant because I was only looking for smoke!". I mean I know they teach it in naive introductory experiment design in some soft science departments but c'mon it's neither smart nor correct. If you need a separate experiments for everything because otherwise you will run into "one of the 100 hypothesis is bound to be correct" problem then your priors are too weak, you're relying on getting lucky anyway and just end up discarding useful datapoints. If it's 5 out of 100 hypothesis fitting your data or 5 hypothesis out of 100 experiments passing "statistical significance" check doesn't matter at the end of the day. Both lead to shaky conclusions. The data being relevant depends on the experiment design (how it's collected). It doesn't depend on what you thought it's relevant for beforehand.
  - sokoloff 6y ago
    The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial function fits the data quite closely!” You can’t look at your A/B test results every morning and stop the test the moment one measurement “shows significance”. That’s not how it works.
    - marcosdumay 6y ago
      That's why the GP was taking about weak priors. You just don't run an A/B test for 6 months hoping that a new design increases sales, get results pointing that the new design reduces sales by 90%, and then go make another 6 months long test to make sure the sales indeed fell. Acknowledging the numbers on that single dimension are different from what you expected isn't the same as testing a bunch of different dimensions. And if your hypothesis does not allow for it, you completely failed at rational thinking.
  - renewiltord 6y ago
    Well, you can test multiple hypotheses at the same time. You just can't decide what the hypotheses are after you collect the data because you can't reasonably determine the correction factor.
- tgv 6y ago
  I also think that the average reader of such an article can make the inference themselves.
- throwaway894345 6y ago
  > If the results contradict your hypothesis, it doesn't mean the opposite of your hypothesis is true, simply that your hypothrsis was false, and that's ok. I don’t think the parent was arguing that the paper should claim the opposite is true, but perhaps recognizing that the findings support the opposite hypothesis, concretely that men and non-Asians are discriminated against in early stage investment pitches. Of course, a perfectly valid response might be, “that is unconventional in these kinds of papers” (I don’t know).

by sltkr 6y ago

maccard 6y ago

This is exactly how science should work. You can't perform a study, look at the data and pick a result that suits your bias. You're supposed to suggest a hypothesis, and conduct your experiment. If the results contradict your hypothesis, it doesn't mean the opposite of your hypothesis is true, simply that your hypothrsis was false, and that's ok. It's also more impressive that they have simply published their results, rather than burying them and running another experiment.

mc32 6y ago
So what if they had found bias, do they say “look we found bias!” Or do they do additional studies to make sure it’s not an aberration or improper methods?
- sokoloff 6y ago
  It seems ideal if they did both of those things. Publish and make recommendations for future research, exactly as this paper does.
- maccard 6y ago
  Both. They publish and say "yes, we found bias on X, and our next study will verify via Y"
bluecalm 6y ago
So you think the data is only relevant to the hypothesis you decided it's relevant for before you collect it? "Have you seen the fire sir?" "I have only seen smoke" "But have you seen the fire?" "It's irrelevant because I was only looking for smoke!". I mean I know they teach it in naive introductory experiment design in some soft science departments but c'mon it's neither smart nor correct. If you need a separate experiments for everything because otherwise you will run into "one of the 100 hypothesis is bound to be correct" problem then your priors are too weak, you're relying on getting lucky anyway and just end up discarding useful datapoints. If it's 5 out of 100 hypothesis fitting your data or 5 hypothesis out of 100 experiments passing "statistical significance" check doesn't matter at the end of the day. Both lead to shaky conclusions. The data being relevant depends on the experiment design (how it's collected). It doesn't depend on what you thought it's relevant for beforehand.
- sokoloff 6y ago
  The reason you don’t collect some data and then see what curves you can fit to it is exactly the problem you describe. “Eureka, a ninth-order polynomial function fits the data quite closely!” You can’t look at your A/B test results every morning and stop the test the moment one measurement “shows significance”. That’s not how it works.
  - marcosdumay 6y ago
    That's why the GP was taking about weak priors. You just don't run an A/B test for 6 months hoping that a new design increases sales, get results pointing that the new design reduces sales by 90%, and then go make another 6 months long test to make sure the sales indeed fell. Acknowledging the numbers on that single dimension are different from what you expected isn't the same as testing a bunch of different dimensions. And if your hypothesis does not allow for it, you completely failed at rational thinking.
- renewiltord 6y ago
  Well, you can test multiple hypotheses at the same time. You just can't decide what the hypotheses are after you collect the data because you can't reasonably determine the correction factor.
tgv 6y ago
I also think that the average reader of such an article can make the inference themselves.
throwaway894345 6y ago
> If the results contradict your hypothesis, it doesn't mean the opposite of your hypothesis is true, simply that your hypothrsis was false, and that's ok. I don’t think the parent was arguing that the paper should claim the opposite is true, but perhaps recognizing that the findings support the opposite hypothesis, concretely that men and non-Asians are discriminated against in early stage investment pitches. Of course, a perfectly valid response might be, “that is unconventional in these kinds of papers” (I don’t know).