5 ms·

I don't see a better system. Maybe I'm misunderstanding his or your point. To me this is essentially capitalism. 90% or more of businesses fail, and of those th

I don't see a better system. Maybe I'm misunderstanding his or your point. To me this is essentially capitalism. 90% or more of businesses fail, and of those that don't, 90% or more suck. But if you tried to call that 99% "waste" that could be avoided by only running the good business, you would fail miserably. Research is exactly the same. We can't just identify and fund the good research. (But this should be obvious to a VC so I may be reading it wrong)

yuy910616 5y ago
Well said - Perhaps a way to look at it is that 'good' is a subjective target that should be define as 'distance from average/medium'. So let's be thankful of the crappy medium haha
jollybean 5y ago
This is definitely not the right analogy because one is a matter of risk, the other a matter of integrity. New business ventures are inherently risky. Research is inherently risky, and we would expect 90% to 'fail' but for 90% of papers to be rubbish would be a matter of 'extreme corruption' along the lines of 90% of judges taking bribes etc.. "I don't see a better system." Well, one thing we could do is punish people for knowingly publishing crap results, i.e. have a 'secondary review system' that investigates 'why' results were rubbish, and if there was malfeasance, well, put in place some kind of retribution. We could also change our tune with respect to expectations, and come to 'expect' that the majority of research should publish then 'null' result and that should just be it.
- yuy910616 5y ago
  I think this is a good point too - venture companies and research SHOULD have two very different distributions
- nostrademons 5y ago
  Be careful about assuming malice where ignorance exists. It's possible for 90% of papers to be rubbish without any ill intent or dishonesty on the part of the experimenter. They might simply suffer from "works for me" syndrome, as it afflicts software developers. In science, this equates to failing to control for all the possible confounding factors that might affect the result of the experiment. They might have carefully controlled for everything they thought of and gotten methodologically valid results, but then when another experimenter tries to replicate it, they find their experimental environment is different in a way that neither considered important but alters the results. Finding all these conflaters is the point of science. If you can't replicate the experiment, it means the results don't hold, but it doesn't mean the investigators were bad people - and assuming they are is unlikely to get better science.
JacobDotVI 5y ago
Research in general is the same, but shouldn’t _published_ research have already gone through a filter such that it represents the 10% that hasn’t failed?
- nostrademons 5y ago
  Only if you tie publication to reproduction. But you can't really do that, because to reproduce an experiment independently, the independent researchers need to have learned about your result and methodology, which = publication. Without being able to independently verify results, a lot of journals rely on signals like your research & results being noteworthy, which is anticorrelated with them being true. If our understanding of the world is at all correct, you'd expect most experiments to give boring, status-quo results, and your first impulse when you get something noteworthy is to assume the experimenter made a mistake. (Interestingly, this is how things work at the undergrad level - if you measure the speed of light at 1 million meters/sec, your prof will assume you screwed up.) Probably we need a separate verification step after publication, with researchers evaluated on what fraction of their published results survive verification, not just how many papers they publish. Publish all the noteworthy results you get, but assume that published research doesn't mean much until it's been independently reproduced.

yuy910616 5y ago

Well said - Perhaps a way to look at it is that 'good' is a subjective target that should be define as 'distance from average/medium'. So let's be thankful of the crappy medium haha

jollybean 5y ago

This is definitely not the right analogy because one is a matter of risk, the other a matter of integrity. New business ventures are inherently risky. Research is inherently risky, and we would expect 90% to 'fail' but for 90% of papers to be rubbish would be a matter of 'extreme corruption' along the lines of 90% of judges taking bribes etc.. "I don't see a better system." Well, one thing we could do is punish people for knowingly publishing crap results, i.e. have a 'secondary review system' that investigates 'why' results were rubbish, and if there was malfeasance, well, put in place some kind of retribution. We could also change our tune with respect to expectations, and come to 'expect' that the majority of research should publish then 'null' result and that should just be it.

yuy910616 5y ago
I think this is a good point too - venture companies and research SHOULD have two very different distributions
nostrademons 5y ago
Be careful about assuming malice where ignorance exists. It's possible for 90% of papers to be rubbish without any ill intent or dishonesty on the part of the experimenter. They might simply suffer from "works for me" syndrome, as it afflicts software developers. In science, this equates to failing to control for all the possible confounding factors that might affect the result of the experiment. They might have carefully controlled for everything they thought of and gotten methodologically valid results, but then when another experimenter tries to replicate it, they find their experimental environment is different in a way that neither considered important but alters the results. Finding all these conflaters is the point of science. If you can't replicate the experiment, it means the results don't hold, but it doesn't mean the investigators were bad people - and assuming they are is unlikely to get better science.

JacobDotVI 5y ago

Research in general is the same, but shouldn’t _published_ research have already gone through a filter such that it represents the 10% that hasn’t failed?

nostrademons 5y ago
Only if you tie publication to reproduction. But you can't really do that, because to reproduce an experiment independently, the independent researchers need to have learned about your result and methodology, which = publication. Without being able to independently verify results, a lot of journals rely on signals like your research & results being noteworthy, which is anticorrelated with them being true. If our understanding of the world is at all correct, you'd expect most experiments to give boring, status-quo results, and your first impulse when you get something noteworthy is to assume the experimenter made a mistake. (Interestingly, this is how things work at the undergrad level - if you measure the speed of light at 1 million meters/sec, your prof will assume you screwed up.) Probably we need a separate verification step after publication, with researchers evaluated on what fraction of their published results survive verification, not just how many papers they publish. Publish all the noteworthy results you get, but assume that published research doesn't mean much until it's been independently reproduced.