6 ms·

The acceptance of bad scientific practice to the benefit of productivity

54 points by gbear605 8y ago

ccapo 8y ago
I wholeheartedly agree with this assessment. I think the culture is extremely toxic, and it pushes out many promising students and researchers.
raziel2701 8y ago
J.M. Ziman wrote in 1962: "It is typical of modern physicists that they will erect skyscrapers of theory upon the slender foundations of outrageously simplified models." I guess it is only a matter of time before the proverbial skyscraper collapses. I have had first hand experience with publishing papers of decreasing relevance. It's almost always due to professors being unwilling to give up their mental model of reality. I had one professor tell me he would not let me publish my result because it contradicted a paper he wrote in the late 70s. It was the grossest violation of the spirit and purpose of science I've encountered, and it quickly turned me off from the whole institution. Egos in academia are often unchecked.
- nerdponx 8y ago
  At least these physics models lead to testable predictions. You should see what goes on in economics.
  - AlexCoventry 8y ago
    Curious about why the parent comment is getting downvoted.
    - deleted 8y ago
      [deleted]
  - deleted 8y ago
    [deleted]
  - pythonslayer 8y ago
    Yeah it seems the academic posturing (in almost every discipline) is just insulation for the field of Economics. An MBA friend’s professor told him that economics is just like gravity. It’s not worth wasting his time on Earth thinking about it.
throwaway770 8y ago
Is there any advice onto how one deals with research misconduct in the industry? That allows to avoid expensive pitfalls like Theranos? Is leaving the place the best option? Simply leaving the place doesn't seem optimal. It feels like, it just leaves an opening for a recent H1 from a poor country, with a different tolerance for dishonesty. And the end-result is continuing misconduct.
- raziel2701 8y ago
  I think you should speak up with the expectation that whistleblowers don't do well, but with the knowledge that science corrects only if scientists correct it. It's unlikely a single person can take down the culture of an industry, but one person can certainly start a movement. You're definitely well justified in feeling torn about this. I agree that leaving in silence is not the best idea, but I certainly wouldn't blame you if you did that. It's hard to take action.
ianbicking 8y ago
"This problem cannot be solved by appointing non-experts to review panels – that merely creates incentives for research that’s easy to comprehend." That doesn't sound SO bad. Selection criteria seems like a major issue here, and the author only talks about the validity of the results. Why do we care about physics? Or psychology? There are answers to these simple questions, but I don't think the answers inform the selection criteria. We don't care about physics because academic researchers need to publish papers or get funding. Having any old non-expert on the panel could be problematic. Especially if they become political tools, and right now they'd definitely become political tools. But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics? These aren't hostile interlopers. But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.
- rossdavidh 8y ago
  Agreed. The article also said that having outsiders on the panel would just be an incentive to write easy to understand research proposals, and I was reminded of the idea that you need to understand something really well to be able to explain it clearly. If a physicist cannot explain their research proposal to an engineer or physicist, perhaps they need to go off and think about it some more.
  - archgoon 8y ago
    > perhaps they need to go off and think about it some more. Well, that's exactly what they're asking for funding for. The entire crux of the issue is that these are not complete ideas, and need to be developed further; which takes time, and therefore, money.
- addicted 8y ago
  You’re right. What we should also do is that all critical code that may significantly impact lives and people’s wealth and security should be written in a manner that can be understood by laymen. So functional programming is right out. Anything more complex than BASIC is probably not a solution. Probably only literate programming is allowed.
majos 8y ago
I like that this article includes suggested changes: > For starters, every scientist should know how being part of a group can affect their opinion. Grants should not be awarded based on popularity. Researchers who leave fields of declining promise need encouragement, not punishment because their productivity may dwindle while they retrain. And we should generally require scientists to name both advantages and shortcomings of their hypotheses. The first and last points look good to me. The middle two, not so much. They certainly sound reasonable, but what does a grant system not based on "popularity" mean? How is it different from what we have now, where we find some (putative) experts in the area, put it in front of them, and ask what they think? We need some way of deciding what to fund and what not to fund, and who exactly is better equipped to make those decisions than existing researchers? Similarly, I like the idea that people should be able to switch fields. But the implicit argument seems to be that the emphasis on publications makes this difficult, so publications should be de-emphasized in assessing researcher quality. But what is better? Asking a researcher's peers? Then it is a popularity contest again. Weighting publications differently, i.e. "what are your 5 best papers?" rather than "how many papers do you have?"? Then who judges the 5? For all the criticism heaped upon bibliometrics: they get used because assessing the quality of most research is difficult, so even noisy filters like conferences (in CS) or journals are taken as a useful signal. It's not clear to me what better options exist, especially when trying to choose between many options. It's just not possible for one person or even a small group of people to truly familiarize themselves with the bodies of work of all 100 candidates for a faculty position/grant/fellowship/etc.
- rossdavidh 8y ago
  Agreed, and I would add that suggestions 1 and 4, although fine, will not be enough to make a substantive difference. To be honest, Sen. Paul's suggestion (if implemented properly, i.e. add experts from other fields rather than political hacks) sounds more likely to help.
majos 8y ago
A graduating PhD student in theoretical physics from a Top 5 school said something similar to me last year. Roughly, he said it was very easy to slip into bullshitty fooling yourself when doing theoretical physics research, and it took vigilance from within to do truly good research, because external bodies were unlikely to really punish you as long as the math worked out.
- dwaltrip 8y ago
  > it took vigilance from within to do truly good research Well said. I've been mulling on the idea that a scientific mindset must include a deeply embedded skepticism that isn't afraid to analyze and question even the most pervasive and mundane aspects of one's own being. Of course, one has to get through the day and take care of life, in a practical fashion. But I think too often we assume our actions to be much more intentional than they actually are.
TangoTrotFox 8y ago
I think this article is looking at the effect rather than the cause. Why might it be that so much of science has started to edge away from well... science. Today nearly 1 in 8 people have a masters degree or higher. And in one in 33 have a doctorate or professional degree. [1] And these numbers seem to be increasing. That's really insane if you think about it. We now have more people with postgraduate degrees than had any degree in 1970, and postgraduate education generally implies a necessity to publish. But discovering genuinely new and novel information is not easy. So what do people publish instead? Enter the current state of science and academia. I think the most informative paragraph in this article was: "Or look at Brian Wansink, the Cornell Professor with the bottomless soup bowl experiment. He recently drew unwanted attention to himself with a blogpost in which he advised a student to try harder getting results out of data because it “cost us a lot of time and our own money to collect.” Had Wansink been aware that massaging data until it delivers is not sound statistical procedure, he’d probably not have blogged about it." Physics is one of the least susceptible to this issue, but no field is immune. The issue is that 'science' has become more about publishing than actual discovery. The two should be synonymous, but they are not. [1] - https://en.wikipedia.org/wiki/Educational_attainment_in_the_United_States#General_attainment_of_degrees/diplomas https://en.wikipedia.org/wiki/Educational_attainment_in_the_...
- throwawayjava 8y ago
  The statistics about masters and professional degrees are mostly irrelevant; those aren't research degrees and, although there are some exceptions, people with those degrees don't publish science. 1.77% of the 25+ population has a doctorate degree. Even among those, a huge amount are not PhDs and are not really research-focused. E.g., go read the CV of the typical University of Phoenix doctorate holder. Furthermore, it's worth noting that the world got a lot more complex since the mid-20th century. The entire field of Computer Science happened, for example.
  - TangoTrotFox 8y ago
    Definitely great points. On the other hand, even when you really lowball the numbers - I do think the point remains. 1.77% is 3.5 million people with doctorates in the USA alone. Perhaps we can see this issue manifest is something more directly quantifiable. Postdoctoral researchers tend to be some of the most highly educated and ostensibly skilled individuals in our population. Yet an average postdoc [1] will tend to earn [much] less than a mailman [2]. It seems difficult to explain this outside of a severe oversupply. And should the rate of people pursuing doctorates be increasing, which seems to be the case, I think we can expect things to get even more silly both in terms of compensation as well as output quality. [1] - https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-048.html https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-0... [2] - https://www.bls.gov/oes/current/oes435052.htm https://www.bls.gov/oes/current/oes435052.htm

54 points by gbear605 8y ago

I wholeheartedly agree with this assessment. I think the culture is extremely toxic, and it pushes out many promising students and researchers.

raziel2701 8y ago

J.M. Ziman wrote in 1962: "It is typical of modern physicists that they will erect skyscrapers of theory upon the slender foundations of outrageously simplified models." I guess it is only a matter of time before the proverbial skyscraper collapses. I have had first hand experience with publishing papers of decreasing relevance. It's almost always due to professors being unwilling to give up their mental model of reality. I had one professor tell me he would not let me publish my result because it contradicted a paper he wrote in the late 70s. It was the grossest violation of the spirit and purpose of science I've encountered, and it quickly turned me off from the whole institution. Egos in academia are often unchecked.

nerdponx 8y ago
At least these physics models lead to testable predictions. You should see what goes on in economics.
- AlexCoventry 8y ago
  Curious about why the parent comment is getting downvoted.
  - deleted 8y ago
    [deleted]
- deleted 8y ago
  [deleted]
- pythonslayer 8y ago
  Yeah it seems the academic posturing (in almost every discipline) is just insulation for the field of Economics. An MBA friend’s professor told him that economics is just like gravity. It’s not worth wasting his time on Earth thinking about it.

throwaway770 8y ago

Is there any advice onto how one deals with research misconduct in the industry? That allows to avoid expensive pitfalls like Theranos? Is leaving the place the best option? Simply leaving the place doesn't seem optimal. It feels like, it just leaves an opening for a recent H1 from a poor country, with a different tolerance for dishonesty. And the end-result is continuing misconduct.

raziel2701 8y ago
I think you should speak up with the expectation that whistleblowers don't do well, but with the knowledge that science corrects only if scientists correct it. It's unlikely a single person can take down the culture of an industry, but one person can certainly start a movement. You're definitely well justified in feeling torn about this. I agree that leaving in silence is not the best idea, but I certainly wouldn't blame you if you did that. It's hard to take action.

ianbicking 8y ago

"This problem cannot be solved by appointing non-experts to review panels – that merely creates incentives for research that’s easy to comprehend." That doesn't sound SO bad. Selection criteria seems like a major issue here, and the author only talks about the validity of the results. Why do we care about physics? Or psychology? There are answers to these simple questions, but I don't think the answers inform the selection criteria. We don't care about physics because academic researchers need to publish papers or get funding. Having any old non-expert on the panel could be problematic. Especially if they become political tools, and right now they'd definitely become political tools. But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics? These aren't hostile interlopers. But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.

rossdavidh 8y ago
Agreed. The article also said that having outsiders on the panel would just be an incentive to write easy to understand research proposals, and I was reminded of the idea that you need to understand something really well to be able to explain it clearly. If a physicist cannot explain their research proposal to an engineer or physicist, perhaps they need to go off and think about it some more.
- archgoon 8y ago
  > perhaps they need to go off and think about it some more. Well, that's exactly what they're asking for funding for. The entire crux of the issue is that these are not complete ideas, and need to be developed further; which takes time, and therefore, money.
addicted 8y ago
You’re right. What we should also do is that all critical code that may significantly impact lives and people’s wealth and security should be written in a manner that can be understood by laymen. So functional programming is right out. Anything more complex than BASIC is probably not a solution. Probably only literate programming is allowed.

majos 8y ago

I like that this article includes suggested changes: > For starters, every scientist should know how being part of a group can affect their opinion. Grants should not be awarded based on popularity. Researchers who leave fields of declining promise need encouragement, not punishment because their productivity may dwindle while they retrain. And we should generally require scientists to name both advantages and shortcomings of their hypotheses. The first and last points look good to me. The middle two, not so much. They certainly sound reasonable, but what does a grant system not based on "popularity" mean? How is it different from what we have now, where we find some (putative) experts in the area, put it in front of them, and ask what they think? We need some way of deciding what to fund and what not to fund, and who exactly is better equipped to make those decisions than existing researchers? Similarly, I like the idea that people should be able to switch fields. But the implicit argument seems to be that the emphasis on publications makes this difficult, so publications should be de-emphasized in assessing researcher quality. But what is better? Asking a researcher's peers? Then it is a popularity contest again. Weighting publications differently, i.e. "what are your 5 best papers?" rather than "how many papers do you have?"? Then who judges the 5? For all the criticism heaped upon bibliometrics: they get used because assessing the quality of most research is difficult, so even noisy filters like conferences (in CS) or journals are taken as a useful signal. It's not clear to me what better options exist, especially when trying to choose between many options. It's just not possible for one person or even a small group of people to truly familiarize themselves with the bodies of work of all 100 candidates for a faculty position/grant/fellowship/etc.

rossdavidh 8y ago
Agreed, and I would add that suggestions 1 and 4, although fine, will not be enough to make a substantive difference. To be honest, Sen. Paul's suggestion (if implemented properly, i.e. add experts from other fields rather than political hacks) sounds more likely to help.

majos 8y ago

A graduating PhD student in theoretical physics from a Top 5 school said something similar to me last year. Roughly, he said it was very easy to slip into bullshitty fooling yourself when doing theoretical physics research, and it took vigilance from within to do truly good research, because external bodies were unlikely to really punish you as long as the math worked out.

dwaltrip 8y ago
> it took vigilance from within to do truly good research Well said. I've been mulling on the idea that a scientific mindset must include a deeply embedded skepticism that isn't afraid to analyze and question even the most pervasive and mundane aspects of one's own being. Of course, one has to get through the day and take care of life, in a practical fashion. But I think too often we assume our actions to be much more intentional than they actually are.

TangoTrotFox 8y ago

I think this article is looking at the effect rather than the cause. Why might it be that so much of science has started to edge away from well... science. Today nearly 1 in 8 people have a masters degree or higher. And in one in 33 have a doctorate or professional degree. [1] And these numbers seem to be increasing. That's really insane if you think about it. We now have more people with postgraduate degrees than had any degree in 1970, and postgraduate education generally implies a necessity to publish. But discovering genuinely new and novel information is not easy. So what do people publish instead? Enter the current state of science and academia. I think the most informative paragraph in this article was: "Or look at Brian Wansink, the Cornell Professor with the bottomless soup bowl experiment. He recently drew unwanted attention to himself with a blogpost in which he advised a student to try harder getting results out of data because it “cost us a lot of time and our own money to collect.” Had Wansink been aware that massaging data until it delivers is not sound statistical procedure, he’d probably not have blogged about it." Physics is one of the least susceptible to this issue, but no field is immune. The issue is that 'science' has become more about publishing than actual discovery. The two should be synonymous, but they are not. [1] - https://en.wikipedia.org/wiki/Educational_attainment_in_the_United_States#General_attainment_of_degrees/diplomas https://en.wikipedia.org/wiki/Educational_attainment_in_the_...

throwawayjava 8y ago
The statistics about masters and professional degrees are mostly irrelevant; those aren't research degrees and, although there are some exceptions, people with those degrees don't publish science. 1.77% of the 25+ population has a doctorate degree. Even among those, a huge amount are not PhDs and are not really research-focused. E.g., go read the CV of the typical University of Phoenix doctorate holder. Furthermore, it's worth noting that the world got a lot more complex since the mid-20th century. The entire field of Computer Science happened, for example.
- TangoTrotFox 8y ago
  Definitely great points. On the other hand, even when you really lowball the numbers - I do think the point remains. 1.77% is 3.5 million people with doctorates in the USA alone. Perhaps we can see this issue manifest is something more directly quantifiable. Postdoctoral researchers tend to be some of the most highly educated and ostensibly skilled individuals in our population. Yet an average postdoc [1] will tend to earn [much] less than a mailman [2]. It seems difficult to explain this outside of a severe oversupply. And should the rate of people pursuing doctorates be increasing, which seems to be the case, I think we can expect things to get even more silly both in terms of compensation as well as output quality. [1] - https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-048.html https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-0... [2] - https://www.bls.gov/oes/current/oes435052.htm https://www.bls.gov/oes/current/oes435052.htm