9 ms·

The job market right now in the US forces grad students to attempt innovative research with remarkable conclusions. As a consequence, everyone's trying to prove

The job market right now in the US forces grad students to attempt innovative research with remarkable conclusions. As a consequence, everyone's trying to prove more outlandish things while few are bothering with replications. If the field were sane, you would train all the apprentices on replication studies. Once they demonstrated dispassionate expertise with the tools, only then would they be allowed to try to use those tools to test their own ideas, where they will have a strong emotional preference for how the study will come out. If universities hired grad students based on their replication work, not on their eye-popping original research, we'd have better science and better scientists.

x5n1 11y ago
If the field was sane we would not expect breakthrough results or any results all of the time. We would happily say to scientists, thanks for your continued efforts... and allow them the money and time to do real science. But this is not how the crazy Capitalist world works. You have sociopaths which run these institutions always barking orders and demanding results. So the poor employee gives you what you ask, and you say good boy, and give them a treat.
themartorana 11y ago
A lot of the time those same students are doing the bulk of the legwork for those findings then attributed to a professor, who is under greater pressure to advance (not confirm) science, who may or may not have even participated heavily in the grunt work of that research. The professor then punishes the work under his/her name, with no credit to the work of the students.
- LeifCarrotson 11y ago
  Publishes. I assume that was a typo?
  - eplanit 11y ago
    No doubt a typo. Punishment is the consequence of not publishing. :)
- deleted 11y ago
  [deleted]
- privong 11y ago
  > A lot of the time those same students are doing the bulk of the legwork for those findings then attributed to a professor, who is under greater pressure to advance (not confirm) science, who may or may not have even participated heavily in the grunt work of that research. The professor then punishes the work under his/her name, with no credit to the work of the students. It's important to note that though this seems to happen in all fields, it is far less common in some. I rarely hear of such things in astronomy; it does happen, but more often I hear about faculty explicitly working to ensure they can protect projects for their students so their students can get the credit deserved for doing the project. I hear of professors appropriating student research in biology and chemistry more frequently, however. The occurrences of professors taking and publishing student research is certainly a problem, is unethical, and should be stopped. But the way in which it is usually discussed ("in science") implies that it's a systemic issue across all science, which (from my experience) isn't true. This topic of credit and attribution for research deserves more nuanced discussion and fewer blanket statements. Edit: Spelling, wording clarification.
  - ska 11y ago
    I think it does vary a lot by discipline. In my area of mathematics I don't recall ever hearing of it. For whatever it is worth I even published papers without my supervisor being on them at all, if he hadn't been involved in that work.
    - qmalzp 11y ago
      I was thinking the same thing. I think math lends itself to this because the work is so esoteric that often the advisor doesn't even completely understand the work, much less can claim credit for it.
      - nmrm2 11y ago
        I think the size of labs is more important. Bio and Chem labs -- especially the best ones -- are often freaking huge. There's no way the PI actually has time to meaningfully contribute to every paper when they are bringing in enough money to support dozens of full-time positions. In math, the group sizes are way smaller. A PI might only have 1 or 2 students.
compactmani 11y ago
If the work you are doing isn't reproducible, then it won't be cited as the building blocks for future work. The system is cleans itself out as a side effect. Therefore I prefer students to work on cutting edge research.
- tines 11y ago
  The point is that nobody knows if it's reproducible, because nobody's attempting to reproduce it.
  - compactmani 11y ago
    Yes that may be true. Instead of trying to reproduce it, they attempt to build upon it. The projects which successfully build on previous projects become original research articles and the cycle continues.
    - privong 11y ago
      > Yes that may be true. Instead of trying to reproduce it, they attempt to build upon it. The projects which successfully build on previous projects become original research articles and the cycle continues. Whether this "works" or not depends on how the previous projects enter into it. If the results of previous projects are used as assumptions to justify the methods, data, etc. of the subsequent project, there is no check and we risk the research becoming a house of cards which could collapse due to faultly, untested assumptions that were used. If the subsequent projects are performed in such a way that they also test the previous results/assumption, this can be avoided. I can't tell from your wording which you are suggesting, though it seems to lean towards the former.
    - worik 11y ago
      No. That completely misrepresents the problem. tl;dr A false technique can be described and it can be hard or impossible to detect the technique is flawed by using it For example the paper at http://www.jstor.org/stable/222500 http://www.jstor.org/stable/222500 describes a method of using the stationary bootstrap to eliminate data snooping bias in studies of "technical analysis" in finance. Published in the Journal of Finance in 1999 at the time I worked with it in 2010 it had been cited over 500 times. The proof in the paper is inscrutable. I could find no one who could explain or verify the proofs at my institution. I attempted to reproduce the results in the paper which is where the problems started. The authors did not give enough information to do this in all cases, but in some cases I was able to reconstruct the algorithms. They did not perform as described. Of the five algorithms I could reproduce (from memory) one of them worked roughly as described and two others were not completely hopeless. Looking more closely I realised that the original authors completely disregarded important factors in the implementation of the techniques described by the algorithms. Transaction costs. Allowing for transaction costs (a difficult but not impossible task) the effects noticed by the authors disappeared completely. Looking even closer I examined the assumptions behind "White's Reality Check" that the paper relied on the Stationary Bootstrap by Politis and Romano from 1994. But financial returns are not stationary. Not at all. So dodgy logic, misuse of statistics, irreproducible experiments, ignoring important aspects of the data and I suspect wishful thinking add up to a paper that is comprehensively false. Cited hundreds of times and used many times to verify other results. taking a huge risk here is where my research is: https://ourarchive.otago.ac.nz/bitstream/handle/10523/4346/TureiStantonWorikM2013MCom.pdf?sequence=1 https://ourarchive.otago.ac.nz/bitstream/handle/10523/4346/T... Section 5.12.1 "Bootstrapping and Other False Discoveries"
basseq 11y ago
Academia has always forced grad students to attempt innovative research. The core requirement of a PhD dissertation is "novel findings"—e.g., something completely new. Apprentices work on papers and research for their professors. No one is incentivized to do replication work: either you replicate it successfully (great) or you find flaws. In the latter case, you'll probably be a nit-picker anyway—also not really additive. On the off-chance that you refute a high-profile study (e.g., some of the outright instances of fraud), you might get some recognition, but now your name is associated with something negative (e.g., fraud: "X is not true") vs. positive ("X is true"). Finally, this is some of the role of review panels in journals: too pass the burden of proof.
Balgair 11y ago
That is nearly the opposite of what the PhD was designed to do. Modern academic training comes out of the church and the old guilds of middle Europe and is still in use today in many fields (chefs and plumbers to name a few). The Bachelor's degree is loosely similar to an apprentice's role. The young boy (they were almost exclusively male) worked in a shop or with a priest for some time. He learned the trade, the tools, and gained some experience from 'level 0'. When you are done with the apprenticeship, you are 'cleared' to work in other shops and are known to not be a total moron or break tools or burn down shops. The master's degree is just that. You are considered a master of the craft (like plumbing or prinitng) or the discipline (like The Book of Mark or Crusader History). As such, you typically have a master's level project. Something that is 'new' or shows that you know your stuff. That might be a very decorative silver bowl or a thesis. The Doctorate means you are 'world class.' Not just a mastery in a field, but a paragon of it. Today, that means that you are the expert in your little niche of underwater basket weaving. There should be no-one better than you. This means you MUST have produced something new or novel way of thinking about the God or something. This has always been the idea, if not the practice. To change that and say that the doctorate should be the bachelor's is very big. To suggest that PhDs should just replicate experiments is anathema to the idea of graduate education and would be a tremendous waste of time and energy. When you enter the Phd, you are assumed to already know how to do all the replication and the facts about the field. Granted, fields are exponentially larger than they were in the 1600's, but you still should know stats and biology if your PhD is in cancer biology. I think you are totally wrong about this. What you are suggesting should be covered in undergrad and I think it largely is.
- bunderbunder 11y ago
  The problem is that nobody's doing replication work in undergrad, either, nor does it happen in Master's programs. I do think it makes sense to stick with the PhD meaning you're a world class expert in some area, but if so then we need to adjust our expectations for what Master's level work means in the sciences. Right now it seems to just represent a hurdle you need to whiz past on your way to the PhD.
  - Balgair 11y ago
CyberDildonics 11y ago
That is a bold claim and conclusion, what data are you basing that on?
- brownbat 11y ago
  Excellent question! Mostly the research of people like John Ioannidis or Brian Nosek or Megan Head. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106 http://journals.plos.org/plosbiology/article?id=10.1371/jour... https://www.google.com/url?sa=t&source=web&rct=j&url=http://robotics.cs.tamu.edu/RSS2015NegativeResults/pmed.0020124.pdf&ved=0ahUKEwiW_87a3p7LAhXsvYMKHaV8ABsQFggeMAA&usg=AFQjCNE2rEvKJYMaZCd1DmPexoGavyZODQ&sig2=1A93admlZoGYVjzwMkb8Qw https://www.google.com/url?sa=t&source=web&rct=j&url=http://... http://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experiment http://www.npr.org/sections/money/2016/01/15/463237871/episo...

x5n1 11y ago

If the field was sane we would not expect breakthrough results or any results all of the time. We would happily say to scientists, thanks for your continued efforts... and allow them the money and time to do real science. But this is not how the crazy Capitalist world works. You have sociopaths which run these institutions always barking orders and demanding results. So the poor employee gives you what you ask, and you say good boy, and give them a treat.

themartorana 11y ago

A lot of the time those same students are doing the bulk of the legwork for those findings then attributed to a professor, who is under greater pressure to advance (not confirm) science, who may or may not have even participated heavily in the grunt work of that research. The professor then punishes the work under his/her name, with no credit to the work of the students.

LeifCarrotson 11y ago
Publishes. I assume that was a typo?
- eplanit 11y ago
  No doubt a typo. Punishment is the consequence of not publishing. :)
deleted 11y ago
[deleted]
privong 11y ago
> A lot of the time those same students are doing the bulk of the legwork for those findings then attributed to a professor, who is under greater pressure to advance (not confirm) science, who may or may not have even participated heavily in the grunt work of that research. The professor then punishes the work under his/her name, with no credit to the work of the students. It's important to note that though this seems to happen in all fields, it is far less common in some. I rarely hear of such things in astronomy; it does happen, but more often I hear about faculty explicitly working to ensure they can protect projects for their students so their students can get the credit deserved for doing the project. I hear of professors appropriating student research in biology and chemistry more frequently, however. The occurrences of professors taking and publishing student research is certainly a problem, is unethical, and should be stopped. But the way in which it is usually discussed ("in science") implies that it's a systemic issue across all science, which (from my experience) isn't true. This topic of credit and attribution for research deserves more nuanced discussion and fewer blanket statements. Edit: Spelling, wording clarification.
- ska 11y ago
  I think it does vary a lot by discipline. In my area of mathematics I don't recall ever hearing of it. For whatever it is worth I even published papers without my supervisor being on them at all, if he hadn't been involved in that work.
  - qmalzp 11y ago
    I was thinking the same thing. I think math lends itself to this because the work is so esoteric that often the advisor doesn't even completely understand the work, much less can claim credit for it.
    - nmrm2 11y ago
      I think the size of labs is more important. Bio and Chem labs -- especially the best ones -- are often freaking huge. There's no way the PI actually has time to meaningfully contribute to every paper when they are bringing in enough money to support dozens of full-time positions. In math, the group sizes are way smaller. A PI might only have 1 or 2 students.

compactmani 11y ago

If the work you are doing isn't reproducible, then it won't be cited as the building blocks for future work. The system is cleans itself out as a side effect. Therefore I prefer students to work on cutting edge research.

tines 11y ago
The point is that nobody knows if it's reproducible, because nobody's attempting to reproduce it.
- compactmani 11y ago
  Yes that may be true. Instead of trying to reproduce it, they attempt to build upon it. The projects which successfully build on previous projects become original research articles and the cycle continues.
  - privong 11y ago
    > Yes that may be true. Instead of trying to reproduce it, they attempt to build upon it. The projects which successfully build on previous projects become original research articles and the cycle continues. Whether this "works" or not depends on how the previous projects enter into it. If the results of previous projects are used as assumptions to justify the methods, data, etc. of the subsequent project, there is no check and we risk the research becoming a house of cards which could collapse due to faultly, untested assumptions that were used. If the subsequent projects are performed in such a way that they also test the previous results/assumption, this can be avoided. I can't tell from your wording which you are suggesting, though it seems to lean towards the former.
  - worik 11y ago
    No. That completely misrepresents the problem. tl;dr A false technique can be described and it can be hard or impossible to detect the technique is flawed by using it For example the paper at http://www.jstor.org/stable/222500 http://www.jstor.org/stable/222500 describes a method of using the stationary bootstrap to eliminate data snooping bias in studies of "technical analysis" in finance. Published in the Journal of Finance in 1999 at the time I worked with it in 2010 it had been cited over 500 times. The proof in the paper is inscrutable. I could find no one who could explain or verify the proofs at my institution. I attempted to reproduce the results in the paper which is where the problems started. The authors did not give enough information to do this in all cases, but in some cases I was able to reconstruct the algorithms. They did not perform as described. Of the five algorithms I could reproduce (from memory) one of them worked roughly as described and two others were not completely hopeless. Looking more closely I realised that the original authors completely disregarded important factors in the implementation of the techniques described by the algorithms. Transaction costs. Allowing for transaction costs (a difficult but not impossible task) the effects noticed by the authors disappeared completely. Looking even closer I examined the assumptions behind "White's Reality Check" that the paper relied on the Stationary Bootstrap by Politis and Romano from 1994. But financial returns are not stationary. Not at all. So dodgy logic, misuse of statistics, irreproducible experiments, ignoring important aspects of the data and I suspect wishful thinking add up to a paper that is comprehensively false. Cited hundreds of times and used many times to verify other results. taking a huge risk here is where my research is: https://ourarchive.otago.ac.nz/bitstream/handle/10523/4346/TureiStantonWorikM2013MCom.pdf?sequence=1 https://ourarchive.otago.ac.nz/bitstream/handle/10523/4346/T... Section 5.12.1 "Bootstrapping and Other False Discoveries"

basseq 11y ago

Academia has always forced grad students to attempt innovative research. The core requirement of a PhD dissertation is "novel findings"—e.g., something completely new. Apprentices work on papers and research for their professors. No one is incentivized to do replication work: either you replicate it successfully (great) or you find flaws. In the latter case, you'll probably be a nit-picker anyway—also not really additive. On the off-chance that you refute a high-profile study (e.g., some of the outright instances of fraud), you might get some recognition, but now your name is associated with something negative (e.g., fraud: "X is not true") vs. positive ("X is true"). Finally, this is some of the role of review panels in journals: too pass the burden of proof.

Balgair 11y ago

That is nearly the opposite of what the PhD was designed to do. Modern academic training comes out of the church and the old guilds of middle Europe and is still in use today in many fields (chefs and plumbers to name a few). The Bachelor's degree is loosely similar to an apprentice's role. The young boy (they were almost exclusively male) worked in a shop or with a priest for some time. He learned the trade, the tools, and gained some experience from 'level 0'. When you are done with the apprenticeship, you are 'cleared' to work in other shops and are known to not be a total moron or break tools or burn down shops. The master's degree is just that. You are considered a master of the craft (like plumbing or prinitng) or the discipline (like The Book of Mark or Crusader History). As such, you typically have a master's level project. Something that is 'new' or shows that you know your stuff. That might be a very decorative silver bowl or a thesis. The Doctorate means you are 'world class.' Not just a mastery in a field, but a paragon of it. Today, that means that you are the expert in your little niche of underwater basket weaving. There should be no-one better than you. This means you MUST have produced something new or novel way of thinking about the God or something. This has always been the idea, if not the practice. To change that and say that the doctorate should be the bachelor's is very big. To suggest that PhDs should just replicate experiments is anathema to the idea of graduate education and would be a tremendous waste of time and energy. When you enter the Phd, you are assumed to already know how to do all the replication and the facts about the field. Granted, fields are exponentially larger than they were in the 1600's, but you still should know stats and biology if your PhD is in cancer biology. I think you are totally wrong about this. What you are suggesting should be covered in undergrad and I think it largely is.

bunderbunder 11y ago
The problem is that nobody's doing replication work in undergrad, either, nor does it happen in Master's programs. I do think it makes sense to stick with the PhD meaning you're a world class expert in some area, but if so then we need to adjust our expectations for what Master's level work means in the sciences. Right now it seems to just represent a hurdle you need to whiz past on your way to the PhD.
- Balgair 11y ago

CyberDildonics 11y ago

That is a bold claim and conclusion, what data are you basing that on?

brownbat 11y ago
Excellent question! Mostly the research of people like John Ioannidis or Brian Nosek or Megan Head. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106 http://journals.plos.org/plosbiology/article?id=10.1371/jour... https://www.google.com/url?sa=t&source=web&rct=j&url=http://robotics.cs.tamu.edu/RSS2015NegativeResults/pmed.0020124.pdf&ved=0ahUKEwiW_87a3p7LAhXsvYMKHaV8ABsQFggeMAA&usg=AFQjCNE2rEvKJYMaZCd1DmPexoGavyZODQ&sig2=1A93admlZoGYVjzwMkb8Qw https://www.google.com/url?sa=t&source=web&rct=j&url=http://... http://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experiment http://www.npr.org/sections/money/2016/01/15/463237871/episo...