4 ms·

I think people are hesitant to over-criticise academia. Maybe they can already hear the resulting anti-science sentiment, and fear losing hard fought ground if

I think people are hesitant to over-criticise academia. Maybe they can already hear the resulting anti-science sentiment, and fear losing hard fought ground if academia is called into question. However, I think there are serious signs that we should be looking for radical changes to the metstructures of academia. They're not necessarily related, but they point to something being really wrong. First is the replicability crisis. Have we actually been accumulating knowledge in fields where this is bad? Do we know more about human behaviour due to the human behaviour research done in the last generation? What's the point of all these "more research needed" conclusions. The process of academic publishing and peer reveiw directs this whole thing, and it doesn't seem to be all that well directed.. While we're on publishing, everyone in academia complains bitterly about the publish-or-perish problem and about grant/funding politics. One of the big complaints about grants is that senior academics spend all their time on grants and administration. Success in many fields is more a function of being good on administration and politics, than being good at research. While we're on administration... the number of administrators (not teachers or researchers) has skyrocketed over the last 2 generations. Ultimately, I think that the "scientific method" in practice is in large part embodied the academic publishing system. ...the algorithm determining how science works.

wgerard 9y ago
> I think people are hesitant to over-criticise academia. Really? Maybe inside academia itself that's true, but I think people outside are fairly eager to criticize academia and put down academics. I feel like anytime a study with even a slightly intriguing/controversial abstract gets circulated, one of the following happens (in order of frequency): 1) Narrow criticism of something very specific (usually sample sizes) with very little understanding about why that criticism might be misinformed (again, usually relating to sample sizes and how statistical significance works) which they use to discredit the whole paper 2) Pointing to previous papers that may (but often don't) contradict what the paper in question is suggesting - and "suggesting" is a loose term here since people tend to draw all sorts of conclusions that the author often doesn't. This is then used to basically throw hands up and say "it's unknowable." 3) Just plain dismissal, especially when the paper's implication is something that the reader doesn't want to hear. This comes in all forms, from the pedigree-based "oh who cares what someone from Montana State says what do they know" to the more direct ones like "why the hell would I listen to Bernanke talk about business, he's an ivory tower economist"
- dalbasal 9y ago
  I guess hesitance is a relative :) In any case, I hesitate a little, for the above reasons. Criticizing papers is..erm.. good and proper, I think. Sometimes the criticism will be crap, but a critical instinct isn't a bad thing here. Bias is, c'est la vie. Critising academics... I feel we're far too comfortable dishing out personal criticism in the internet era. Leave that one there. Criticising the system... All those "flags" that I raised are actually common and recurring criticisms of the system, within academia. But, they seem to fall short on how meta they go. Publish-or-perish & grant systems, the insider criticism sounds more like "my boss is a moron" talk, cheap shots. What I'm hoping for is big ideas from academia, I guess. Ideas about how to structure the institutions of academia: publishing, grants, whatever else determines what research gets done, by whom. Something that promotes efficiency.^ Something that promotes completion/conclusion so that we end up with knowledge, not a collection of research findings. Negative results and data pooling. Seperation of data & interpretation. Replication. This is not an excel-vs-R problem. It's an "incentives-within-academia" issue. ^Remember, there're problems that these grant bureaucracies are clumsily trying to solve. This is a market, and a bad market can be a lot worse than a good one.
- castle-bravo 9y ago
  I think there are two main kinds of criticism here; the first comes from people who think a lot of published work is misleading at best and fraudulent at worst (see replicability crisis); the second comes from people who think there's something horribly wrong with the culture of academia, notwithstanding the quality of output. Criticism in the second category touches on topics like adjunct professors (most instructors of undergraduate courses are temp workers making near minimum wage with no benefits), the takeover of universities by a class of professional administrators (many of whom have unbelievably cushy jobs), the distribution of research grants to researchers with the most political pull rather than those doing the most interesting work, the pressure put on researchers to acquire grants (see Stephan Grimm), and the crushing debt burden most undergraduates take on to get a university degree (most of which is spent frivolously while instructors and researchers work themselves to death). I think what you've described sits more in the first category than the second, while the parent belongs more to the second.
m-watson 9y ago
I think it is reasonable to question academia and research but I think a problem, often, is people want science and research to give us answers. It does not, it provides evidence, and once that evidence gets supported enough times it is viewed as closer and closer to reality. I would say it is asymptotic though, we never reach what is fact because there can always be a missing piece of evidence that shows a different result. There is one statement you make that I do want to directly comment on. The point of more research is needed is simply, that more research is needed. It goes directly to your statement of "do we know more about human behavior due to the ..." We learn more with all research and the way research currently gets disseminated is through publication. More research is needed publications as well as null result research is extremely useful in the research community and often does not get the attention outside (or inside sadly) the scientific community that it deserves.
- dalbasal 9y ago
  I'm not arguing at the philosophical level. My views are fairly mundane on that. I'm talking about the implementation of that philosophy (where I agree with you) in academia, as a system. When I say "build knowledge", I'm speaking colloquially. For a contrived example, we know traits are inherited by animals via genetics. We're confident enough to treat this as knowledge and design other expirements that take this as an assumption. Are we building this kind of knowledge in social science, currently? Does "more research needed" lead to more research being done, typically in a field? If not, why? Why are we moving on to a new question, when the last one is still unanswered. Take a "willpower is a limited resource" hypothesis (replication issues aside). It does not seem impossible to answer this question. Nothing in science is "knowledge" in an absolute sense, but we can get to some level of confidence. So.. do we want to know the answer to this question? If not, why do the the first expirement? If yes, then what would it take to get to an answer? We can answer that partially in advance. Independant replication will be part of the answer. Sufficient sample size also. If we know in advance that these things won't be done, what is the point? I'm an outsider and I realize I may be throwing unwarranted dirt, so if/where I am, apologies. I think that in some fields we systematically do not do the things necessary to get to useful knowledge, a sufficient accumulation of evidence to treat a result as more than anecdote.
Vinnl 9y ago
I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the problems in academia, once you start to dive into one you will learn a lot about the others as well, as they're somewhat intertwined. And really, the scope and breadth of the problems is astonishing. Personally, my hunch is that the root cause for most of them is a somewhat misguided focus on "excellence" combined with it being practically impossible to measure researcher performance with a reasonable amount of effort. Then again, the problems are so intertwined and have so many different actors, that finding the root cause is really hard, so who knows whether I'm correct.
- dalbasal 9y ago
  I think measuring research performance is a dead end. People can do this, although they carry a ton of blindness and blindspots. Bureaucracies, systems and such can't. They end up measuring what they can measure, like grades, attendance, number of published papers, number of citations maybe... It just won't end well. But, we can have systems that promote performance without measuring it. I think those are the kinds of ideas we should look at.
  - Vinnl 9y ago
    I agree with your first paragraph, but I don't see how your second paragraph would work. Do you have any concrete examples/ideas of how such a system could work? (Without requiring e.g. those on tenure committees to read and understand all the research of all applications - unless you believe that could work somehow.)
    - dalbasal 9y ago
      Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start. Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot. Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes." Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices... ..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important? A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong. If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on. Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.
api 9y ago
Treating science like a religion is also anti-science.
losvedir 9y ago
I think this comment is spot on. In particular, > I think people are hesitant to over-criticise academia. is a message to people here. Of course, there's all the "anti-global warming, intelligent design flat earther republicans" who have always criticized academia, but that's not what you're talking about (I think). I think you mean that people here - i.e. relatively educated, orthodox liberals for the most part - should really start to look hard at academia. I'm someone with a traditional academic pedigree (double major in Econ + Math from MIT, most of my friends have PhDs) but I, too, am growing increasingly wary of much published research. For example, there's this eye-opening blog post[0] by Andrew Gelman, a professor of statistics at Columbia who collaborates on a lot of psychology research, about the problem of bad statistics in research in psychology. There's this [1] famous paper by the Fed about problems of replication in what I studied, economics. Or how the NIMH "is re-orienting its research away from DSM categories".[2] And then there's the endless pop-science ideas that show up and get shot down like Power Pose, Air Rage, or Learning Styles. Or the constantly changing recommendations on nutrition. I don't really take for granted anymore that academia is generally working towards greater understanding of the world. It's an imperfect system that has done some incredible things, but it definitely is dealing with some serious issues at the moment. I hope that initiatives like journals for null results, changes in the way grants are apportioned (like in this posted article), etc, can put it back on track. [0] http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he... [1] https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf https://www.federalreserve.gov/econresdata/feds/2015/files/2... [2] https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2013/transforming-diagnosis.shtml https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2...

wgerard 9y ago

> I think people are hesitant to over-criticise academia. Really? Maybe inside academia itself that's true, but I think people outside are fairly eager to criticize academia and put down academics. I feel like anytime a study with even a slightly intriguing/controversial abstract gets circulated, one of the following happens (in order of frequency): 1) Narrow criticism of something very specific (usually sample sizes) with very little understanding about why that criticism might be misinformed (again, usually relating to sample sizes and how statistical significance works) which they use to discredit the whole paper 2) Pointing to previous papers that may (but often don't) contradict what the paper in question is suggesting - and "suggesting" is a loose term here since people tend to draw all sorts of conclusions that the author often doesn't. This is then used to basically throw hands up and say "it's unknowable." 3) Just plain dismissal, especially when the paper's implication is something that the reader doesn't want to hear. This comes in all forms, from the pedigree-based "oh who cares what someone from Montana State says what do they know" to the more direct ones like "why the hell would I listen to Bernanke talk about business, he's an ivory tower economist"

dalbasal 9y ago
I guess hesitance is a relative :) In any case, I hesitate a little, for the above reasons. Criticizing papers is..erm.. good and proper, I think. Sometimes the criticism will be crap, but a critical instinct isn't a bad thing here. Bias is, c'est la vie. Critising academics... I feel we're far too comfortable dishing out personal criticism in the internet era. Leave that one there. Criticising the system... All those "flags" that I raised are actually common and recurring criticisms of the system, within academia. But, they seem to fall short on how meta they go. Publish-or-perish & grant systems, the insider criticism sounds more like "my boss is a moron" talk, cheap shots. What I'm hoping for is big ideas from academia, I guess. Ideas about how to structure the institutions of academia: publishing, grants, whatever else determines what research gets done, by whom. Something that promotes efficiency.^ Something that promotes completion/conclusion so that we end up with knowledge, not a collection of research findings. Negative results and data pooling. Seperation of data & interpretation. Replication. This is not an excel-vs-R problem. It's an "incentives-within-academia" issue. ^Remember, there're problems that these grant bureaucracies are clumsily trying to solve. This is a market, and a bad market can be a lot worse than a good one.
castle-bravo 9y ago
I think there are two main kinds of criticism here; the first comes from people who think a lot of published work is misleading at best and fraudulent at worst (see replicability crisis); the second comes from people who think there's something horribly wrong with the culture of academia, notwithstanding the quality of output. Criticism in the second category touches on topics like adjunct professors (most instructors of undergraduate courses are temp workers making near minimum wage with no benefits), the takeover of universities by a class of professional administrators (many of whom have unbelievably cushy jobs), the distribution of research grants to researchers with the most political pull rather than those doing the most interesting work, the pressure put on researchers to acquire grants (see Stephan Grimm), and the crushing debt burden most undergraduates take on to get a university degree (most of which is spent frivolously while instructors and researchers work themselves to death). I think what you've described sits more in the first category than the second, while the parent belongs more to the second.

m-watson 9y ago

I think it is reasonable to question academia and research but I think a problem, often, is people want science and research to give us answers. It does not, it provides evidence, and once that evidence gets supported enough times it is viewed as closer and closer to reality. I would say it is asymptotic though, we never reach what is fact because there can always be a missing piece of evidence that shows a different result. There is one statement you make that I do want to directly comment on. The point of more research is needed is simply, that more research is needed. It goes directly to your statement of "do we know more about human behavior due to the ..." We learn more with all research and the way research currently gets disseminated is through publication. More research is needed publications as well as null result research is extremely useful in the research community and often does not get the attention outside (or inside sadly) the scientific community that it deserves.

dalbasal 9y ago
I'm not arguing at the philosophical level. My views are fairly mundane on that. I'm talking about the implementation of that philosophy (where I agree with you) in academia, as a system. When I say "build knowledge", I'm speaking colloquially. For a contrived example, we know traits are inherited by animals via genetics. We're confident enough to treat this as knowledge and design other expirements that take this as an assumption. Are we building this kind of knowledge in social science, currently? Does "more research needed" lead to more research being done, typically in a field? If not, why? Why are we moving on to a new question, when the last one is still unanswered. Take a "willpower is a limited resource" hypothesis (replication issues aside). It does not seem impossible to answer this question. Nothing in science is "knowledge" in an absolute sense, but we can get to some level of confidence. So.. do we want to know the answer to this question? If not, why do the the first expirement? If yes, then what would it take to get to an answer? We can answer that partially in advance. Independant replication will be part of the answer. Sufficient sample size also. If we know in advance that these things won't be done, what is the point? I'm an outsider and I realize I may be throwing unwarranted dirt, so if/where I am, apologies. I think that in some fields we systematically do not do the things necessary to get to useful knowledge, a sufficient accumulation of evidence to treat a result as more than anecdote.

Vinnl 9y ago

I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the problems in academia, once you start to dive into one you will learn a lot about the others as well, as they're somewhat intertwined. And really, the scope and breadth of the problems is astonishing. Personally, my hunch is that the root cause for most of them is a somewhat misguided focus on "excellence" combined with it being practically impossible to measure researcher performance with a reasonable amount of effort. Then again, the problems are so intertwined and have so many different actors, that finding the root cause is really hard, so who knows whether I'm correct.

dalbasal 9y ago
I think measuring research performance is a dead end. People can do this, although they carry a ton of blindness and blindspots. Bureaucracies, systems and such can't. They end up measuring what they can measure, like grades, attendance, number of published papers, number of citations maybe... It just won't end well. But, we can have systems that promote performance without measuring it. I think those are the kinds of ideas we should look at.
- Vinnl 9y ago
  I agree with your first paragraph, but I don't see how your second paragraph would work. Do you have any concrete examples/ideas of how such a system could work? (Without requiring e.g. those on tenure committees to read and understand all the research of all applications - unless you believe that could work somehow.)
  - dalbasal 9y ago
    Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start. Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot. Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes." Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices... ..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important? A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong. If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on. Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.

api 9y ago

Treating science like a religion is also anti-science.

losvedir 9y ago

I think this comment is spot on. In particular, > I think people are hesitant to over-criticise academia. is a message to people here. Of course, there's all the "anti-global warming, intelligent design flat earther republicans" who have always criticized academia, but that's not what you're talking about (I think). I think you mean that people here - i.e. relatively educated, orthodox liberals for the most part - should really start to look hard at academia. I'm someone with a traditional academic pedigree (double major in Econ + Math from MIT, most of my friends have PhDs) but I, too, am growing increasingly wary of much published research. For example, there's this eye-opening blog post[0] by Andrew Gelman, a professor of statistics at Columbia who collaborates on a lot of psychology research, about the problem of bad statistics in research in psychology. There's this [1] famous paper by the Fed about problems of replication in what I studied, economics. Or how the NIMH "is re-orienting its research away from DSM categories".[2] And then there's the endless pop-science ideas that show up and get shot down like Power Pose, Air Rage, or Learning Styles. Or the constantly changing recommendations on nutrition. I don't really take for granted anymore that academia is generally working towards greater understanding of the world. It's an imperfect system that has done some incredible things, but it definitely is dealing with some serious issues at the moment. I hope that initiatives like journals for null results, changes in the way grants are apportioned (like in this posted article), etc, can put it back on track. [0] http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he... [1] https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf https://www.federalreserve.gov/econresdata/feds/2015/files/2... [2] https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2013/transforming-diagnosis.shtml https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2...