5 ms·

I don't know if the comic properly represents the book, but all of the suggestions are somewhere between ridiculous and useless. First, about identifying bad r

I don't know if the comic properly represents the book, but all of the suggestions are somewhere between ridiculous and useless. First, about identifying bad research: The "extraordinary claims require extraordinary evidence" is already practiced. It's not that "we've overturned quantum theory" articles that are causing problem - those are quickly and effectively shot down. And it is rare that the "perfectly aligns with a political interest" can be applied. The only actionable one is "see what others think about it", and it's no panacea either. Bad and fraudulent science like the recently retracted Surgisphere covid paper is abundant. I was trying to track down the origin of the "reduce salt intake" and "limit egg consumption to no more than 2 per day / 2 per week" recommendation in the past, assuming there was hard science behind them. There isn't; and indeed they're slowly being reversed everywhere - but they were prevalent for half a century, with a lot of other research taking them as axioms. The remdesivir trials have been p-hacked to death - anyone who took an interest was seeing it happen in real time - yet, the scientific community turns a blind eye. The ketogenic diet is vilified in every mainstream media and most nutritional "science" publications; the headlines are rarely inline with the the actual results, but that's what people remember (Not that nutrition science is really science) And the other recommendations about fixing it are comparable to "lets solve evils of the US 2-party system! All we have to do is make those two parties vote to take away their power". Academia and science publishing are where they are now because it benefits essentially all the incumbents (at the expense of the rest of society). The problem description (at least in the comic) is good. Any suggested action ... not so much.

joppy 6y ago
I think the suggestion that "universities could change their hiring policies" is a very good recommendation. There are a lot of problems in academia (bad research, lack of diversity, pressure to publish) that could be reduced if universities changed their hiring policy to something further away from the metric of (number of papers published since receiving PhD) / (time since receiving PhD). Of course no university hires explicitly on that metric, but many of the metrics they use are not far from that, mixing in quality of journals and norms for the field, etc. Each of the suggested actions, taken together, would seem to have a very positive improvement on the status quo. Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?"
- meow1032 6y ago
  Not OP here, but my issue with the recommendations are that they've pretty accurately listed a whole bunch of mostly structural problems with academia, but all of the suggestions boil down to "we all just need to try harder". You can say something like: "journals need to demand higher standards" but what incentive do they actually have to do so? Then you can counter with "scientists could vote with their feet", but what incentives do they have to do that?? You're asking people to consider seriously damaging their career for some nebulous quality metric. Frankly, having worked in academia long enough to see at least a couple shifts in culture, the only thing I can see that comes out of this is a couple more things get added on to the ever growing checklist of publishing a paper/submitting a grant application. I think we need to get away from the sort of thinking where large structural problems can be solved by tiny incremental improvements. If you really want to solve the problem, one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up.
  - joppy 6y ago
    It seems to me, still, that a lot of these problems you bring up can be addressed by universities changing their hiring policies. Which makes sense: academics ultimately rely on universities for their income, and so it is the hiring policies which are setting the perverse incentives. And I don’t think changing hiring policies would be an incremental change, it would be a huge change (and not likely to be made by any university any time soon, since students rank universities on similar metrics to how universities hire staff — a prestigious university will lose prestige even if it changes its hiring policies for the better).
    - meow1032 6y ago
      > academics ultimately rely on universities for their income Sort of, a huge portion of income is from grants, particularly after the first few years from being hired. More importantly, a huge portion of the University income is from grants. When a researcher recieves a grant, there is an "overhead" percentage that goes to the University. Universities hire, in part, to maximize those overheads, which means getting the researchers with the best chance at getting big grants. Changing the hiring process may affect how PHD students act, but once they're "in the system", they are subject to all the same problematic incentives.
asddubs 6y ago
thank god we can eat a ton of salt again, to be honest I was doing it the whole time
- jschwartzi 6y ago
  I never stopped. The whole idea had a couple smells to it: * Maybe increases sodium intake is linked to poor health outcomes because highly processed foods are linked to poor health outcomes. We know that sodium content increases significantly during food processing and that most highly processed foods are really unhealthy. * Maybe people in early stages of renal failure are more likely to progress to a noticeable state if they consume lots of salt. Then it would stand to reason that people with healthy kidneys have nothing to worry about.
AnIdiotOnTheNet 6y ago
> The ketogenic diet is vilified in every mainstream media and most nutritional "science" publications; the headlines are rarely inline with the the actual results, but that's what people remember (Not that nutrition science is really science) Nutrition science is real science, but unfortunately any actual nutrition science you might accidentally hear about is overwhelmed by people trying to sell a lifestyle, a book about healthy eating, or your eyeballs (to advertisers). Ironically keto promoters are a big offender here themselves.
- amanaplanacanal 6y ago
  Most nutrition science is pretty bad. As is often said, correlation is not causation, but that's what most nutritional studies are. It's easy to study large populations and find correlations. You publish a paper, the media reports something, and that becomes the accepted wisdom. But you have no idea how to factor out all the possible confounders in your study. It's hard to do a study where you make a change to people's diet and see if that affects health outcomes, so it is rarely done. And really that's what you need to do to see what is really going on. So what we are left with is a bunch of associations that may or may not hold up.
- exolymph 6y ago
  Counterpoint: https://meaningness.com/nutrition https://meaningness.com/nutrition
  - beagle3 6y ago
    This is an excellent summary, thanks!
Bobbcatt 6y ago
>"extraordinary claims require extraordinary evidence" And yet most people accept climate change as fact, even though the claim itself is so extraordinary that we as a species are not capable of producing enough evidence for it. You can't predict accurately within 1 degree celcius the temperature 7 days from now, but you want me to believe you when you try to do it for 50 years from now.
csours 6y ago
I think that pre-registering methods is important. I think that a requirement to publish results either way on any study that's been pre-registered is important.
xenocyon 6y ago
> The problem description (at least in the comic) is good. Any suggested action ... not so much. Well put. In theory, people like the idea of making science better. In practice, people don't like the idea of fewer papers, boring papers, ambiguity in hiring and tenure, uncertainty on financial return-on-investment, and less institutional/national pride talking points. The bad incentives and metrics we have haven't happened by chance - they have emerged from our collective desire for science to be useful, sexy, and a reliable function of money/effort spent.
clairity 6y ago
> 'And the other recommendations about fixing it are comparable to "lets solve evils of the US 2-party system! All we have to do is make those two parties vote to take away their power".' that's a bit unfair. yes, it does require that in some semblance, but that's not necessarily the one and only lever we have. most of us realistically expect, i hope, that we'd need a variety of political maneuvers to move our democracy toward a more representative and less insular direction. science is no different--it took many small steps to get into this situation, and will take many small steps to get out, any one of which will seem wholly incapable on its own.
0d9eooo 6y ago
I think the horse has left he barn in terms of the suggestions in the comic. I love the comic and the attention it's getting but the issues discussed are such a small part of the problem. Many of the comments here are on point. At this point in time, many of the people in power (full professors, administration, etc) see papers as essentially useless, in part because of all the problems being discussed here. p-hacking, guest authorship, etc and so forth and so on. They're seen as a dime a dozen and somewhat ignored. This might sound good until you understand that this means that they're dismissing the entirety of the scientific dialogue essentially, and that their alternative is grant money. My institution had training workshops for grad students where faculty would tell students that research is essentially worthless unless grant money is attached to it. The idea is that papers are a dime a dozen, and that if something is worthwhile, the feds will put their money where their mouth is so to speak. The problem with this, of course, is that the grant system is horribly nepotistic and distorted. The biggest predictor of successful grant application last time I looked (based on empirical research in peer-reviewed journals) is co-authoring papers with someone on the review panel. Grant receipt is only weakly related to citation metrics, and from personal experience I can say that institutions are constantly encouraging researchers to inflate grant costs to bring in more indirect costs. You could fix everything about journals and the publication process and it would do nothing about the shadow scientific world that exists in parallel that drives all the rest of the problems. Eliminate indirect funds, require tenure of all researchers, set aside funding mechanisms for researchers that aren't tied to specific grant applications, randomize grant rewards, depriortize journals, ... there's a lot of things that need to happen.

joppy 6y ago

I think the suggestion that "universities could change their hiring policies" is a very good recommendation. There are a lot of problems in academia (bad research, lack of diversity, pressure to publish) that could be reduced if universities changed their hiring policy to something further away from the metric of (number of papers published since receiving PhD) / (time since receiving PhD). Of course no university hires explicitly on that metric, but many of the metrics they use are not far from that, mixing in quality of journals and norms for the field, etc. Each of the suggested actions, taken together, would seem to have a very positive improvement on the status quo. Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?"

meow1032 6y ago
Not OP here, but my issue with the recommendations are that they've pretty accurately listed a whole bunch of mostly structural problems with academia, but all of the suggestions boil down to "we all just need to try harder". You can say something like: "journals need to demand higher standards" but what incentive do they actually have to do so? Then you can counter with "scientists could vote with their feet", but what incentives do they have to do that?? You're asking people to consider seriously damaging their career for some nebulous quality metric. Frankly, having worked in academia long enough to see at least a couple shifts in culture, the only thing I can see that comes out of this is a couple more things get added on to the ever growing checklist of publishing a paper/submitting a grant application. I think we need to get away from the sort of thinking where large structural problems can be solved by tiny incremental improvements. If you really want to solve the problem, one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up.
- joppy 6y ago
  It seems to me, still, that a lot of these problems you bring up can be addressed by universities changing their hiring policies. Which makes sense: academics ultimately rely on universities for their income, and so it is the hiring policies which are setting the perverse incentives. And I don’t think changing hiring policies would be an incremental change, it would be a huge change (and not likely to be made by any university any time soon, since students rank universities on similar metrics to how universities hire staff — a prestigious university will lose prestige even if it changes its hiring policies for the better).
  - meow1032 6y ago
    > academics ultimately rely on universities for their income Sort of, a huge portion of income is from grants, particularly after the first few years from being hired. More importantly, a huge portion of the University income is from grants. When a researcher recieves a grant, there is an "overhead" percentage that goes to the University. Universities hire, in part, to maximize those overheads, which means getting the researchers with the best chance at getting big grants. Changing the hiring process may affect how PHD students act, but once they're "in the system", they are subject to all the same problematic incentives.

asddubs 6y ago

thank god we can eat a ton of salt again, to be honest I was doing it the whole time

jschwartzi 6y ago
I never stopped. The whole idea had a couple smells to it: * Maybe increases sodium intake is linked to poor health outcomes because highly processed foods are linked to poor health outcomes. We know that sodium content increases significantly during food processing and that most highly processed foods are really unhealthy. * Maybe people in early stages of renal failure are more likely to progress to a noticeable state if they consume lots of salt. Then it would stand to reason that people with healthy kidneys have nothing to worry about.

AnIdiotOnTheNet 6y ago

> The ketogenic diet is vilified in every mainstream media and most nutritional "science" publications; the headlines are rarely inline with the the actual results, but that's what people remember (Not that nutrition science is really science) Nutrition science is real science, but unfortunately any actual nutrition science you might accidentally hear about is overwhelmed by people trying to sell a lifestyle, a book about healthy eating, or your eyeballs (to advertisers). Ironically keto promoters are a big offender here themselves.

amanaplanacanal 6y ago
Most nutrition science is pretty bad. As is often said, correlation is not causation, but that's what most nutritional studies are. It's easy to study large populations and find correlations. You publish a paper, the media reports something, and that becomes the accepted wisdom. But you have no idea how to factor out all the possible confounders in your study. It's hard to do a study where you make a change to people's diet and see if that affects health outcomes, so it is rarely done. And really that's what you need to do to see what is really going on. So what we are left with is a bunch of associations that may or may not hold up.
exolymph 6y ago
Counterpoint: https://meaningness.com/nutrition https://meaningness.com/nutrition
- beagle3 6y ago
  This is an excellent summary, thanks!

Bobbcatt 6y ago

>"extraordinary claims require extraordinary evidence" And yet most people accept climate change as fact, even though the claim itself is so extraordinary that we as a species are not capable of producing enough evidence for it. You can't predict accurately within 1 degree celcius the temperature 7 days from now, but you want me to believe you when you try to do it for 50 years from now.

csours 6y ago

I think that pre-registering methods is important. I think that a requirement to publish results either way on any study that's been pre-registered is important.

xenocyon 6y ago

> The problem description (at least in the comic) is good. Any suggested action ... not so much. Well put. In theory, people like the idea of making science better. In practice, people don't like the idea of fewer papers, boring papers, ambiguity in hiring and tenure, uncertainty on financial return-on-investment, and less institutional/national pride talking points. The bad incentives and metrics we have haven't happened by chance - they have emerged from our collective desire for science to be useful, sexy, and a reliable function of money/effort spent.

clairity 6y ago

> 'And the other recommendations about fixing it are comparable to "lets solve evils of the US 2-party system! All we have to do is make those two parties vote to take away their power".' that's a bit unfair. yes, it does require that in some semblance, but that's not necessarily the one and only lever we have. most of us realistically expect, i hope, that we'd need a variety of political maneuvers to move our democracy toward a more representative and less insular direction. science is no different--it took many small steps to get into this situation, and will take many small steps to get out, any one of which will seem wholly incapable on its own.

0d9eooo 6y ago

I think the horse has left he barn in terms of the suggestions in the comic. I love the comic and the attention it's getting but the issues discussed are such a small part of the problem. Many of the comments here are on point. At this point in time, many of the people in power (full professors, administration, etc) see papers as essentially useless, in part because of all the problems being discussed here. p-hacking, guest authorship, etc and so forth and so on. They're seen as a dime a dozen and somewhat ignored. This might sound good until you understand that this means that they're dismissing the entirety of the scientific dialogue essentially, and that their alternative is grant money. My institution had training workshops for grad students where faculty would tell students that research is essentially worthless unless grant money is attached to it. The idea is that papers are a dime a dozen, and that if something is worthwhile, the feds will put their money where their mouth is so to speak. The problem with this, of course, is that the grant system is horribly nepotistic and distorted. The biggest predictor of successful grant application last time I looked (based on empirical research in peer-reviewed journals) is co-authoring papers with someone on the review panel. Grant receipt is only weakly related to citation metrics, and from personal experience I can say that institutions are constantly encouraging researchers to inflate grant costs to bring in more indirect costs. You could fix everything about journals and the publication process and it would do nothing about the shadow scientific world that exists in parallel that drives all the rest of the problems. Eliminate indirect funds, require tenure of all researchers, set aside funding mechanisms for researchers that aren't tied to specific grant applications, randomize grant rewards, depriortize journals, ... there's a lot of things that need to happen.