11 ms·

My wife and I were just talking about this yesterday. She is an extremely specialized surgeon and she is horrified by the academic publishing in her area of exp

by conorh 5y ago

My wife and I were just talking about this yesterday. She is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience. I'm not sure she'd agree with the reasons mentioned in this article here as the main issue - although we have personally seen less ethically motivated doctors publishing research to help line their pockets via the industry. In her opinion the main issue is that academic doctors HAVE to publish in order to advance their careers. So you end up with a never ending stream of terrible research, churned out by doctors that are not experts in doing research and that actually spend most of their time doing other things. This happens even at the best centers; at least in her area of expertise. I don't know what the solution is, but I do know that whenever she has needed treatment for herself she generally ignores most of the research and talks to the most experienced surgeon she can find.

rscho 5y ago
This is all true. Welcome to the real world. > academic doctors HAVE to publish in order to advance their careers Indeed. But the issue is what criteria will you use for promotion if not that? Don't even think of evaluating clinical results, that's even worse than what we have. > This happens even at the best centers This happens ESPECIALLY at the best centers. Because centers are in good part ranked according to citation counts. > I don't know what the solution is, but I do know that whenever she has needed treatment for herself she generally ignores most of the research and talks to the most experienced surgeon she can find. After 10+ years in the system, I'm in the same boat. And although I've given it a good thought, I don't see a solution either.
- derbOac 5y ago
  The truth this article touches on is that it's not even really about citation counts anymore, it's about dollars. Via drug companies, private grants, federal grants, any sorts of money coming in from outside the university or hospital. I've sat on promotion meetings where people aren't publishing, but are bringing in 2 million dollar grants and "it's all ok". This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way.
  - prox 5y ago
    Put controls in place. Monetary openness. Every dollar accounted for.
    - rscho 5y ago
      That's possibly the worst thing that could happen to science. Academic research should not be an industry. The fact that it is now considered as such in the western world will be our downfall.
      - prox 5y ago
        From the looks of it, it already seems like an industry even it tries to appear not to. Ofcourse not all sectors will have the issue, just some (medicine, high tech for example) Putting safeguards in place is a good thing. It’s like traffic rules that people need to adhere to.
        inkblotuniverse 5y ago
        Nothing will ever get done, but at least the busywork will be well documented!
  - wolverine876 5y ago
    Stop trying to apply the 'free market' or capitalism to every situation. The problem is more fundamental than your field: As long as those economic tools are treated as sacrosanct, universal solutions by society in general, your field inevitably will suffer. (They are very useful tools, but like very tool, not the solution to every problem.)
  - kafkaIncarnate 5y ago
    > This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way. A lot of the reasons of this distortion is due to the lack of funding from government in state universities which keeps going down as the cost of running a university keeps going up. Because of the rising costs, things get pushed to extremes and the accountants/administrators are put between a rock and a hard place. Depending on the university in question, it might not be just about "making money" but just not shutting the university down and trying to swim upstream in a never-ending battle.
- acover 5y ago
  Would estimating the prior help? Eg this study claims the sky is red and most people estimate that is wrong. Therefore this study is likely wrong and needs a lot of evidence to overcome the prior.
  - pfortuny 5y ago
    Yes but the thing is… papers are counted, not read. They are published to be counted, not read. Really.
- klabb3 5y ago
  > But the issue is what criteria will you use for promotion if not that? Dice roll? Alphabetical order? If it's gonna be arbitrary or close-to, it would be preferable for everyone if it didn't erode science as a side effect.
- BeFlatXIII 5y ago
  How is ranking by clinical outcomes “even worse” than grading by quantity of papers published?
  - rscho 5y ago
    Because it pushes us to treat cases where we expect a good outcome preferentially, i.e. biases the case mix towards easy patients and increases inequalities in access to healthcare.
austinjp 5y ago
> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience. Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse? I'm certainly not denying that she may be correct, and I'm not unquestioningly defending research evidence either. We've come a long way from "eminence-based medicine", where the senior clinician was impossible to challenge. Research has helped us move towards a more nuanced approach to care (I would say "holistic" but that word has become tarnished). EBM "phase one" alienated clinicians despite explicitly acknowledging clinician expertise, by overemphasizing evidence and undervaluing experience. EBM "phase two" aims to redress the balance, but perhaps the mistrust is too significant now. It would be a shame if so. Some reading: https://www.bmj.com/content/348/bmj.g3725 https://www.bmj.com/content/348/bmj.g3725 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence-based-medicine-2.0 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence...
- conorh 5y ago
  This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.
  - ModernMech 5y ago
    >I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too. And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...
Alex3917 5y ago
The pandemic was an interesting case study in this. In the sense that although the alternative medicine people who started writing about covid after the pandemic started were generally giving terrible advice, the alternative medicine people who were writing about how to survive a novel coronavirus epidemic before December 2019 were generally much more accurate than the mainstream medical people who only started learning about coronaviruses after the pandemic started. It turns out, being genuinely interested in something when there is zero money to be made is a much better predictor of success than having formal training and insider status.
- tangjurine 5y ago
  It would be cool if you could do an analysis of the alternative medicine people.
whatever1 5y ago
This is true in any field. Many publications are trash. It takes a lot of experience to be able to tell apart bad science from good science in a specific field. Every PhD knows how to read literature. But throw them outside their narrow line of expertise, and they will not be able to tell apart a good from a bad paper. It's the same reason you cannot trust your FB Karen who found a publication that supports her conspiracy theory. So yes your wife is right in looking for the most experienced person in the field for an opinion.
steve76 5y ago
deleted 5y ago
[deleted]
andy_ppp 5y ago
I question that Golang* is a good programming language but lots of expert coders disagree. While your theory has merit I wonder if at the top of surgery everyone has subtle and nuanced ideas to the point of when telling others of their experience that because these surgeons are so good their claims can’t be translated as easily as every expert has different ideas. I don’t doubt that there are some people as you say with a profit motive to bullshit everyone I just question the proportion given how bespoke being at the top of any industry might be. * sorry Golang people and anyone offended by this it’s just an example
- athrowaway3z 5y ago
  25 guys from IBM, 20 from Microsoft, 3 from Facebook, 10 from Google, 14 professors from different universities sit on a body that dictate how you're suppose to program. If you don't put 20 references in your source code to "The Gang of 4", you're credentials will never be enough to do anything but write doubly nested for loops. Luckily the IT industry can just ignore them and demonstrate something better without bothering 6 other organs to check if they are allowed to build it. But in truth i'm starting to worry about the Hollywood-NSA-Savethechildren 'consultants' influence on the laws that panders 'technical' solutions to make general computation a service instead of a human right.
kqr 5y ago
> never ending stream of terrible research When I was in elementary school, I saw the same thing. People forced to write essays on things they were not necessarily good at, just in order to advance to the next year. The result, of course, was a crapton of half-assed essays, many of which were bullshitted together not long before the deadline. I sincerely doubt even half of them where in any way useful whatsoever. At least in my school, few people improved their essay skills over time. Maybe they learned more technically impressive ways to bullshit, but their writing did not improve. (Some people's writing improved, but often they turned out to be writing things on their own interest in their spare time.) I know at the time I said to myself, "This is dumb, but it's just school. It's not like this for adults, of course. They have their shit together." Turns out many adults still do that! This is bad on so many levels it's bordering on unintentional evil. Someone should start a revolution.
- taylorius 5y ago
  Jeez, everyone here at level X are a bunch of clowns winging it. It's not like this for level X+1, of course. They have their shit together. :-) Then at some point it dawns that it's just clowns from bottom to top.
  - kqr 5y ago
    Exactly. That's the big mystery as far as I'm concerned: how the hell does the world work as well as it does, given how clueless the people running it are? I'm starting to suspect it speaks to the adaptability of people.
mihaaly 5y ago
Not knowing the medical field but being involved in research in technical fields before I can confirm the problem with the "HAVE to publish" phenomenon. I came accross sooo much long but barren papers that it was a tiresome and most times fruitless excercise just to follow proper scentific methodology in terms of being informed of related works that I am actually glad I do not need to live in the academic reasearch world anymore (I still do in industrial settings, "improperly", tied to developments). I also hated that I had to make up a quota even if the research did not yield interesting enough (part) results yet, figuring out something that could go into a paper, for the sake of writing a paper alone, not for communicating new knowledge yet. Formality not content ruled. (I just hope other places/institutions/fields do it better)
Ekaros 5y ago
Also if you are a effective surgeon, how much time you have or have to spend on writing and formulating scientific research? I wonder if there is addition of research to the old adage of teaching...
- ModernMech 5y ago
  So much of what you use today and the modern world runs on research and the scientific process in general, warts and all. Your comment is hubris at its finest, and insulting to researchers and teachers.

by conorh 5y ago

rscho 5y ago

This is all true. Welcome to the real world. > academic doctors HAVE to publish in order to advance their careers Indeed. But the issue is what criteria will you use for promotion if not that? Don't even think of evaluating clinical results, that's even worse than what we have. > This happens even at the best centers This happens ESPECIALLY at the best centers. Because centers are in good part ranked according to citation counts. > I don't know what the solution is, but I do know that whenever she has needed treatment for herself she generally ignores most of the research and talks to the most experienced surgeon she can find. After 10+ years in the system, I'm in the same boat. And although I've given it a good thought, I don't see a solution either.

derbOac 5y ago
The truth this article touches on is that it's not even really about citation counts anymore, it's about dollars. Via drug companies, private grants, federal grants, any sorts of money coming in from outside the university or hospital. I've sat on promotion meetings where people aren't publishing, but are bringing in 2 million dollar grants and "it's all ok". This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way.
- prox 5y ago
  Put controls in place. Monetary openness. Every dollar accounted for.
  - rscho 5y ago
    That's possibly the worst thing that could happen to science. Academic research should not be an industry. The fact that it is now considered as such in the western world will be our downfall.
    - prox 5y ago
      From the looks of it, it already seems like an industry even it tries to appear not to. Ofcourse not all sectors will have the issue, just some (medicine, high tech for example) Putting safeguards in place is a good thing. It’s like traffic rules that people need to adhere to.
      - inkblotuniverse 5y ago
        Nothing will ever get done, but at least the busywork will be well documented!
- wolverine876 5y ago
  Stop trying to apply the 'free market' or capitalism to every situation. The problem is more fundamental than your field: As long as those economic tools are treated as sacrosanct, universal solutions by society in general, your field inevitably will suffer. (They are very useful tools, but like very tool, not the solution to every problem.)
- kafkaIncarnate 5y ago
  > This incentive structure distorts research. What to do about it? I'm not exactly sure but having university administrations who see research as a means to make money, rather than an end in itself, is not the way. A lot of the reasons of this distortion is due to the lack of funding from government in state universities which keeps going down as the cost of running a university keeps going up. Because of the rising costs, things get pushed to extremes and the accountants/administrators are put between a rock and a hard place. Depending on the university in question, it might not be just about "making money" but just not shutting the university down and trying to swim upstream in a never-ending battle.
acover 5y ago
Would estimating the prior help? Eg this study claims the sky is red and most people estimate that is wrong. Therefore this study is likely wrong and needs a lot of evidence to overcome the prior.
- pfortuny 5y ago
  Yes but the thing is… papers are counted, not read. They are published to be counted, not read. Really.
klabb3 5y ago
> But the issue is what criteria will you use for promotion if not that? Dice roll? Alphabetical order? If it's gonna be arbitrary or close-to, it would be preferable for everyone if it didn't erode science as a side effect.
BeFlatXIII 5y ago
How is ranking by clinical outcomes “even worse” than grading by quantity of papers published?
- rscho 5y ago
  Because it pushes us to treat cases where we expect a good outcome preferentially, i.e. biases the case mix towards easy patients and increases inequalities in access to healthcare.

austinjp 5y ago

> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience. Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse? I'm certainly not denying that she may be correct, and I'm not unquestioningly defending research evidence either. We've come a long way from "eminence-based medicine", where the senior clinician was impossible to challenge. Research has helped us move towards a more nuanced approach to care (I would say "holistic" but that word has become tarnished). EBM "phase one" alienated clinicians despite explicitly acknowledging clinician expertise, by overemphasizing evidence and undervaluing experience. EBM "phase two" aims to redress the balance, but perhaps the mistrust is too significant now. It would be a shame if so. Some reading: https://www.bmj.com/content/348/bmj.g3725 https://www.bmj.com/content/348/bmj.g3725 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence-based-medicine-2.0 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence...

conorh 5y ago
This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.
- ModernMech 5y ago
  >I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too. And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...

Alex3917 5y ago

The pandemic was an interesting case study in this. In the sense that although the alternative medicine people who started writing about covid after the pandemic started were generally giving terrible advice, the alternative medicine people who were writing about how to survive a novel coronavirus epidemic before December 2019 were generally much more accurate than the mainstream medical people who only started learning about coronaviruses after the pandemic started. It turns out, being genuinely interested in something when there is zero money to be made is a much better predictor of success than having formal training and insider status.

tangjurine 5y ago
It would be cool if you could do an analysis of the alternative medicine people.

whatever1 5y ago

This is true in any field. Many publications are trash. It takes a lot of experience to be able to tell apart bad science from good science in a specific field. Every PhD knows how to read literature. But throw them outside their narrow line of expertise, and they will not be able to tell apart a good from a bad paper. It's the same reason you cannot trust your FB Karen who found a publication that supports her conspiracy theory. So yes your wife is right in looking for the most experienced person in the field for an opinion.

steve76 5y ago

deleted 5y ago

[deleted]

andy_ppp 5y ago

I question that Golang* is a good programming language but lots of expert coders disagree. While your theory has merit I wonder if at the top of surgery everyone has subtle and nuanced ideas to the point of when telling others of their experience that because these surgeons are so good their claims can’t be translated as easily as every expert has different ideas. I don’t doubt that there are some people as you say with a profit motive to bullshit everyone I just question the proportion given how bespoke being at the top of any industry might be. * sorry Golang people and anyone offended by this it’s just an example

athrowaway3z 5y ago
25 guys from IBM, 20 from Microsoft, 3 from Facebook, 10 from Google, 14 professors from different universities sit on a body that dictate how you're suppose to program. If you don't put 20 references in your source code to "The Gang of 4", you're credentials will never be enough to do anything but write doubly nested for loops. Luckily the IT industry can just ignore them and demonstrate something better without bothering 6 other organs to check if they are allowed to build it. But in truth i'm starting to worry about the Hollywood-NSA-Savethechildren 'consultants' influence on the laws that panders 'technical' solutions to make general computation a service instead of a human right.

kqr 5y ago

> never ending stream of terrible research When I was in elementary school, I saw the same thing. People forced to write essays on things they were not necessarily good at, just in order to advance to the next year. The result, of course, was a crapton of half-assed essays, many of which were bullshitted together not long before the deadline. I sincerely doubt even half of them where in any way useful whatsoever. At least in my school, few people improved their essay skills over time. Maybe they learned more technically impressive ways to bullshit, but their writing did not improve. (Some people's writing improved, but often they turned out to be writing things on their own interest in their spare time.) I know at the time I said to myself, "This is dumb, but it's just school. It's not like this for adults, of course. They have their shit together." Turns out many adults still do that! This is bad on so many levels it's bordering on unintentional evil. Someone should start a revolution.

taylorius 5y ago
Jeez, everyone here at level X are a bunch of clowns winging it. It's not like this for level X+1, of course. They have their shit together. :-) Then at some point it dawns that it's just clowns from bottom to top.
- kqr 5y ago
  Exactly. That's the big mystery as far as I'm concerned: how the hell does the world work as well as it does, given how clueless the people running it are? I'm starting to suspect it speaks to the adaptability of people.

mihaaly 5y ago

Not knowing the medical field but being involved in research in technical fields before I can confirm the problem with the "HAVE to publish" phenomenon. I came accross sooo much long but barren papers that it was a tiresome and most times fruitless excercise just to follow proper scentific methodology in terms of being informed of related works that I am actually glad I do not need to live in the academic reasearch world anymore (I still do in industrial settings, "improperly", tied to developments). I also hated that I had to make up a quota even if the research did not yield interesting enough (part) results yet, figuring out something that could go into a paper, for the sake of writing a paper alone, not for communicating new knowledge yet. Formality not content ruled. (I just hope other places/institutions/fields do it better)

Ekaros 5y ago

Also if you are a effective surgeon, how much time you have or have to spend on writing and formulating scientific research? I wonder if there is addition of research to the old adage of teaching...

ModernMech 5y ago
So much of what you use today and the modern world runs on research and the scientific process in general, warts and all. Your comment is hubris at its finest, and insulting to researchers and teachers.