7 ms·

> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and confli

> My wife ... is an extremely specialized surgeon and she is horrified by the academic publishing in her area of expertise. It is often utterly wrong and conflicts directly with her experience. Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse? I'm certainly not denying that she may be correct, and I'm not unquestioningly defending research evidence either. We've come a long way from "eminence-based medicine", where the senior clinician was impossible to challenge. Research has helped us move towards a more nuanced approach to care (I would say "holistic" but that word has become tarnished). EBM "phase one" alienated clinicians despite explicitly acknowledging clinician expertise, by overemphasizing evidence and undervaluing experience. EBM "phase two" aims to redress the balance, but perhaps the mistrust is too significant now. It would be a shame if so. Some reading: https://www.bmj.com/content/348/bmj.g3725 https://www.bmj.com/content/348/bmj.g3725 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence-based-medicine-2.0 https://www.cebm.ox.ac.uk/news/views/it-is-time-for-evidence...

conorh 5y ago
This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.
- ModernMech 5y ago
  >I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too. And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...
XMPPwocky 5y ago
"how does someone in her position know they're right and the research is wrong, and not the reverse?" This is where you pull out the rusty ol' tools of Figuring Things Out From Evidence, which mostly boil down to: come up with your own theories and listen to theories from others think really hard about which one explains historical evidence the most consistently. think about experiments to make sure theories aren't overfit to historical evidence conduct those experiments, and "penalize" theories which don't explain the experimental results these tools are strange things because they (noisily, but surprisingly reliably when applied truly honestly and openly) predict which theories are predictive of future experimental results. and that's all you want out of a theory, really.
- XMPPwocky 5y ago
  and why should anybody else trust her? well, if she just says "actually, I'm right and the science is wrong", nobody should! Anybody can say that, regardless of whether they're right or not. I could say that! if she says, though, "actually, I'm right and the science is wrong though, and here's why:" and explains her entire process, what results she's seen hands-on in her own surgeries, and is careful to also include anything she saw that isn't consistent with her theory..., shows her reasoning, and shows her work... that's convincing and persuasive. It's also not easy, and requires lots of work. So does figuring out things in general. And that's called "an argument".
  - ironmagma 5y ago
    It’s sometimes called reasoning from first principles too.
  - Dylan16807 5y ago
    And if one bad paper causes a thousand people to have to put in all that work to rebut it, we have a big problem with the system.
DVassallo 5y ago
You need just one contradicting observation to invalidate something, while a million consistent observations can never fully prove it. (Paraphrasing Karl Popper, mentioned in the article).
- adwn 5y ago
  > You need just one contradicting observation to invalidate something No, that's wrong. There are several reasons why a negative result doesn't necessarily disprove a theory or a hypothesis: measurement uncertainties/noise, experimental errors, software bugs, bit flips, ...
- renewiltord 5y ago
  Right, but you only have aggregates here moving the needle. For instance, your experience is that intervention A leads to >90% mobility elbow post-surgery in 90% of patients and intervention B leads to 80% mobility in 90% of patients. What epistemological techniques do you use to distinguish intervention A from intervention B and how do you distinguish your experience from the paper that shows the opposite result? The number of variables are large: including, distressingly, your skill at technique A and technique B. Is the difference in the outcome merely your lack of skill in one? Is the difference merely the lack of skill of the guys observed by the paper? My parents are also experienced surgeons and the normalizing method they use is to share their findings with fellow surgeons. Patients where they work are much more amenable to having photographs of their surgeries used as discussion material. Medical science is a primitive science and it is practised with primitive statistics, so most evidence is just missing. Lots of surgical techniques are discussed based on too little evidence with massive true error bars.
- raverbashing 5y ago
  And what's puzzling to me is how so many people take Popper's theoretical discussions to be of such validity in the real world The experimentalist world is messy and full of hidden dependencies. (And to be fair it's usually the people with less experimental experience that take him the most seriously)
- dennis_jeeves1 5y ago
AniseAbyss 5y ago
Reminds me of that poor guy who told surgeons they should wash their hands. They didn't take kindly to that.
- LeonB 5y ago
  I’d give the doctor above a little more credit than that, and also would expect the papers she dislikes to be far less impactful than that. Clinicians I know who get frustrated with academic research that draws conclusions about clinical work are criticising papers that are more like this: (I’m exaggerating for effect, but not exaggerating a lot) “People think this clinical technique is effective. But when some academics with very little clinical experience performed a rough approximation of the technique, in the wrong area, with the wrong tools, in the wrong way, on a small number of randomly chosen patients, they found it performed no better than placebo. Therefore no one should use that technique on any patient.”
- IG_Semmelweiss 5y ago
  That would be Ignaz Semmelweiss. I think you are understating his actual life outcome for having challenged the medical establishment with hard proof. Poor doesn't begin to describe it. https://en.wikipedia.org/wiki/Ignaz_Semmelweiss https://en.wikipedia.org/wiki/Ignaz_Semmelweiss
mirntyfirty 5y ago
As a statistician, I see a constant flood of incorrect statistical conclusions and poorly designed experiments.
- entropicdrifter 5y ago
  The fact of the matter is that the vast majority of people are just barely competent at even bullshit paper-pushing jobs, let alone highly technical work. The reason the majority of people end up with higher degrees and the jobs that require them has more to do with wealth/class and social privilege than actual competence or intelligence. It's almost to the point where people who are actually good at something end up with an appropriate line of work almost by coincidence. That's the natural outcome when you organize your society by wealth instead of skill.
rscho 5y ago
> Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse? She doesn't know, and that's the problem with medical research: being a specialized surgeon is like a carpenter who's made hundreds of a particular kind of furniture. Then a scientist comes along, saying we should change how we do it. As an experienced carpenter, you know that most of those scientists actually have very little practical experience, because they used that time to publish and are actually very bad, practically speaking. And therefore oftentimes reach nonsensical conclusions. So, what to do? You go ask the most practically experienced guy you can find, hoping he'll be able to guide you. Repeat 100x/year. I'm not exaggerating: some of our academic practitioners who've published a lot are really among the worst practitioners I know, and do kill people who should have lived on a regular basis! Ironically, some of them are respected professors and patients often seek their care thinking it'll be better. One of them is a surgeon and has received an award for outstanding teaching. He's the absolute worst surgeon I know. So, no easy conclusion here...
- dr_dshiv 5y ago
  > scientist comes along, saying we should change how we do it That doesn’t sound like science. An example or pointer would be helpful.
  - rscho 5y ago
    Remember the start of the pandemic? Some prof published that we should withold steroids and intubate ASAP on the basis of preliminary evidence, throwing basically everything we knew about ARDS out of the window. Most reasonable ICU guys were skeptical on the basis of their own practical experience, but had no choice because who would dare contradict science, right?? We very likely killed many people that way. What do we do now? The exact opposite, and much closer to usual ARDS practices. You can argue that wasn't a mistake because it followed the process of scientific discovery, but it's nevertheless a tragic example of experience trumping the advice of the scientific community.
    - dr_dshiv 5y ago
derbOac 5y ago
Not a surgeon, but in my area often it's not even so much that the results conflict with experience in a clinician versus researcher sense (although they often do), but that the research design itself is obviously flawed, in the sense that the assumptions made in the design, or the corners that are cut, or the way the data is massaged, or whatnot, are completely unrealistic, and the conclusions being drawn are completely inappropriate compared to what was done. It might even be that you don't have to be an experienced clinician to tell something is wrong, but an experienced clinician would pick up on the problems immediately just through familiarity. Feynman used the term "cargo cult science" and in my experience it applies to EBM more than any other area I've encountered.
RandomLensman 5y ago
Cannot speak for to this situation in particular, but some highly specialized surgeons track results, in- and out-going data etc. very thoroughly and might therefore actually have better visibility than a researcher looking at a small data set.

conorh 5y ago

This is an excellent question (and reading through those articles now, thank you). I know she thinks about this a lot because her patients bring her research which is obviously broken, but why should they trust her? In her specific case she has done thousands and thousands of an extremely specialized operation - one of very few people in the world to have done that many. She keeps an open mind of course, but she also sees a lot of harm done. Medical research is hard, lots of confounding factors, and almost all the time in her particular area researchers have a very small n, so they draw conclusions because of the small n (and because they are often just not good at research, they are doctors!). I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too.

ModernMech 5y ago
>I'm sure the scientific method will come through in the long run, that's my opinion, but along the way it is doing a lot of harm too. And that's the rub. Most people who try to improve the system reinvent it with the same or worse flaws. You'll notice for all the hand wringing and complaining here about how flawed the process is, no one really has put forth a suggestion on how to fix it. A lot of people here are not even part of the process and looking in from the outside, so they don't even understand the incentives involved enough to fix it. Hence the endless complaining...

XMPPwocky 5y ago

"how does someone in her position know they're right and the research is wrong, and not the reverse?" This is where you pull out the rusty ol' tools of Figuring Things Out From Evidence, which mostly boil down to: come up with your own theories and listen to theories from others think really hard about which one explains historical evidence the most consistently. think about experiments to make sure theories aren't overfit to historical evidence conduct those experiments, and "penalize" theories which don't explain the experimental results these tools are strange things because they (noisily, but surprisingly reliably when applied truly honestly and openly) predict which theories are predictive of future experimental results. and that's all you want out of a theory, really.

XMPPwocky 5y ago
and why should anybody else trust her? well, if she just says "actually, I'm right and the science is wrong", nobody should! Anybody can say that, regardless of whether they're right or not. I could say that! if she says, though, "actually, I'm right and the science is wrong though, and here's why:" and explains her entire process, what results she's seen hands-on in her own surgeries, and is careful to also include anything she saw that isn't consistent with her theory..., shows her reasoning, and shows her work... that's convincing and persuasive. It's also not easy, and requires lots of work. So does figuring out things in general. And that's called "an argument".
- ironmagma 5y ago
  It’s sometimes called reasoning from first principles too.
- Dylan16807 5y ago
  And if one bad paper causes a thousand people to have to put in all that work to rebut it, we have a big problem with the system.

DVassallo 5y ago

You need just one contradicting observation to invalidate something, while a million consistent observations can never fully prove it. (Paraphrasing Karl Popper, mentioned in the article).

adwn 5y ago
> You need just one contradicting observation to invalidate something No, that's wrong. There are several reasons why a negative result doesn't necessarily disprove a theory or a hypothesis: measurement uncertainties/noise, experimental errors, software bugs, bit flips, ...
renewiltord 5y ago
Right, but you only have aggregates here moving the needle. For instance, your experience is that intervention A leads to >90% mobility elbow post-surgery in 90% of patients and intervention B leads to 80% mobility in 90% of patients. What epistemological techniques do you use to distinguish intervention A from intervention B and how do you distinguish your experience from the paper that shows the opposite result? The number of variables are large: including, distressingly, your skill at technique A and technique B. Is the difference in the outcome merely your lack of skill in one? Is the difference merely the lack of skill of the guys observed by the paper? My parents are also experienced surgeons and the normalizing method they use is to share their findings with fellow surgeons. Patients where they work are much more amenable to having photographs of their surgeries used as discussion material. Medical science is a primitive science and it is practised with primitive statistics, so most evidence is just missing. Lots of surgical techniques are discussed based on too little evidence with massive true error bars.
raverbashing 5y ago
And what's puzzling to me is how so many people take Popper's theoretical discussions to be of such validity in the real world The experimentalist world is messy and full of hidden dependencies. (And to be fair it's usually the people with less experimental experience that take him the most seriously)
dennis_jeeves1 5y ago

AniseAbyss 5y ago

Reminds me of that poor guy who told surgeons they should wash their hands. They didn't take kindly to that.

LeonB 5y ago
I’d give the doctor above a little more credit than that, and also would expect the papers she dislikes to be far less impactful than that. Clinicians I know who get frustrated with academic research that draws conclusions about clinical work are criticising papers that are more like this: (I’m exaggerating for effect, but not exaggerating a lot) “People think this clinical technique is effective. But when some academics with very little clinical experience performed a rough approximation of the technique, in the wrong area, with the wrong tools, in the wrong way, on a small number of randomly chosen patients, they found it performed no better than placebo. Therefore no one should use that technique on any patient.”
IG_Semmelweiss 5y ago
That would be Ignaz Semmelweiss. I think you are understating his actual life outcome for having challenged the medical establishment with hard proof. Poor doesn't begin to describe it. https://en.wikipedia.org/wiki/Ignaz_Semmelweiss https://en.wikipedia.org/wiki/Ignaz_Semmelweiss

mirntyfirty 5y ago

As a statistician, I see a constant flood of incorrect statistical conclusions and poorly designed experiments.

entropicdrifter 5y ago
The fact of the matter is that the vast majority of people are just barely competent at even bullshit paper-pushing jobs, let alone highly technical work. The reason the majority of people end up with higher degrees and the jobs that require them has more to do with wealth/class and social privilege than actual competence or intelligence. It's almost to the point where people who are actually good at something end up with an appropriate line of work almost by coincidence. That's the natural outcome when you organize your society by wealth instead of skill.

rscho 5y ago

> Genuine question: how does someone in her position know they're right and the research is wrong, and not the reverse? She doesn't know, and that's the problem with medical research: being a specialized surgeon is like a carpenter who's made hundreds of a particular kind of furniture. Then a scientist comes along, saying we should change how we do it. As an experienced carpenter, you know that most of those scientists actually have very little practical experience, because they used that time to publish and are actually very bad, practically speaking. And therefore oftentimes reach nonsensical conclusions. So, what to do? You go ask the most practically experienced guy you can find, hoping he'll be able to guide you. Repeat 100x/year. I'm not exaggerating: some of our academic practitioners who've published a lot are really among the worst practitioners I know, and do kill people who should have lived on a regular basis! Ironically, some of them are respected professors and patients often seek their care thinking it'll be better. One of them is a surgeon and has received an award for outstanding teaching. He's the absolute worst surgeon I know. So, no easy conclusion here...

dr_dshiv 5y ago
> scientist comes along, saying we should change how we do it That doesn’t sound like science. An example or pointer would be helpful.
- rscho 5y ago
  Remember the start of the pandemic? Some prof published that we should withold steroids and intubate ASAP on the basis of preliminary evidence, throwing basically everything we knew about ARDS out of the window. Most reasonable ICU guys were skeptical on the basis of their own practical experience, but had no choice because who would dare contradict science, right?? We very likely killed many people that way. What do we do now? The exact opposite, and much closer to usual ARDS practices. You can argue that wasn't a mistake because it followed the process of scientific discovery, but it's nevertheless a tragic example of experience trumping the advice of the scientific community.
  - dr_dshiv 5y ago

derbOac 5y ago

Not a surgeon, but in my area often it's not even so much that the results conflict with experience in a clinician versus researcher sense (although they often do), but that the research design itself is obviously flawed, in the sense that the assumptions made in the design, or the corners that are cut, or the way the data is massaged, or whatnot, are completely unrealistic, and the conclusions being drawn are completely inappropriate compared to what was done. It might even be that you don't have to be an experienced clinician to tell something is wrong, but an experienced clinician would pick up on the problems immediately just through familiarity. Feynman used the term "cargo cult science" and in my experience it applies to EBM more than any other area I've encountered.

RandomLensman 5y ago

Cannot speak for to this situation in particular, but some highly specialized surgeons track results, in- and out-going data etc. very thoroughly and might therefore actually have better visibility than a researcher looking at a small data set.