5 ms·

Psychologists Welcome Analysis Casting Doubt on Studies

37 points by evolve2k 11y ago

evolve2k 11y ago
"One piece of evidence that preregistration can act as a strong corrective comes from clinicaltrials.gov, a registry of publicly and privately funded clinical studies involving human subjects. Before 2000, when the site was established, 57 percent of large clinical trials funded by the National Heart, Lung and Blood Institute showed a significant benefit of drugs or other intervention, according to a recent analysis published in the journal PLOS One. After the registry was put in place, only 2 percent of such trials found a clear benefit."
- jMyles 11y ago
  I think it's healthy to suspect that the entire pharmacopoeia-industrial-complex has a similar rate of veracity.
  - ArekDymalski 11y ago
    I'm absolutely sure about that. As a student I was working (as a technician) on many medical science conferences, so I had an opportunity to see the "bleeding edge" research presentations. I was shocked by the level of confidence demonstrated by "scientists" who promoted results of their N=5 experiments (without a comparison group) as "convincing". If all therapies are based on such research, we have to be happy that our bodies are impressively self-repairing machines.
- var_eps 11y ago
  I'd recommend Irving Kirsch's __Emperor's new drugs: Exploring the antidepressant myth__ as an accessible introduction to this topic. The book goes beyond statistics and draws attention to "clinical significance" that is crucial to keep in mind when evaluating drugs. Statistical significance helps to inform whether a metric has changed; clinical significance tells whether it actually mattered from a patient's perspective.
  - keithflower 11y ago
    Dr. Kirsch's (a psychologist) work has been valuable, with some justified criticisms of antidepressant therapy and their trials. But his work did not show that antidepressants are clinically ineffective. Dr. Kirsch's analysis showed that for mild depression treated with antidepressants, there is in fact an improvement in symptoms of depression based on rating scales like the Hamilton Depression Scale - but for mild depression, this improvement in the numbers doesn't necessarily translate into what you correctly identify as clinical significance. However, Dr. Kirsch's own "cutoff" for whether or not such a numerical increase constitutes "clinical significance" or not is itself just an arbitrary number that itself has been criticized. Also, Kirsch's own metaanalyses (and other metaanalyses) show clearly that for moderate and severe depression (and other conditions), antidepressants are indeed clinically effective (in other words, yes, the benefits actually "mattered" from a patient's perspective), and, importantly, they are highly effective at preventing relapse. They work, and work well. In addition, there are a variety of reasons why many short-term clinical trials of antidepressants (which have made up the bulk of clinical trials of SSRIs, for example) are entirely different from the way physicians prescribe antidepressants (and other medications) clinically, and may underestimate their benefit even for cases of mild depression. The beneficial effects of antidepressants increase with longer exposure (i.e. exceeding the length of a six week clinical drug study), probably because of downstream neurotrophic factor(s) increases. Many trials of antidepressants limited dose adjustment or dose adjustment rate. Many clinical trials have excluded patients with severe symptoms or dual diagnoses because inclusion introduces a safety issue or may introduce too many variables. Also, physicians will switch patients to another antidepressant if there is no sign of early efficacy, and while one antidepressant may work very well (probably as a result of genetics), another will not work at all. Most studies focusing on one drug are not going to switch to another medication in the same class (for example, a serotonin reuptake inhibitor) during the trial, even though that's routinely done in clinical practice with good result. Antidepressants work. Psychotherapy can also be helpful as a sole treatment for mild depression (although please note that psychotherapy has its own set of risks, costs, and drawbacks), and patients often get additional benefit by engaging in psychotherapy along with medication treatment. However, a psychologist or counselor treating a moderately to severely depressed person who fails to refer that patient for evaluation for treatment with an antidepressant would be risking a malpractice suit.
bobcostas55 11y ago
How much of this is the result of outright fraud? I know Hanlon's razor and all, but I find it extremely difficult to believe that virtually everybody in this field is incompetent. It seems like the majority of these highly intelligent people, despite spending a decade in training, were essentially statistically illiterate, even though statistics are absolutely central to their field. That's not a catastrophe in itself, you can always hit up the "real" statistician two floors down...but it seems like they didn't want to bother with that. Did they overestimate their ability or did they know they were serving up shitty research? The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)...I'm not sure how to interpret this. Did they get lucky and stumble upon the one area of psychological research that had a "high yield"? Did they implement a superior research methodology that nobody could/wanted to imitate?
- jMyles 11y ago
  I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.
- mettamage 11y ago
  Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.
- DanBC 11y ago
  Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary. http://web.archive.org/web/20150115080542/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://web.archive.org/web/20150115080542/http://wjh.harvard...
  - 11y ago
dominotw 11y ago
At the risk of sounding really stupid, What is the crowning achievement of psychology ? What gives this field the legitimacy that it enjoys ?
- SolaceQuantum 11y ago
  Well, we we now understand a lot more about how people think than before. We're finally past that id-ego-superego and black box theory bullshit (although some people still ascribe to freudian ideas). There is real treatment for people with mental illnesses other than hospitalization and drugs (e.g. people with personality disorders, dissociation disorders, etc. can now receive structured talk therapies) The crowning achievement would probably be the ability for people who are mentally ill to receive help and understanding from trained professionals rather than be considered possessed, needing drugs, shoved into insane asylums, or eugenics'd.
- chubot 11y ago
  I would say that one of the most important "meta" ideas, that comes up in many places, is that your conscious mind and rationality are largely a veneer/illusion over an "animal" brain. Emotion comes first, then thought. This has consequences in the real world, including economic ones. And it was something that people got wrong -- e.g. Enlightenment thinkers overstated how different and privileged humans are in this respect. Take the example of optical illusions. Even though I can tell how the illusion works, and you rationally know that two lines are the same length, and you can literally measure it yourself with a ruler -- your mind will still see them as different [1]. This is a benign example, but there are lots of other examples where "knowing" that you are wrong doesn't prevent you from acting as if you held the wrong beliefs. If you are an investor, you should be familiar with the concept of "loss aversion", which is easily verified with an experiment and replicated. Daniel Kahneman won a (deserved) Nobel Prize for many interesting phenonema with real economic consequences. Intelligent people are always questioning their psychology: http://www.paulgraham.com/swan.html http://www.paulgraham.com/swan.html The first rule I knew intellectually, but didn't really grasp till it happened to us I don't know whether these expectations are innate or learned, but whatever the cause, we are just not prepared for the 1000x variation in outcomes that one finds in startup investing. I assert that certain ideas from psychology have permeated into the public consciousness -- and 200 or 300 years ago, you would not hear non-psychologists talking in this fashion. You can do very repeatable psychological experiments on how people perceive quantities of numbers and so forth. For evolutionary reasons, there are lots of quick "short circuits" in your brain that perform resource allocation and risk assessment, but they are not adapted for the modern world. [1] https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion
deleted 11y ago
[deleted]

37 points by evolve2k 11y ago

"One piece of evidence that preregistration can act as a strong corrective comes from clinicaltrials.gov, a registry of publicly and privately funded clinical studies involving human subjects. Before 2000, when the site was established, 57 percent of large clinical trials funded by the National Heart, Lung and Blood Institute showed a significant benefit of drugs or other intervention, according to a recent analysis published in the journal PLOS One. After the registry was put in place, only 2 percent of such trials found a clear benefit."

jMyles 11y ago
I think it's healthy to suspect that the entire pharmacopoeia-industrial-complex has a similar rate of veracity.
- ArekDymalski 11y ago
  I'm absolutely sure about that. As a student I was working (as a technician) on many medical science conferences, so I had an opportunity to see the "bleeding edge" research presentations. I was shocked by the level of confidence demonstrated by "scientists" who promoted results of their N=5 experiments (without a comparison group) as "convincing". If all therapies are based on such research, we have to be happy that our bodies are impressively self-repairing machines.
var_eps 11y ago
I'd recommend Irving Kirsch's __Emperor's new drugs: Exploring the antidepressant myth__ as an accessible introduction to this topic. The book goes beyond statistics and draws attention to "clinical significance" that is crucial to keep in mind when evaluating drugs. Statistical significance helps to inform whether a metric has changed; clinical significance tells whether it actually mattered from a patient's perspective.
- keithflower 11y ago
  Dr. Kirsch's (a psychologist) work has been valuable, with some justified criticisms of antidepressant therapy and their trials. But his work did not show that antidepressants are clinically ineffective. Dr. Kirsch's analysis showed that for mild depression treated with antidepressants, there is in fact an improvement in symptoms of depression based on rating scales like the Hamilton Depression Scale - but for mild depression, this improvement in the numbers doesn't necessarily translate into what you correctly identify as clinical significance. However, Dr. Kirsch's own "cutoff" for whether or not such a numerical increase constitutes "clinical significance" or not is itself just an arbitrary number that itself has been criticized. Also, Kirsch's own metaanalyses (and other metaanalyses) show clearly that for moderate and severe depression (and other conditions), antidepressants are indeed clinically effective (in other words, yes, the benefits actually "mattered" from a patient's perspective), and, importantly, they are highly effective at preventing relapse. They work, and work well. In addition, there are a variety of reasons why many short-term clinical trials of antidepressants (which have made up the bulk of clinical trials of SSRIs, for example) are entirely different from the way physicians prescribe antidepressants (and other medications) clinically, and may underestimate their benefit even for cases of mild depression. The beneficial effects of antidepressants increase with longer exposure (i.e. exceeding the length of a six week clinical drug study), probably because of downstream neurotrophic factor(s) increases. Many trials of antidepressants limited dose adjustment or dose adjustment rate. Many clinical trials have excluded patients with severe symptoms or dual diagnoses because inclusion introduces a safety issue or may introduce too many variables. Also, physicians will switch patients to another antidepressant if there is no sign of early efficacy, and while one antidepressant may work very well (probably as a result of genetics), another will not work at all. Most studies focusing on one drug are not going to switch to another medication in the same class (for example, a serotonin reuptake inhibitor) during the trial, even though that's routinely done in clinical practice with good result. Antidepressants work. Psychotherapy can also be helpful as a sole treatment for mild depression (although please note that psychotherapy has its own set of risks, costs, and drawbacks), and patients often get additional benefit by engaging in psychotherapy along with medication treatment. However, a psychologist or counselor treating a moderately to severely depressed person who fails to refer that patient for evaluation for treatment with an antidepressant would be risking a malpractice suit.

bobcostas55 11y ago

How much of this is the result of outright fraud? I know Hanlon's razor and all, but I find it extremely difficult to believe that virtually everybody in this field is incompetent. It seems like the majority of these highly intelligent people, despite spending a decade in training, were essentially statistically illiterate, even though statistics are absolutely central to their field. That's not a catastrophe in itself, you can always hit up the "real" statistician two floors down...but it seems like they didn't want to bother with that. Did they overestimate their ability or did they know they were serving up shitty research? The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)...I'm not sure how to interpret this. Did they get lucky and stumble upon the one area of psychological research that had a "high yield"? Did they implement a superior research methodology that nobody could/wanted to imitate?

jMyles 11y ago
I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.
mettamage 11y ago
Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.
DanBC 11y ago
Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary. http://web.archive.org/web/20150115080542/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://web.archive.org/web/20150115080542/http://wjh.harvard...
- 11y ago

dominotw 11y ago

At the risk of sounding really stupid, What is the crowning achievement of psychology ? What gives this field the legitimacy that it enjoys ?

SolaceQuantum 11y ago
Well, we we now understand a lot more about how people think than before. We're finally past that id-ego-superego and black box theory bullshit (although some people still ascribe to freudian ideas). There is real treatment for people with mental illnesses other than hospitalization and drugs (e.g. people with personality disorders, dissociation disorders, etc. can now receive structured talk therapies) The crowning achievement would probably be the ability for people who are mentally ill to receive help and understanding from trained professionals rather than be considered possessed, needing drugs, shoved into insane asylums, or eugenics'd.
chubot 11y ago
I would say that one of the most important "meta" ideas, that comes up in many places, is that your conscious mind and rationality are largely a veneer/illusion over an "animal" brain. Emotion comes first, then thought. This has consequences in the real world, including economic ones. And it was something that people got wrong -- e.g. Enlightenment thinkers overstated how different and privileged humans are in this respect. Take the example of optical illusions. Even though I can tell how the illusion works, and you rationally know that two lines are the same length, and you can literally measure it yourself with a ruler -- your mind will still see them as different [1]. This is a benign example, but there are lots of other examples where "knowing" that you are wrong doesn't prevent you from acting as if you held the wrong beliefs. If you are an investor, you should be familiar with the concept of "loss aversion", which is easily verified with an experiment and replicated. Daniel Kahneman won a (deserved) Nobel Prize for many interesting phenonema with real economic consequences. Intelligent people are always questioning their psychology: http://www.paulgraham.com/swan.html http://www.paulgraham.com/swan.html The first rule I knew intellectually, but didn't really grasp till it happened to us I don't know whether these expectations are innate or learned, but whatever the cause, we are just not prepared for the 1000x variation in outcomes that one finds in startup investing. I assert that certain ideas from psychology have permeated into the public consciousness -- and 200 or 300 years ago, you would not hear non-psychologists talking in this fashion. You can do very repeatable psychological experiments on how people perceive quantities of numbers and so forth. For evolutionary reasons, there are lots of quick "short circuits" in your brain that perform resource allocation and risk assessment, but they are not adapted for the modern world. [1] https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion

deleted 11y ago

[deleted]