9 ms·

Either this will win the Nobel Prize, or it won't hold up. (Guess which one is more likely.) A few reasons I don't think it will hold up: - this is a universit

Either this will win the Nobel Prize, or it won't hold up. (Guess which one is more likely.) A few reasons I don't think it will hold up: - this is a university press release - N = 21 - no control group - no blinding - university press release doesn't even claim it's a new approach, just incremental improvement on an existing approach - the paper (link below) is about evaluating safety, feasibility, and "preliminary efficacy" (i.e., they're not advertising this as a robust result) Paper available here: https://ajp.psychiatryonline.org/doi/10.1176/appi.ajp.2019.19070720 https://ajp.psychiatryonline.org/doi/10.1176/appi.ajp.2019.1... https://www.biorxiv.org/content/10.1101/581280v3.full https://www.biorxiv.org/content/10.1101/581280v3.full

solinent 7y ago
The criteria for what "depression" is has changed over time, no wonder our results are changing. Almost everyone has depression given the current criteria [1]. Most people have 2,4,5,6,7. The criteria is so vague your study would have to control for each item on the list. Here, they're not even using that criteria, and instead just using a simple test which involves self-reports. Placebo could be a huge factor here. [1] https://www.psycom.net/depression-definition-dsm-5-diagnostic-criteria/ https://www.psycom.net/depression-definition-dsm-5-diagnosti...
- simiones 7y ago
  I would highly dispute that "almost everyone" experiences any of these symptoms (note that they must happen "(almost) every day"). You've also missed a very important requirement: "To receive a diagnosis of depression, these symptoms must cause the individual clinically significant distress or impairment in social, occupational, or other important areas of functioning." You might accuse this definition of being too subjective. But it is certainly not something that includes "almost everyone".
  - solinent 7y ago
    > To receive a diagnosis of depression, these symptoms must cause the individual clinically significant distress or impairment in social, occupational, or other important areas of functioning. I'd say most of the lower class has this issue. > But it is certainly not something that includes "almost everyone". How do you know this? edit: maybe almost everyone is too much, I mean about 97% of the population.
    - caymanjim 7y ago
      > I'd say most of the lower class has this issue. Why would you say that? Do you think that rich people are happy and poor people are depressed? Anecdotally, the poorest people I know are often the happiest and the richest people are often the most miserable. I was a whole lot happier when I was younger and poorer. One thing has nothing to do with the other.
      - solinent 7y ago
        > Why would you say that? Do you think that rich people are happy and poor people are depressed? There are numerous studies showing this, for almost all disease categories. All psychological illness is negatively corelated with socioeconomic status. https://en.wikipedia.org/wiki/Socioeconomic_status https://en.wikipedia.org/wiki/Socioeconomic_status I'd say it's hard to find good sources of nutrition and the time to exercise when you're part of the lower class. edit: negatively correlated...
        caymanjim 7y ago
        A correlation between economic status and disease does not mean that most of the lower class is depressed.
        solinent 7y ago
        https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=+socioeconomic+status+depression&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=+soc... Also depression. It took me one minute to find three sources, could you please make some effort here? I have other things to do. And you're right, it's not a causative factor necessarily, but I think there's pretty strong evidence at this point, it's hard to find a poor person who doesn't have some condition. [1] https://www.sciencedirect.com/science/article/abs/pii/S0022399902003033 https://www.sciencedirect.com/science/article/abs/pii/S00223... [2] https://psycnet.apa.org/doiLanding?doi=10.1037/0021-843X.111.4.578 https://psycnet.apa.org/doiLanding?doi=10.1037/0021-843X.111... [3] https://ajph.aphapublications.org/doi/full/10.2105/AJPH.93.11.1844 https://ajph.aphapublications.org/doi/full/10.2105/AJPH.93.1...
- 0xcde4c3db 7y ago
  Most people certainly do not meet all of the criteria you list, especially #2 ("Markedly diminished interest or pleasure in all, or almost all, activities most of the day, nearly every day."). I can understand some confusion over #4 ("A slowing down of thought and a reduction of physical movement (observable by others, not merely subjective feelings of restlessness or being slowed down)."), which can be read as describing ordinary fatigue but is actually about a different pattern called "psychomotor retardation" [1]. [1] https://en.wikipedia.org/wiki/Psychomotor_retardation https://en.wikipedia.org/wiki/Psychomotor_retardation
  - solinent 7y ago
    With epidemology it's hard to tell the difference, there's no identified mechanism for depression or psychomotor retardation. Perhaps they're in the same category? All psychological disease categories are co-morbid with each other. What about existential crises? Do people who have those also have depression?
- markroseman 7y ago
  Applying the (current for any given time) criteria for depression requires a certain amount of training and skill, and the point form list of symptoms that you see as the first section in each diagnosis in the DSM-5 is not standalone. To mention one huge but often-overlooked example, something isn't a mental illness unless it significantly interferes with your life and functioning. Not to say that people don't incorrectly make diagnoses all the time. Many of these studies also rely on rating scales (typically HAMD or even PHQ9 for depression) that are a measure of depressive symptoms but don't in themselves have the power to make a diagnosis.
  - solinent 7y ago
    > Many of these studies also rely on rating scales (typically HAMD or even PHQ9 for depression) that are a measure of depressive symptoms but don't in themselves have the power to make a diagnosis. This is my point again if you read my original post, thank you. :) I'd go further and say the DSM-V is pretty bad at narrowing down which medication to use for depression. I know so many people on damn SNRIs whose diet is complete trash.
    - markroseman 7y ago
      Totally agree on your last point. There's little connection between DSM and treatment guidelines (which do exist, separately). And treatment is so fragmented between multiple care providers (with most people accessing a small subset) that many obvious things get lost. (Caveat: my wife is a psychiatrist. We wrote a book on this: https://mhnav.com/book/ https://mhnav.com/book/)
MelioRatio 7y ago
I very much agree, we have to be cautious with our optimism. Concerning the approach, this study looked into a fairly new form of rTMS, namely iTBS. Here is paper detailing how it differs from 10Hz rTMS, with a specific focus on potential cost savings: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6742475/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6742475/ We certainly need more data on iTBS effectiveness and I hope that a properly controlled, double-blind study can yield some information on whether this is actually effective on a larger scale and in the long term. A sample size of 21 is simply to small to make any judgement on that.
_bxg1 7y ago
No blinding on a study of a psychological condition just seems doubly bad
riyadparvez 7y ago
As someone who is in therapy for quite some time and read quite a lot about psychology, I can please take any claims of "evidence"-based short-term therapy approach with gain of salt. Particularly, if it's a very short term manualized approach of therapy eg, CBT style therapy. Lots of research in CBT is plagued with abuse of statistics, manipulation of randomized controlled trials, sensalization, and over-hype. For the vast majority of time CBT is good for symptomp reduction and patients relapse frequently. Frankly, the depressed people I know in real life, I don't know how many of them would be benefitted by CBT style therapy. My own experience is it took me almost a year of therapy, reading psychology and philosophy, lots of self-relfection and going to dark places in my mind to see marked improvement in my depression. If you are suffering from depression, please give psychotherapy a try. It'd be hard, but it's worth it!
_prototype_ 7y ago
I'm not a scientist, but I'm curious if any of you know: why do these important studies use such small N? I think its pretty obvious that a small N is not statistically significant, yet for this study, or the myriad of COVID-19 studies, the N is usually < 20. It's just so frustrating that professional scientists just completely ignore this statistical fact. Why not use use N >= 100 or not at all. It just confuses things in my opinion.
- jdm2212 7y ago
  The paper doesn't claim to be proving efficacy, just safety and feasibility. But university PR office gotta PR.
- superhuzza 7y ago
  >why do these important studies use such small N Generally because bigger N means the study is more complicated logistically ( cost, finding participants, whatever). And if the results are promising you can always get more funding to move on to bigger trials. Nobody is going to pay for you to recruit 1000 participants if you can't show an effect size with a smaller number first. > I think its pretty obvious that a small N is not statistically significant Well this totally depends on the effect size. You can't just dismiss results because N is small, that ignores how probability works. >Why not use use N >= 100 or not at all. Because N > 100 is a totally arbitrary value. The size of N that you need is totally dependent on the effect size, not an arbitrary value. You'd miss lots of real but small effects by having an N of only 100. You'd miss out on lots of real but large effects which can be observed at N < 100. Basically this approach would severely limit the branches of research we can do, for a questionable gain in statistical reliability.
  - grugagag 7y ago
    > And if the results are promising you can always get more funding to move on to bigger trials. This seems to be such a case. I wonder whether it was set from the beginning with more funding in mind.
- gshdg 7y ago
jimmyswimmy 7y ago
Indeed. >With SAINT, study participants underwent 10 sessions per day of 10-minute treatments, with 50-minute breaks in between. After a day of therapy, Lehman’s mood score indicated she was no longer depressed; it took up to five days for other participants. On average, three days of the therapy were enough for participants to have relief from depression. One imagines that three to five 10 hour (!) days of a presumably highly-trained professional spending time and attention and focus on a patient serves as no less than an incredibly strong placebo. It would be interesting to see a blinded study where the transducer functionality was replaced by a small heater (or whatever is needed to make the system act, feel and sound the same for all involved). It would be lovely if this held up. Depression and other psychological issues are insidious.
- conradev 7y ago
  They are doing said study. From the second sentence: > The researchers are conducting a larger, double-blinded trial in which half the participants are receiving fake treatment.
  - lozaning 7y ago
    How does fake treatment work in a clinical therapy test setting? I'd think I'd either figure out that the person giving me therapy was talking nonsense pretty quickly, or their 'placebo' therapy is indistinguishable from actual therapy, in which case am I still a control?
    - jefftk 7y ago
      This is magnetic stimulation. You could have a control group where you don't turn on the magnet.
      - msandford 7y ago
        Yeah but there was an article posted a few days ago about how Reiki can't possibly work but it still does. IMO the only way to blind this is to put people in an automated machine that either turns the magnet on or it doesn't (and it makes some noises either way). You can't have highly trained professionals in the loop making people feel valuable by giving them all this attention, as that could be a confounding effect in both the alleged-treatment and the alleged-placebo. https://news.ycombinator.com/item?id=22764824 https://news.ycombinator.com/item?id=22764824
c3534l 7y ago
Transcranial Magnetic Stimulation has had poor results in the past, failing to live up to its previous hype. So I want to know what they did that's fundamentally different from past results. If what they did differently was do a poor job in their experimental setup, then that is enough to explain it.
JamesBarney 7y ago
I'm more hopeful. The remission rate probably won't hold up in subsequent studies, but the current remission rate is enormous. I think there is a very good chance that some effect holds up. Also I don't think your analysis that it's all or nothing, i.e. that it has a true effect size of "Nobel prize winning" or nothing is true. I think it's very likely that there is some effect size it's just not world changing.
- jdm2212 7y ago
  > I think there is a very good chance that some effect holds up. Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs which are the first thing doctors prescribe for depression after telling you to go get some exercise and sleep better. This study has N=21, isn't an RCT, and isn't even claiming to be studying efficacy. In terms of result robustness, this is about as strong as posting a poll in your team Slack channel. Anyway, the study wasn't even focused on efficacy! It says so in the paper. But that didn't stop Stanford PR from putting the supposed efficacy in the headline anyway to get clicks/shares.
  - JamesBarney 7y ago
    > Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs. This has everything to do with small effect size for SSRIs and nothing to do with this study. Doctor's prescribe SSRI's not because they are especially effective but because they are especially safe. For example MAOI's have much higher effect sizes than SSRI's. Also I'm pretty sure every single RCTs with thousands of subjects regarding SSRI efficacy has found an effect, just not always a large one, but I'd love to see a study as a counter example. But on this to this study What do you think a control group remission rate would be 20%, with ECT it's 48%. They had a remission rate of 90%. An effect that size is most likely a true effect or outright fraud. What is the probability this happened by accident? My probability is rusty but that seems like that would be highly unlikely. It's gotta be less than 1/1000. If you gave a pill you thought might help people lose weight to 21 people and 90% of them last 80 lbs. Would you think "hey there's no control group", or "holy shit I probably found something".

solinent 7y ago

The criteria for what "depression" is has changed over time, no wonder our results are changing. Almost everyone has depression given the current criteria [1]. Most people have 2,4,5,6,7. The criteria is so vague your study would have to control for each item on the list. Here, they're not even using that criteria, and instead just using a simple test which involves self-reports. Placebo could be a huge factor here. [1] https://www.psycom.net/depression-definition-dsm-5-diagnostic-criteria/ https://www.psycom.net/depression-definition-dsm-5-diagnosti...

simiones 7y ago
I would highly dispute that "almost everyone" experiences any of these symptoms (note that they must happen "(almost) every day"). You've also missed a very important requirement: "To receive a diagnosis of depression, these symptoms must cause the individual clinically significant distress or impairment in social, occupational, or other important areas of functioning." You might accuse this definition of being too subjective. But it is certainly not something that includes "almost everyone".
- solinent 7y ago
  > To receive a diagnosis of depression, these symptoms must cause the individual clinically significant distress or impairment in social, occupational, or other important areas of functioning. I'd say most of the lower class has this issue. > But it is certainly not something that includes "almost everyone". How do you know this? edit: maybe almost everyone is too much, I mean about 97% of the population.
  - caymanjim 7y ago
    > I'd say most of the lower class has this issue. Why would you say that? Do you think that rich people are happy and poor people are depressed? Anecdotally, the poorest people I know are often the happiest and the richest people are often the most miserable. I was a whole lot happier when I was younger and poorer. One thing has nothing to do with the other.
    - solinent 7y ago
      > Why would you say that? Do you think that rich people are happy and poor people are depressed? There are numerous studies showing this, for almost all disease categories. All psychological illness is negatively corelated with socioeconomic status. https://en.wikipedia.org/wiki/Socioeconomic_status https://en.wikipedia.org/wiki/Socioeconomic_status I'd say it's hard to find good sources of nutrition and the time to exercise when you're part of the lower class. edit: negatively correlated...
      - caymanjim 7y ago
        A correlation between economic status and disease does not mean that most of the lower class is depressed.
        solinent 7y ago
        https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=+socioeconomic+status+depression&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=+soc... Also depression. It took me one minute to find three sources, could you please make some effort here? I have other things to do. And you're right, it's not a causative factor necessarily, but I think there's pretty strong evidence at this point, it's hard to find a poor person who doesn't have some condition. [1] https://www.sciencedirect.com/science/article/abs/pii/S0022399902003033 https://www.sciencedirect.com/science/article/abs/pii/S00223... [2] https://psycnet.apa.org/doiLanding?doi=10.1037/0021-843X.111.4.578 https://psycnet.apa.org/doiLanding?doi=10.1037/0021-843X.111... [3] https://ajph.aphapublications.org/doi/full/10.2105/AJPH.93.11.1844 https://ajph.aphapublications.org/doi/full/10.2105/AJPH.93.1...
0xcde4c3db 7y ago
Most people certainly do not meet all of the criteria you list, especially #2 ("Markedly diminished interest or pleasure in all, or almost all, activities most of the day, nearly every day."). I can understand some confusion over #4 ("A slowing down of thought and a reduction of physical movement (observable by others, not merely subjective feelings of restlessness or being slowed down)."), which can be read as describing ordinary fatigue but is actually about a different pattern called "psychomotor retardation" [1]. [1] https://en.wikipedia.org/wiki/Psychomotor_retardation https://en.wikipedia.org/wiki/Psychomotor_retardation
- solinent 7y ago
  With epidemology it's hard to tell the difference, there's no identified mechanism for depression or psychomotor retardation. Perhaps they're in the same category? All psychological disease categories are co-morbid with each other. What about existential crises? Do people who have those also have depression?
markroseman 7y ago
Applying the (current for any given time) criteria for depression requires a certain amount of training and skill, and the point form list of symptoms that you see as the first section in each diagnosis in the DSM-5 is not standalone. To mention one huge but often-overlooked example, something isn't a mental illness unless it significantly interferes with your life and functioning. Not to say that people don't incorrectly make diagnoses all the time. Many of these studies also rely on rating scales (typically HAMD or even PHQ9 for depression) that are a measure of depressive symptoms but don't in themselves have the power to make a diagnosis.
- solinent 7y ago
  > Many of these studies also rely on rating scales (typically HAMD or even PHQ9 for depression) that are a measure of depressive symptoms but don't in themselves have the power to make a diagnosis. This is my point again if you read my original post, thank you. :) I'd go further and say the DSM-V is pretty bad at narrowing down which medication to use for depression. I know so many people on damn SNRIs whose diet is complete trash.
  - markroseman 7y ago
    Totally agree on your last point. There's little connection between DSM and treatment guidelines (which do exist, separately). And treatment is so fragmented between multiple care providers (with most people accessing a small subset) that many obvious things get lost. (Caveat: my wife is a psychiatrist. We wrote a book on this: https://mhnav.com/book/ https://mhnav.com/book/)

MelioRatio 7y ago

I very much agree, we have to be cautious with our optimism. Concerning the approach, this study looked into a fairly new form of rTMS, namely iTBS. Here is paper detailing how it differs from 10Hz rTMS, with a specific focus on potential cost savings: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6742475/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6742475/ We certainly need more data on iTBS effectiveness and I hope that a properly controlled, double-blind study can yield some information on whether this is actually effective on a larger scale and in the long term. A sample size of 21 is simply to small to make any judgement on that.

_bxg1 7y ago

No blinding on a study of a psychological condition just seems doubly bad

riyadparvez 7y ago

As someone who is in therapy for quite some time and read quite a lot about psychology, I can please take any claims of "evidence"-based short-term therapy approach with gain of salt. Particularly, if it's a very short term manualized approach of therapy eg, CBT style therapy. Lots of research in CBT is plagued with abuse of statistics, manipulation of randomized controlled trials, sensalization, and over-hype. For the vast majority of time CBT is good for symptomp reduction and patients relapse frequently. Frankly, the depressed people I know in real life, I don't know how many of them would be benefitted by CBT style therapy. My own experience is it took me almost a year of therapy, reading psychology and philosophy, lots of self-relfection and going to dark places in my mind to see marked improvement in my depression. If you are suffering from depression, please give psychotherapy a try. It'd be hard, but it's worth it!

_prototype_ 7y ago

I'm not a scientist, but I'm curious if any of you know: why do these important studies use such small N? I think its pretty obvious that a small N is not statistically significant, yet for this study, or the myriad of COVID-19 studies, the N is usually < 20. It's just so frustrating that professional scientists just completely ignore this statistical fact. Why not use use N >= 100 or not at all. It just confuses things in my opinion.

jdm2212 7y ago
The paper doesn't claim to be proving efficacy, just safety and feasibility. But university PR office gotta PR.
superhuzza 7y ago
>why do these important studies use such small N Generally because bigger N means the study is more complicated logistically ( cost, finding participants, whatever). And if the results are promising you can always get more funding to move on to bigger trials. Nobody is going to pay for you to recruit 1000 participants if you can't show an effect size with a smaller number first. > I think its pretty obvious that a small N is not statistically significant Well this totally depends on the effect size. You can't just dismiss results because N is small, that ignores how probability works. >Why not use use N >= 100 or not at all. Because N > 100 is a totally arbitrary value. The size of N that you need is totally dependent on the effect size, not an arbitrary value. You'd miss lots of real but small effects by having an N of only 100. You'd miss out on lots of real but large effects which can be observed at N < 100. Basically this approach would severely limit the branches of research we can do, for a questionable gain in statistical reliability.
- grugagag 7y ago
  > And if the results are promising you can always get more funding to move on to bigger trials. This seems to be such a case. I wonder whether it was set from the beginning with more funding in mind.
gshdg 7y ago

jimmyswimmy 7y ago

Indeed. >With SAINT, study participants underwent 10 sessions per day of 10-minute treatments, with 50-minute breaks in between. After a day of therapy, Lehman’s mood score indicated she was no longer depressed; it took up to five days for other participants. On average, three days of the therapy were enough for participants to have relief from depression. One imagines that three to five 10 hour (!) days of a presumably highly-trained professional spending time and attention and focus on a patient serves as no less than an incredibly strong placebo. It would be interesting to see a blinded study where the transducer functionality was replaced by a small heater (or whatever is needed to make the system act, feel and sound the same for all involved). It would be lovely if this held up. Depression and other psychological issues are insidious.

conradev 7y ago
They are doing said study. From the second sentence: > The researchers are conducting a larger, double-blinded trial in which half the participants are receiving fake treatment.
- lozaning 7y ago
  How does fake treatment work in a clinical therapy test setting? I'd think I'd either figure out that the person giving me therapy was talking nonsense pretty quickly, or their 'placebo' therapy is indistinguishable from actual therapy, in which case am I still a control?
  - jefftk 7y ago
    This is magnetic stimulation. You could have a control group where you don't turn on the magnet.
    - msandford 7y ago
      Yeah but there was an article posted a few days ago about how Reiki can't possibly work but it still does. IMO the only way to blind this is to put people in an automated machine that either turns the magnet on or it doesn't (and it makes some noises either way). You can't have highly trained professionals in the loop making people feel valuable by giving them all this attention, as that could be a confounding effect in both the alleged-treatment and the alleged-placebo. https://news.ycombinator.com/item?id=22764824 https://news.ycombinator.com/item?id=22764824

c3534l 7y ago

Transcranial Magnetic Stimulation has had poor results in the past, failing to live up to its previous hype. So I want to know what they did that's fundamentally different from past results. If what they did differently was do a poor job in their experimental setup, then that is enough to explain it.

JamesBarney 7y ago

I'm more hopeful. The remission rate probably won't hold up in subsequent studies, but the current remission rate is enormous. I think there is a very good chance that some effect holds up. Also I don't think your analysis that it's all or nothing, i.e. that it has a true effect size of "Nobel prize winning" or nothing is true. I think it's very likely that there is some effect size it's just not world changing.

jdm2212 7y ago
> I think there is a very good chance that some effect holds up. Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs which are the first thing doctors prescribe for depression after telling you to go get some exercise and sleep better. This study has N=21, isn't an RCT, and isn't even claiming to be studying efficacy. In terms of result robustness, this is about as strong as posting a poll in your team Slack channel. Anyway, the study wasn't even focused on efficacy! It says so in the paper. But that didn't stop Stanford PR from putting the supposed efficacy in the headline anyway to get clicks/shares.
- JamesBarney 7y ago
  > Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs. This has everything to do with small effect size for SSRIs and nothing to do with this study. Doctor's prescribe SSRI's not because they are especially effective but because they are especially safe. For example MAOI's have much higher effect sizes than SSRI's. Also I'm pretty sure every single RCTs with thousands of subjects regarding SSRI efficacy has found an effect, just not always a large one, but I'd love to see a study as a counter example. But on this to this study What do you think a control group remission rate would be 20%, with ECT it's 48%. They had a remission rate of 90%. An effect that size is most likely a true effect or outright fraud. What is the probability this happened by accident? My probability is rusty but that seems like that would be highly unlikely. It's gotta be less than 1/1000. If you gave a pill you thought might help people lose weight to 21 people and 90% of them last 80 lbs. Would you think "hey there's no control group", or "holy shit I probably found something".