6 ms·

The Trouble with Brain Scans

53 points by dnetesn 6y ago

arafa 6y ago
This article reads like a longform version of why statistical power/effect size is important but doesn't mention it by name, only alluding to it. I suspect there would be much less of a replication crisis if, instead of just focusing on statistical significance, there was also a focus on effect size, or both.
jessriedel 6y ago
My impression is that Nautilus and Quanta are, among pop science magazines, fairly high brow, but that Nautilus is substantially more likely to use human-interest narratives (e.g., the opening vignette) to keep the reader's attention. Is that right, or do I have the wrong impression?
- tsjq 6y ago
  I have noticed / felt the same about Nautilus.
- gwern 6y ago
  I agree. Quanta is substantially more 'hardcore'. Perhaps it's the Simons Foundation funding?
- COGlory 6y ago
  I've more or less stopped reading nautilus for that reason
Sephr 6y ago
You'll never get a full brain scan with fMRIs. For real detail, you'll need a planetary-scale XFEL with a human head-sized reticule and be okay with your irl head being vaporized. Optimized models for simulating your brain accurately don't exist yet either. I highly suspect that we will have to finish the above first.
- thaeli 6y ago
  Well, that's one way to dodge the "uploaded mind is a copy" problem.
  - mjevans 6y ago
    Short version: Go play the videogame Soma. Longer version. It's still a copy, but is a copy of you still you; are you still you if you've had a copy? (My belief is yes; though at that point timelines diverge and neither 'you' is fungible for the other.)
    - danielheath 6y ago
      Is a token referring to the copy fungible?
- jnwatson 6y ago
  The first episode of the Amazon Prime comedy "Upload" has a humorous/shocking scene of this.
checkyoursudo 6y ago
As a cognitive scientist with only 1 graduate level fMRI course and no hands-on experience outside of playing around with Neurodebian, I will not be so bold (haha) as to make a claim that there is absolutely nothing to fMRI. However, I am very skeptical. I would not be willing to make any strong claims about causality when the alleged neural event and the measurement thereof are 15-30 seconds apart, and the thing being measured (blood flow) is not the event itself (neural activity), and there are so, so many potential sources of error. Now, structural MRI is super cool. I have a bunch of friends at my university who work in fMRI brain imaging. I feel bad, but when we talk about their work, I can't help but think they are chasing alchemy. Or maybe a better analogy is that it is almost like they are trying to learn about the world (brain, cognition) based on the shadows at the back of Plato's cave (fMRI BOLD signals). I sincerely hope they overcome the challenges, but I expect they will be facing the exact or nearly the same challenges 20 years from now.
- tartoran 6y ago
  Even if they’re on a deadend they may discover a thing or two that can be applied somewhere else. But I hear you, your intuition is probably right. But if they left this area unexplored it would still be a loss.
  - checkyoursudo 6y ago
    Yes, of course this is reasonable. Progress still requires ruling out reasonable avenues of exploration. And even some relatively unreasonable ones.
- glial 6y ago
  > I am very skeptical As a fellow cognitive scientist, I am too.
- Toenex 6y ago
  It's a while since I ran fMRI studies but my favourite analogy was to tell people that it was like trying to measure your electricity usage by watching your water meter.
beefman 6y ago
Tom Mitchell has been using machine learning to interpret fMRI data.[1] The method struck me as one that could potentially work (though I was a bit surprised by the apparent success rate of studies in his lab). The fundamental problem with fMRI is its poor time resolution. MEG is promising in this respect.[2] [1] https://www.youtube.com/watch?v=zy0-_HNF1us&t=6m8s https://www.youtube.com/watch?v=zy0-_HNF1us&t=6m8s [2] https://en.wikipedia.org/wiki/Magnetoencephalography https://en.wikipedia.org/wiki/Magnetoencephalography
ska 6y ago
> "Functional magnetic resonance imaging has transformed medicine." This is not even close to true. The clinical impact of fMRI is very constrained. I've met many clinicians who have completely written it off at this point. Others that remain interested but find it mostly impractical and error prone. The only people I've met who seem to have any significant investment in it as a technology are cognitive scientists, and that's a ways from medicine. > It allows non-invasive mapping of a patient’s brain regions to enable more accurate, precise neurosurgery,1" Approximately nobody does this. fMRI isn't really even on the radar for most neurosurgeons. It's only recently that even detailed structural information (e.g. DTI) has got some real traction, let alone functional info. Note commercial vendors have had clinical packages on the scanners for (nearly? I forget who shipped DWI stuff first) a couple decades now. No such thing exists for fMRI, at minimum you need to buy a 3rd party processing system and may need a research key on your scanner. Some researchers are interested in this; that's a long way from transforming anything. > " as well as validating pharmacological effects of potential drugs on human brains.2" There is perhaps a bit more promise here, but it's still way into the research only end. In the 15 years since the referenced paper was published, not much of this "promise" has been realized as far as I know (but this is further from my wheelhouse)
- astrange 6y ago
  What is a research key? I'm imagining a license dongle.
  - ska 6y ago
    Basically yes, a license package.
- mountainriver 6y ago
  I was just reading about them using fMRI in conjunction with high frequency focused ultrasound to lesion parts of the brain associated with certain mental disorders such as OCD. They seemed to be having decent success with that
  - 6y ago
ramraj07 6y ago
This passage is the crux of the problem: > As for the Eklund, et al. paper, Rosen conceded, “It was an excellent point of statistics. It’s actually a point we understood, but there are no doubt lots of people that didn’t understand.” But he thinks the paper’s implications were overblown. “The impact of that paper was actually pretty modest in terms of the number of results that were invalidated, that were important results. And whether we were significantly misleading people or sending doctors astray or anything like that, was negligible as best as I could tell. But the paper got a lot of press, and suddenly, now fMRI has a black eye.” “You see, it’s the others that don’t do it right, we do things correctly so _our_ results are accurate and relevant. It’s those other researchers who are not contributing to curing cancer/Alzheimer’s/etc” - this is the argument I have heard time and again from many colleagues when we point some fundamental issue with their whole field. Rest assured they also do the same mistakes but have often built large cognitive fortresses to shield their conscience from the reality that they are also a charlatan at least partially. The only endgame that you can truly live with given such realisations is to leave the field but many can’t afford to do that, so you have a perpetuation of such fields which shouldn’t exist at the scale that they do.
KingMachiavelli 6y ago
It seems like there are two main points here. 1. fMRI only measure brain activity indirectly. 2. Nothing stops people from messing with any one parameter, setting, statistical knob out of houndreds. I don't know if #1 is a problem itself. We measure many things indirectly. As long as the limitations are known and respected, it is still useful. #2 Is the real issue. What we need is a movement towards saving and publishing the the actual data as close to the hardware as possible. But I'm sure medical hardware is full of quirky, proprietary Windows CE junk that makes this almost impossible. I don't know what the solution is. If the healthcare industry wanted better software they would have to pay even more than they already do. Cheaper and better software doesn't happen because those companies and developers would rather work elsewhere. Real disruptors will have a hard time convincing the 100 vendors they need to integrate with to change their ways. It seems that large industries will always be 10-15 years behind 'what they should be'.
- stinos 6y ago
  What we need is a movement towards saving and publishing the the actual data as close to the hardware as possible. But I'm sure medical hardware is full of quirky, proprietary Windows CE junk that makes this almost impossible. It's always easy to blame proprietary junk. But in this case, from the couple of things I know about it, I don't think the real problem in publishing actual data is that it's proprietary. Rather that doing something with that data is very hard. To the point that if you don't know how to build an MRI scanner, you probably also don't know how to transform the raw data into a usable and correct image.
SubiculumCode 6y ago
Crap bloated article. But everyone loves to bash neuroimaging. It's like a past time. Oh and, no one says 'lights up' in any serious context.
doggodaddo78 6y ago
IIRC, functional / structural scanning needs more than an fMRI such as a PET and a CT with contrast. Source: I know a guy who worked on Alzheimer's and they combined several radiology sources to form a composite brain scan to study structure.
greazy 6y ago
Its an intetesting article from a personal prespective. I found it very frustrating all the imagery was essentially the same graphic presented multiple times. Show a real brain scan! or a picture of thr bloody machine.

53 points by dnetesn 6y ago

This article reads like a longform version of why statistical power/effect size is important but doesn't mention it by name, only alluding to it. I suspect there would be much less of a replication crisis if, instead of just focusing on statistical significance, there was also a focus on effect size, or both.

jessriedel 6y ago

My impression is that Nautilus and Quanta are, among pop science magazines, fairly high brow, but that Nautilus is substantially more likely to use human-interest narratives (e.g., the opening vignette) to keep the reader's attention. Is that right, or do I have the wrong impression?

tsjq 6y ago
I have noticed / felt the same about Nautilus.
gwern 6y ago
I agree. Quanta is substantially more 'hardcore'. Perhaps it's the Simons Foundation funding?
COGlory 6y ago
I've more or less stopped reading nautilus for that reason

Sephr 6y ago

You'll never get a full brain scan with fMRIs. For real detail, you'll need a planetary-scale XFEL with a human head-sized reticule and be okay with your irl head being vaporized. Optimized models for simulating your brain accurately don't exist yet either. I highly suspect that we will have to finish the above first.

thaeli 6y ago
Well, that's one way to dodge the "uploaded mind is a copy" problem.
- mjevans 6y ago
  Short version: Go play the videogame Soma. Longer version. It's still a copy, but is a copy of you still you; are you still you if you've had a copy? (My belief is yes; though at that point timelines diverge and neither 'you' is fungible for the other.)
  - danielheath 6y ago
    Is a token referring to the copy fungible?
jnwatson 6y ago
The first episode of the Amazon Prime comedy "Upload" has a humorous/shocking scene of this.

checkyoursudo 6y ago

As a cognitive scientist with only 1 graduate level fMRI course and no hands-on experience outside of playing around with Neurodebian, I will not be so bold (haha) as to make a claim that there is absolutely nothing to fMRI. However, I am very skeptical. I would not be willing to make any strong claims about causality when the alleged neural event and the measurement thereof are 15-30 seconds apart, and the thing being measured (blood flow) is not the event itself (neural activity), and there are so, so many potential sources of error. Now, structural MRI is super cool. I have a bunch of friends at my university who work in fMRI brain imaging. I feel bad, but when we talk about their work, I can't help but think they are chasing alchemy. Or maybe a better analogy is that it is almost like they are trying to learn about the world (brain, cognition) based on the shadows at the back of Plato's cave (fMRI BOLD signals). I sincerely hope they overcome the challenges, but I expect they will be facing the exact or nearly the same challenges 20 years from now.

tartoran 6y ago
Even if they’re on a deadend they may discover a thing or two that can be applied somewhere else. But I hear you, your intuition is probably right. But if they left this area unexplored it would still be a loss.
- checkyoursudo 6y ago
  Yes, of course this is reasonable. Progress still requires ruling out reasonable avenues of exploration. And even some relatively unreasonable ones.
glial 6y ago
> I am very skeptical As a fellow cognitive scientist, I am too.
Toenex 6y ago
It's a while since I ran fMRI studies but my favourite analogy was to tell people that it was like trying to measure your electricity usage by watching your water meter.

beefman 6y ago

Tom Mitchell has been using machine learning to interpret fMRI data.[1] The method struck me as one that could potentially work (though I was a bit surprised by the apparent success rate of studies in his lab). The fundamental problem with fMRI is its poor time resolution. MEG is promising in this respect.[2] [1] https://www.youtube.com/watch?v=zy0-_HNF1us&t=6m8s https://www.youtube.com/watch?v=zy0-_HNF1us&t=6m8s [2] https://en.wikipedia.org/wiki/Magnetoencephalography https://en.wikipedia.org/wiki/Magnetoencephalography

ska 6y ago

> "Functional magnetic resonance imaging has transformed medicine." This is not even close to true. The clinical impact of fMRI is very constrained. I've met many clinicians who have completely written it off at this point. Others that remain interested but find it mostly impractical and error prone. The only people I've met who seem to have any significant investment in it as a technology are cognitive scientists, and that's a ways from medicine. > It allows non-invasive mapping of a patient’s brain regions to enable more accurate, precise neurosurgery,1" Approximately nobody does this. fMRI isn't really even on the radar for most neurosurgeons. It's only recently that even detailed structural information (e.g. DTI) has got some real traction, let alone functional info. Note commercial vendors have had clinical packages on the scanners for (nearly? I forget who shipped DWI stuff first) a couple decades now. No such thing exists for fMRI, at minimum you need to buy a 3rd party processing system and may need a research key on your scanner. Some researchers are interested in this; that's a long way from transforming anything. > " as well as validating pharmacological effects of potential drugs on human brains.2" There is perhaps a bit more promise here, but it's still way into the research only end. In the 15 years since the referenced paper was published, not much of this "promise" has been realized as far as I know (but this is further from my wheelhouse)

astrange 6y ago
What is a research key? I'm imagining a license dongle.
- ska 6y ago
  Basically yes, a license package.
mountainriver 6y ago
I was just reading about them using fMRI in conjunction with high frequency focused ultrasound to lesion parts of the brain associated with certain mental disorders such as OCD. They seemed to be having decent success with that
- 6y ago

ramraj07 6y ago

This passage is the crux of the problem: > As for the Eklund, et al. paper, Rosen conceded, “It was an excellent point of statistics. It’s actually a point we understood, but there are no doubt lots of people that didn’t understand.” But he thinks the paper’s implications were overblown. “The impact of that paper was actually pretty modest in terms of the number of results that were invalidated, that were important results. And whether we were significantly misleading people or sending doctors astray or anything like that, was negligible as best as I could tell. But the paper got a lot of press, and suddenly, now fMRI has a black eye.” “You see, it’s the others that don’t do it right, we do things correctly so _our_ results are accurate and relevant. It’s those other researchers who are not contributing to curing cancer/Alzheimer’s/etc” - this is the argument I have heard time and again from many colleagues when we point some fundamental issue with their whole field. Rest assured they also do the same mistakes but have often built large cognitive fortresses to shield their conscience from the reality that they are also a charlatan at least partially. The only endgame that you can truly live with given such realisations is to leave the field but many can’t afford to do that, so you have a perpetuation of such fields which shouldn’t exist at the scale that they do.

KingMachiavelli 6y ago

It seems like there are two main points here. 1. fMRI only measure brain activity indirectly. 2. Nothing stops people from messing with any one parameter, setting, statistical knob out of houndreds. I don't know if #1 is a problem itself. We measure many things indirectly. As long as the limitations are known and respected, it is still useful. #2 Is the real issue. What we need is a movement towards saving and publishing the the actual data as close to the hardware as possible. But I'm sure medical hardware is full of quirky, proprietary Windows CE junk that makes this almost impossible. I don't know what the solution is. If the healthcare industry wanted better software they would have to pay even more than they already do. Cheaper and better software doesn't happen because those companies and developers would rather work elsewhere. Real disruptors will have a hard time convincing the 100 vendors they need to integrate with to change their ways. It seems that large industries will always be 10-15 years behind 'what they should be'.

stinos 6y ago
What we need is a movement towards saving and publishing the the actual data as close to the hardware as possible. But I'm sure medical hardware is full of quirky, proprietary Windows CE junk that makes this almost impossible. It's always easy to blame proprietary junk. But in this case, from the couple of things I know about it, I don't think the real problem in publishing actual data is that it's proprietary. Rather that doing something with that data is very hard. To the point that if you don't know how to build an MRI scanner, you probably also don't know how to transform the raw data into a usable and correct image.

SubiculumCode 6y ago

Crap bloated article. But everyone loves to bash neuroimaging. It's like a past time. Oh and, no one says 'lights up' in any serious context.

doggodaddo78 6y ago

IIRC, functional / structural scanning needs more than an fMRI such as a PET and a CT with contrast. Source: I know a guy who worked on Alzheimer's and they combined several radiology sources to form a composite brain scan to study structure.

greazy 6y ago

Its an intetesting article from a personal prespective. I found it very frustrating all the imagery was essentially the same graphic presented multiple times. Show a real brain scan! or a picture of thr bloody machine.