3 ms·

1. Maybe, but that should be a tradeoff that is made consciously with some analysis and care, instead of just jumping over a low FDA bar, which is what everyone

1. Maybe, but that should be a tradeoff that is made consciously with some analysis and care, instead of just jumping over a low FDA bar, which is what everyone in this space seems to be doing. I think there are enough abnormalities that can be fatal which a net would just fill in (e.g. aortic dissection?) 2. There are thousands of different abnormalities. From what I understand about the FDA validation process for this sort of thing there would be only 10s of abnormalities. There are likely, many, many of them that are quite obvious to radiologists. And once again, this would be a question that should be studied carefully when people's lives are at risk, instead of just assuming that it will be fine then going ahead to "move fast and break things"

bartimus 8y ago
It's my understanding it's sometimes exactly those obvious abnormalities that are missed by radiologists.
- nitrogen 8y ago
  Adding another lossy step will only make even more anomalies get missed, or worse, misdiagnosed as something else (I'm thinking about the photocopiers that sometimes change numbers in documents). The argument that "our system is already bad so we should just merge this new bad component because it doesn't make it worse" is bad in software, and unacceptable in medicine.
  - tntn 8y ago
    I don't think that is a fair characterization of what I wrote. If the scanning process has a sensitivity of 99.999%, and the next stage in the signal chain has a sensitivity of 50%, and we consider what happens if the MRI sensitivity drops to 99.9%, that's well in the noise in terms of diagnostic value. Use some of the extra money freed up by doing the MRI 10x faster to pay a radiologist to look at the scan for 10% longer, and perhaps the net accuracy has improved. What is unacceptable about this kind of holistic reasoning about the system? High performing systems are frequently not composed of perfect components.
    - nitrogen 8y ago
      It's fine to talk about statistical accuracy when the distribution is somewhat normal. ML systems we've seen so far don't typically degrade in nice, statistical, predictable ways. Omitting 90% of samples (based on 10x speed claim) leaves an awful lot of room for bizarre errors.
  - bartimus 8y ago
    One advantage of software is that it can't really have a bad day. The radiologist can be affected by external human factors. Perhaps an angry wife and a dinner getting cold at home. He might not take the extra picture a bit further down the lower back where the golfball tumor was growing (story from my mother-in-law).
tntn 8y ago
> And once again, this would be a question that should be studied carefully when people's lives are at risk I agree 100% with this, but how can this question be studied of this research to develop these systems doesn't go forward? You earlier characterized the FastMRI research as, in your view, "actually very dangerous." As I see it, the research here is potentially extremely valuable, and the danger comes not from the research, but from somebody deciding to deploy it without considering these questions. Typically the road from research program to wide deployment is quite long, and I disagree that we should discourage the research because of potential flaws in the productization process. (Unless there is ill will on the part of the researchers, which I'm assuming there is not).
- dontreact 8y ago
  I think it’s the lack of will to study these sort of things that is dangerous. There just doesn’t seem to be this sense of caution among the people working on this. They seem to just want to jump over the low bar set by the fda and move ahead to patient care.

bartimus 8y ago

It's my understanding it's sometimes exactly those obvious abnormalities that are missed by radiologists.

nitrogen 8y ago
Adding another lossy step will only make even more anomalies get missed, or worse, misdiagnosed as something else (I'm thinking about the photocopiers that sometimes change numbers in documents). The argument that "our system is already bad so we should just merge this new bad component because it doesn't make it worse" is bad in software, and unacceptable in medicine.
- tntn 8y ago
  I don't think that is a fair characterization of what I wrote. If the scanning process has a sensitivity of 99.999%, and the next stage in the signal chain has a sensitivity of 50%, and we consider what happens if the MRI sensitivity drops to 99.9%, that's well in the noise in terms of diagnostic value. Use some of the extra money freed up by doing the MRI 10x faster to pay a radiologist to look at the scan for 10% longer, and perhaps the net accuracy has improved. What is unacceptable about this kind of holistic reasoning about the system? High performing systems are frequently not composed of perfect components.
  - nitrogen 8y ago
    It's fine to talk about statistical accuracy when the distribution is somewhat normal. ML systems we've seen so far don't typically degrade in nice, statistical, predictable ways. Omitting 90% of samples (based on 10x speed claim) leaves an awful lot of room for bizarre errors.
- bartimus 8y ago
  One advantage of software is that it can't really have a bad day. The radiologist can be affected by external human factors. Perhaps an angry wife and a dinner getting cold at home. He might not take the extra picture a bit further down the lower back where the golfball tumor was growing (story from my mother-in-law).

tntn 8y ago

> And once again, this would be a question that should be studied carefully when people's lives are at risk I agree 100% with this, but how can this question be studied of this research to develop these systems doesn't go forward? You earlier characterized the FastMRI research as, in your view, "actually very dangerous." As I see it, the research here is potentially extremely valuable, and the danger comes not from the research, but from somebody deciding to deploy it without considering these questions. Typically the road from research program to wide deployment is quite long, and I disagree that we should discourage the research because of potential flaws in the productization process. (Unless there is ill will on the part of the researchers, which I'm assuming there is not).

dontreact 8y ago
I think it’s the lack of will to study these sort of things that is dangerous. There just doesn’t seem to be this sense of caution among the people working on this. They seem to just want to jump over the low bar set by the fda and move ahead to patient care.