5 ms·
>> Deep learning algorithms have proven to be better than humans at spotting lung cancer, a development that if applied at scale could save more than 30,000 pat
by YeGoblynQueenne 7y ago
>> Deep learning algorithms have proven to be better than humans at spotting lung
cancer, a development that if applied at scale could save more than 30,000
patients per year.
It's not easy to scale deep learning because deep neural nets have a very
strong tendency to overfit to their training dataset and are very bad at
generalising outside their training dataset.
In a medical context this means that, while a particular deep learning image
classifier might be very good at recognising cancer in images of patients'
scans collected from a specific hospital, the same classifier will be much
worse in the same task on images from a different hospital (or even from a
different department in the same hospital).
To overcome this limitation, the only thing anyone knows that works to some
extent is to train deep neural nets with a lot of data. If you can't avoid
overfitting, at least you can try to overfit to a big enough sample that most
common kinds of instances in your domain of interest will be included in it.
So basically to scale a diagnostic system based on deep neural net image
classification to the nation level one would have to train a deep learning
image classifier with the data from all hospitals in that nation.
This is not an easy task, to say the least. It's not undoable, but it's not as
simple as having someone at Hospital X download a pretrained model in
Tensorflow and train its last few layers on some CT scans.
- bodono 7y agoThis statement is false, as recently demonstrated by DeepMind on retinal scans. Not only did they generalize outside of the training dataset but they were able to use the features learned by the model on an entirely different type of scanning device. https://www.nature.com/articles/s41591-018-0107-6.epdf?author_access_token=PAbvHEuv_YYmrPVbG5HqKdRgN0jAjWel9jnR3ZoTv0P43NEH20hFuvBoJk6cvICihn8kmL6tmejFlnuPlbT_0KmJgK6N07SPh_ZLy0Nxb0-LAGIDBaH1fjJTkD9ahUEQpRlEudtlG9E1v3ca9xNQcQ%3D%3D https://www.nature.com/articles/s41591-018-0107-6.epdf?autho... "Moreover, we demonstrate that the tissue segmentations produced by our architecture act as a device-independent representation; referral accuracy is maintained when using tissue segmentations from a different type of device."
- YeGoblynQueenne 7y agoIn the paper you link to, the researchers trained an image classifier on data collected from 32 sites of the Moorfiel NHS trust. The trained model was tested on, presumably held-out, data from the same dataset. This is an example of scaling a model beyond a dataset collected from a single site. It is not contrary to what I say in my comment. The researchers further tested their model on data obtained from a different device than it was originally trained on. This data was collected from the same hospital sites. The original model performed poorly on this new data and was re-trained to improve its performance. This does not demonstrate an ability to generalise to unseen data- only an ability to adjust a model to new data, by re-training.
- bodono 7y agoIt contradicts your statement: "it's not as simple as having someone at Hospital X download a pretrained model in Tensorflow and train its last few layers on some CT scans" Because in this case it was as easy as taking a model from a totally different modality and retraining the first (in this case) few layers to accommodate the new device. Furthermore the original training used 15k scans and the retraining only required 152 scans. This is totally reasonable and clear evidence of transfer and generalization. Moreover, even human operators require retraining on new devices!
- YeGoblynQueenne 7y agoMy Tensorflow comment was a bit unclear. I meant that you can't just download a generic model like the kind that is readily available, e.g. one trained on ImageNet or CIFAR etc, and expect that you can retrain it easily and get a diagnostic tool that is competitive with an expert. The models in the paper you link were specifically trained on medical imaging data. My point is that you need a lot of work to make this work even for one hospital, let alone scale to many, even more so scale at the level of a national health service. I don't see that the paper you link contradicts this. Edit: if I may summarise: I said "it's not simple" not "you can't do it". Transfer learning is not generalisation to unseen data. If the pre-trained model and the end model don't have any common instances it doesn't work [Edit: "don't have any instances with a common feature space" is more clear]. Also, you're talking about generalisation to new devices. My understanding is that this is only one aspect of the difficulties with scaling image recognition for medical diagnoses to data from different sites.
- epiphanitus 7y ago>>the same classifier will be much worse in the same task on images from a different hospital (or even from a different department in the same hospital). Has anybody figured out why this is the case? Could it be socioeconomic factors? Or the presence of different toxic pollutants across different communities?
- mcguire 7y agoMore likely different imaging set-ups.
- hadsed 7y agoIt is getting easier to scale deep learning in difficult domains. It'll likely be some combo of pretrained semi- or self-supervised models that are transferred and fine-tuned. You mention large data, but we also have the knobs of inductive bias and training objectives. Once we crack inductive biases for CT scans, perhaps by analyzing a model trained on large amounts of data, then scaling gets easier. I don't think the situation is too dire, it's just a very difficult and high-risk domain. It could also get a lot easier once we figure out better training objectives, but just like inductive biases, those are pretty domain specific so they take some time to discover.