5 ms·

Machine-learning improves the prediction of stroke recovery

38 points by rustoo 5y ago

rob_c 5y ago
Hope this doesn't mean that we hold back some treatment for those who would have bucked the trend and recovered better than expected... Very interesting if this can be used to tailor recovery and treatments in an adaptive way. :)
- karmicthreat 5y ago
  Yea, this seems more like a tool insurance companies will use to deny coverage.
light_hue_1 5y ago
Papers like this need machine learning reviewers and even better, coauthors. I cannot believe that anyone with competence in ML reviewed this paper. For one thing, the figure that the linked article presents is very misleading. It is not the performance of the model! It's just the raw data. Patients by how well we think they might recover vs. how well they discovered. The goal of the model is to figure out how will recover well. Literally, nothing in that figure is an output of the model. > In case of stroke lesions affecting the performance of parcellation, the correspondence part of the lesion on the unaffected hemisphere was used and interpolated over the lesioned brain as a brain transplant and fed into the above-described structural imaging processing. The classifier will trivially discover that you are doing this, and it's almost certainly correlated with terrible outcomes. (PS: should be "corresponding" part) This alone calls into question all of the results and never should have gotten past anyone. Just because you can't tell as a human, doesn't mean the data manipulation isn't immediately obvious to even a simple model like an SVM. > By means of k-means clustering, patients were identified as fitters, i.e. revealing a proportional recovery, and non-fitters, i.e. patients lacking proportional recovery. Good god, why? Just set up a simple criteria and use it. This kind of strange arbitrary boundary in your data that comes from k-means can only hurt your model. Finally: The results are terrible. When cross-validating the model does well, 83% accuracy (87% precision). But on the external data the model is worthless. 60% accuracy, 53% precision. They provide no trivial baselines. They don't even list of the base rates for their dataset (I'm guessing that out of their main dataset the classifier is just operating at chance at 60% accuracy / 53% precision is just the base rate). Their data seems to be horrific. Half the patients in the 2nd dataset don't have data for both time points. How can they be assigned to groups that fit or don't fit without this basic data? They don't even describe results on their 3rd dataset. They don't bother to compute a confidence interval, which will be huge because they have so little data. Seriously. Authors. If you are reading this, reach out to a machine learning person. If you don't know anyone, reach out to me, I'm a scientist at a major US university. Papers like this just pollute our science and are a total waste of time on your part.
- rscho 5y ago
  You're just not used to clinical research. A large part of the field output is no better than this. And this is only what the authors think will look good on paper, which is often quite far from what really happened during data collection and analysis. I should know, I'm a clinical researcher at a major european uni ;-) And due to peer pressure, my output is also no better... :-(
  - AlecSchueler 5y ago
    Would be curious to hear more about the systemic issues that lead to you knowingly outputting poor research. Could you perhaps blog about it anonymously?
- riahi 5y ago
  Agreed. I try to accept all ML focused paper review requests in my disciplines, but modern academics do not reward you for the time you take as a reviewer, which then distracts me from working on my own projects. It's very much an uncompensated service to the community...
deeviant 5y ago
What is the value of such a prediction?
- klyrs 5y ago
  To teach a robot when to pull the plug? "The future" sucks

38 points by rustoo 5y ago

Hope this doesn't mean that we hold back some treatment for those who would have bucked the trend and recovered better than expected... Very interesting if this can be used to tailor recovery and treatments in an adaptive way. :)

karmicthreat 5y ago
Yea, this seems more like a tool insurance companies will use to deny coverage.

light_hue_1 5y ago

Papers like this need machine learning reviewers and even better, coauthors. I cannot believe that anyone with competence in ML reviewed this paper. For one thing, the figure that the linked article presents is very misleading. It is not the performance of the model! It's just the raw data. Patients by how well we think they might recover vs. how well they discovered. The goal of the model is to figure out how will recover well. Literally, nothing in that figure is an output of the model. > In case of stroke lesions affecting the performance of parcellation, the correspondence part of the lesion on the unaffected hemisphere was used and interpolated over the lesioned brain as a brain transplant and fed into the above-described structural imaging processing. The classifier will trivially discover that you are doing this, and it's almost certainly correlated with terrible outcomes. (PS: should be "corresponding" part) This alone calls into question all of the results and never should have gotten past anyone. Just because you can't tell as a human, doesn't mean the data manipulation isn't immediately obvious to even a simple model like an SVM. > By means of k-means clustering, patients were identified as fitters, i.e. revealing a proportional recovery, and non-fitters, i.e. patients lacking proportional recovery. Good god, why? Just set up a simple criteria and use it. This kind of strange arbitrary boundary in your data that comes from k-means can only hurt your model. Finally: The results are terrible. When cross-validating the model does well, 83% accuracy (87% precision). But on the external data the model is worthless. 60% accuracy, 53% precision. They provide no trivial baselines. They don't even list of the base rates for their dataset (I'm guessing that out of their main dataset the classifier is just operating at chance at 60% accuracy / 53% precision is just the base rate). Their data seems to be horrific. Half the patients in the 2nd dataset don't have data for both time points. How can they be assigned to groups that fit or don't fit without this basic data? They don't even describe results on their 3rd dataset. They don't bother to compute a confidence interval, which will be huge because they have so little data. Seriously. Authors. If you are reading this, reach out to a machine learning person. If you don't know anyone, reach out to me, I'm a scientist at a major US university. Papers like this just pollute our science and are a total waste of time on your part.

rscho 5y ago
You're just not used to clinical research. A large part of the field output is no better than this. And this is only what the authors think will look good on paper, which is often quite far from what really happened during data collection and analysis. I should know, I'm a clinical researcher at a major european uni ;-) And due to peer pressure, my output is also no better... :-(
- AlecSchueler 5y ago
  Would be curious to hear more about the systemic issues that lead to you knowingly outputting poor research. Could you perhaps blog about it anonymously?
riahi 5y ago
Agreed. I try to accept all ML focused paper review requests in my disciplines, but modern academics do not reward you for the time you take as a reviewer, which then distracts me from working on my own projects. It's very much an uncompensated service to the community...

deeviant 5y ago

What is the value of such a prediction?

klyrs 5y ago
To teach a robot when to pull the plug? "The future" sucks