4 ms·

I'm a junior academic physician with a background in CS whose specific research interest is ECG analysis with deep learning. This paper has some glaring holes.

I'm a junior academic physician with a background in CS whose specific research interest is ECG analysis with deep learning. This paper has some glaring holes. Apart from the total lack of external validation (which I suspect will never come), we already know that ECGs have features associated with emotion: heart rate, frequency of premature ventricular contractions, frequency of premature atrial contractions. Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features? This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL? Of course the article is completely silent on these questions. If I was a reviewer I would refuse to accept until they added extra experiments to address this. This is a symptom of a more general problem in interdisciplinary medical research: Papers are written and reviewed either by engineers who have limited understanding of the clinical background or by clinicians who can code a CNN in pytorch but have no knowledge of classic ML methods or ML theory. And so we see seriously flawed papers get accepted. (I know this is a preprint, but I've seen similar stuff in well regarded journals.) It's incredibly frustrating as someone who tries to do truly rigorous research in this space, only to be desk rejected because ECG magic sounds a lot cooler than incremental gains.

ramblenode 3y ago
> This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL? One reason we will see more deep learning in electrophysiology is simply because it allows one to abstract over a good deal of statistics and signal processing, which are hard. Deep learning is being used as a black box for those who can't or don't want to understand their data.
- Calavar 3y ago
  I agree with all your points. I think that training algorithms without understanding the underlying data is fundamentally a bad thing. This isn't NLP, where we're willing to put up with LLMs being inscrutable because they blow traditional algos out of the water. Accuracy with deep learning in ECG is moderately better than traditional methods, not paradigm shifting. And unfortunately the drawbacks of DL in ECG analysis are underanalyzed and underdiscussed. Most published DL ECG models that I've played around with have pretty significant failure modes that would preclude them from being used clinically. Of course, it's research, it's not meant to be clinical grade. But the failure modes aren't mentioned at all in the publications, and I've seen very little in the way of research into clamping down on some of these failure modes.
NeuroCoder 3y ago
There are far too few neuro + medicine + CS people. Being able to coordinate research in two of those happens now days we need all three.
- __loam 3y ago
  Part of the problem is CS people thinking the brain is a computer and biology people thinking ml is magic.
__loam 3y ago
I've done ECG research in the past and these ai reads your mind posts always make me very skeptical.
4death4 3y ago
> Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features? It’s pretty insane to think that a decent neural net trained on sufficient data wouldn’t be able to outperform basic regression.
- ssivark 3y ago
  What makes you think that a neural net must necessarily be significantly better? How much better is significant? Can one actually train a large enough network on (finite) available data? Sometimes it might take god awful amounts of data+compute+layers to rediscover transforms that we have ready access to, once math has matured enough.
  - 4death4 3y ago
    It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.
    - Calavar 3y ago
      > It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point) > But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression. As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.
refibrillator 3y ago
If regression on wavelet coefficients solves your task then sure DL is unnecessary. I spent half a decade on a data science team developing algorithms for ambulatory ECG monitoring at scale. 100+ human years of 256 Hz multichannel ECG collected everyday. Waves, beats, & rhythms all segmented and classified. PDF reports generated for cardiologists and EPs. DL techniques like unsupervised pre-training blew away every other classical or purely supervised method we ever tried. Especially for rare and critical arrhythmias like VF. It unlocked value in massive unlabeled datasets. It gave us a framework for scaling model performance as a function of dataset size and compute, with no fundamental limit. I do get your frustration about the publication game. You might find industry is more aligned with pragmatism & efficiency.
deleted 3y ago
[deleted]

ramblenode 3y ago

> This matters because if you have a method that is more computationally efficient, more easily interpreted, and more robust to extrapolation to out-of-domain data, why in the world would you go with DL? One reason we will see more deep learning in electrophysiology is simply because it allows one to abstract over a good deal of statistics and signal processing, which are hard. Deep learning is being used as a black box for those who can't or don't want to understand their data.

Calavar 3y ago
I agree with all your points. I think that training algorithms without understanding the underlying data is fundamentally a bad thing. This isn't NLP, where we're willing to put up with LLMs being inscrutable because they blow traditional algos out of the water. Accuracy with deep learning in ECG is moderately better than traditional methods, not paradigm shifting. And unfortunately the drawbacks of DL in ECG analysis are underanalyzed and underdiscussed. Most published DL ECG models that I've played around with have pretty significant failure modes that would preclude them from being used clinically. Of course, it's research, it's not meant to be clinical grade. But the failure modes aren't mentioned at all in the publications, and I've seen very little in the way of research into clamping down on some of these failure modes.

NeuroCoder 3y ago

There are far too few neuro + medicine + CS people. Being able to coordinate research in two of those happens now days we need all three.

__loam 3y ago
Part of the problem is CS people thinking the brain is a computer and biology people thinking ml is magic.

__loam 3y ago

I've done ECG research in the past and these ai reads your mind posts always make me very skeptical.

4death4 3y ago

> Are these deep learning detectors better than running a simple regression after using something like a wavelet transform to detect these specific features? It’s pretty insane to think that a decent neural net trained on sufficient data wouldn’t be able to outperform basic regression.

ssivark 3y ago
What makes you think that a neural net must necessarily be significantly better? How much better is significant? Can one actually train a large enough network on (finite) available data? Sometimes it might take god awful amounts of data+compute+layers to rediscover transforms that we have ready access to, once math has matured enough.
- 4death4 3y ago
  It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.
  - Calavar 3y ago
    > It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point) > But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression. As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.

refibrillator 3y ago

If regression on wavelet coefficients solves your task then sure DL is unnecessary. I spent half a decade on a data science team developing algorithms for ambulatory ECG monitoring at scale. 100+ human years of 256 Hz multichannel ECG collected everyday. Waves, beats, & rhythms all segmented and classified. PDF reports generated for cardiologists and EPs. DL techniques like unsupervised pre-training blew away every other classical or purely supervised method we ever tried. Especially for rare and critical arrhythmias like VF. It unlocked value in massive unlabeled datasets. It gave us a framework for scaling model performance as a function of dataset size and compute, with no fundamental limit. I do get your frustration about the publication game. You might find industry is more aligned with pragmatism & efficiency.

deleted 3y ago

[deleted]