8 ms·

So I've actually tried to implement Eulerian video magnification myself, albeit using Python, because I believe when I first looked into this, ~5 year ago, the

So I've actually tried to implement Eulerian video magnification myself, albeit using Python, because I believe when I first looked into this, ~5 year ago, the code was MatLab only. I was really excited about it, because, as a cardiologist, I immediately thought we could use this to identify irregular heart beats in the patient waiting room, for example. This was pre Apple watch etc. I was thinking people could briefly sit in the chair in front of the camera if they are willing to be recorded, as a quick screening whilst they wait for the doc. Unfortunately, I found it incredibly difficult to reproduce their results, even using similar data. I remember videoing my foot as I have a relatively prominent posterior tibial pulse, which in good light I can actually make out visually (but not in the video I took, as intended). The Eulerian magnification didn't seem to do anything. Has anyone else had better luck? I did wonder if maybe I had to use specific video capture equipment, certain frame rates or filters, but if so that wasn't clear from the documentation? It's been 8 years since this paper came out, and yet I still haven't seen it being used in the 'real world', so I am slightly suspicious that I am not alone in my experiences.

MByte 6y ago
I played with it 7 years ago for a hackathon -- not to transform the video, but just to extract pulse data from a face cam. For that purpose, even without optimization, it did work fairly well, though when we went to demo it live, the frequency of the projector lamp interfered with detection. https://github.com/MegaByte/web-webcam-pulse-detector/tree/raw_feed https://github.com/MegaByte/web-webcam-pulse-detector/tree/r... I did run across this company recently, which seems to be doing something similar: https://www.binah.ai/ https://www.binah.ai/
mkl 6y ago
I haven't played with any myself, but there seem to be quite a few implementations on GitHub now: https://duckduckgo.com/?q=eulerian+video+magnification+site%3Agithub.com&atb=v135-5__&ia=web https://duckduckgo.com/?q=eulerian+video+magnification+site%...
oh_sigh 6y ago
This or similar tech would definitely be integrated into high end baby monitors if it worked. State of the art is putting a wrist or ankle band onto your baby though AFAICT.
- ipsum2 6y ago
  Seems like it works fine for babies, for measuring breathing: https://youtu.be/GrMLeEcSNzY?t=41 https://youtu.be/GrMLeEcSNzY?t=41
  - codetrotter 6y ago
    This sort of video motion magnification seems like it could have a use in cinema movies, in order to accentuate motion that the director wants the audience to notice. Even in little videos on Instagram it could be a cool and eye-catching effect.
46Bit 6y ago
I did a Masters project on this sort of technology and yes, I think it makes fantastic demos but hasn't found a killer usecase in the real-world (publicly, anyway.) Their techniques are at least patent-pending if not fully patented by now, which is going to limit adoption. Also, tuning parameters is hell.
somethingsome 6y ago
A friend reimplementated it in python and it worked pretty well for analyzing cardiac liver movements[1]! I don't think he is willing to share the code, but my guess is that a python implementation should work just fine, with very nice results :) (I saw the results applied in his research on videos with impressive results). It took him something like several months to have a code without major bugs. [1] Hahn, S., Absil, J., Debeir, O., & Metens, T. (2019). Assessment of cardiac‐driven liver movements with filtered harmonic phase image representation, optical flow quantification, and motion amplification. Magnetic resonance in medicine, 81(4), 2788-2798.
- hervature 6y ago
  Doesn't want to share the code because it works or because it doesn't work? Disgrace to science.
  - dr_zoidberg 6y ago
    As much as I'd like that code to be shared, it's the authors decision in the end. Think of it as being a margin too small to write the proof.
  - somethingsome 6y ago
    You are free to contact him! I'm not sure he is not willing to share. Sharing code to the public in research depends on several factors, it can get open sourced when the thesis is done, it may depends on who is paying the research (eg: some industrial PhD are not public), long term plan exist to publish the code but there are only 24h a day and a whole lot of work to do, the author/uni may want to make a business after the thesis, and a lot of other parametres. It gets even worse when the code is used in medical applications as a validation process with all the relevant authorities has to be done. This can take a lot of time and freeze the code in the waiting for a certification. Also sharing code between researchers is very usual, it doesn't mean the code has to go public. In this case I think it is part because the original matlab code is patented, research is ongoing and as it is research code it is not necessarily public ready. It doesn't mean he will never release it. If you desperately need it, the matlab code is available, and has the same functionality, you are free to adapt it or use it directly (for research purposes). I must say I don't like that the original code is patented, but I see hundreds of reasons why the authors patented it even if I don't particularly agree :)
nolanhergert89 6y ago
For detecting a pulse from the face, I was able to get good results using pixel tracking and spatial averaging with no fancy math. However if illumination at all changed due to motion, it threw things off considerably without a more work or band passing around the heart rate frequency. (assuming too much in my book) More notes and code here: http://nhergert.ozeo.org/doku.php?id=projects:remoteppg http://nhergert.ozeo.org/doku.php?id=projects:remoteppg
djmips 6y ago
I saw a Steve Mould video on this recently. https://youtu.be/rEoc0YoALt0 https://youtu.be/rEoc0YoALt0 The guy who used it seems to imply his version had a bit of 'special sauce'
- mark-r 6y ago
  Every huckster in the world will claim "special sauce".
logotype 6y ago
I’d guess you get better results by using a long wave infrared camera instead
Firecracker 6y ago
My friend and I hacked on something like this for a while, and eventually got something pretty accurate for measuring heart rate. Shipped on iOS as [0]. It did indeed require a lot of additional tweaks and benchmarking. Honestly don't think any part of our original algorithm or the tricks in the paper made it into the final version. Just doing a Fourier type thing works rarely (~20%), but the principle does work; you can get it working ~90% of the time. That said, I'd be surprised irregular heartbeat were doable in a short recording period. But I'd love to be surprised! [0] Hands Free Heart Rate Monitor (free, private) https://apps.apple.com/app/id1457939511 https://apps.apple.com/app/id1457939511
ginko 6y ago
A colleague and me implemented it for a university lab course a while ago using a combination of C++ and Python. (Used C++ for the processing and Python for the UI and filter design) One important thing we learned is that the filter design is a very important part of the whole thing. We used scipy.signal[1] for filter design. One thing that was very noticeable was that even small IIR filters performed significantly better than FIR. I thought we put our code online a while ago but looks like we didn't. If you're still interested I might have a look if I can put my stuff online later today. Not that it's all that fancy anyways. The C++ part is pure processing and the python interface just puts a gui around it for filter design and calling the application. [1] https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/signal.html#filter-design https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/s...
cantagi 6y ago
I once tried to reproduce a webcam PPG paper, but not using Eulerian video magnification. The pipeline started with face detection, then found the mean colour value of a rectangle of forehead. It then did some DSP and returned the pulse frequency. It worked, but only when the person held extremely still very close to the camera, with auto exposure turned off. Disappointed with how badly mine worked, I took a look at some open source webcam PPG projects to find out how they were working. It turns out they were pretty similar, except most of them were aggressively filtering the pulse signal to be in the correct range, without really checking there was already a signal there. Of course by taking white noise signal and filtering it to a certain frequency range, you'll get a signal in that frequency range. I now believe that to make video PPG work, you need some of (1) a very good camera, i.e. not just a low end webcam (2) quite a long recording, i.e. 10s-1m (3) a controlled environment

MByte 6y ago

I played with it 7 years ago for a hackathon -- not to transform the video, but just to extract pulse data from a face cam. For that purpose, even without optimization, it did work fairly well, though when we went to demo it live, the frequency of the projector lamp interfered with detection. https://github.com/MegaByte/web-webcam-pulse-detector/tree/raw_feed https://github.com/MegaByte/web-webcam-pulse-detector/tree/r... I did run across this company recently, which seems to be doing something similar: https://www.binah.ai/ https://www.binah.ai/

mkl 6y ago

I haven't played with any myself, but there seem to be quite a few implementations on GitHub now: https://duckduckgo.com/?q=eulerian+video+magnification+site%3Agithub.com&atb=v135-5__&ia=web https://duckduckgo.com/?q=eulerian+video+magnification+site%...

oh_sigh 6y ago

This or similar tech would definitely be integrated into high end baby monitors if it worked. State of the art is putting a wrist or ankle band onto your baby though AFAICT.

ipsum2 6y ago
Seems like it works fine for babies, for measuring breathing: https://youtu.be/GrMLeEcSNzY?t=41 https://youtu.be/GrMLeEcSNzY?t=41
- codetrotter 6y ago
  This sort of video motion magnification seems like it could have a use in cinema movies, in order to accentuate motion that the director wants the audience to notice. Even in little videos on Instagram it could be a cool and eye-catching effect.

46Bit 6y ago

I did a Masters project on this sort of technology and yes, I think it makes fantastic demos but hasn't found a killer usecase in the real-world (publicly, anyway.) Their techniques are at least patent-pending if not fully patented by now, which is going to limit adoption. Also, tuning parameters is hell.

somethingsome 6y ago

A friend reimplementated it in python and it worked pretty well for analyzing cardiac liver movements[1]! I don't think he is willing to share the code, but my guess is that a python implementation should work just fine, with very nice results :) (I saw the results applied in his research on videos with impressive results). It took him something like several months to have a code without major bugs. [1] Hahn, S., Absil, J., Debeir, O., & Metens, T. (2019). Assessment of cardiac‐driven liver movements with filtered harmonic phase image representation, optical flow quantification, and motion amplification. Magnetic resonance in medicine, 81(4), 2788-2798.

hervature 6y ago
Doesn't want to share the code because it works or because it doesn't work? Disgrace to science.
- dr_zoidberg 6y ago
  As much as I'd like that code to be shared, it's the authors decision in the end. Think of it as being a margin too small to write the proof.
- somethingsome 6y ago
  You are free to contact him! I'm not sure he is not willing to share. Sharing code to the public in research depends on several factors, it can get open sourced when the thesis is done, it may depends on who is paying the research (eg: some industrial PhD are not public), long term plan exist to publish the code but there are only 24h a day and a whole lot of work to do, the author/uni may want to make a business after the thesis, and a lot of other parametres. It gets even worse when the code is used in medical applications as a validation process with all the relevant authorities has to be done. This can take a lot of time and freeze the code in the waiting for a certification. Also sharing code between researchers is very usual, it doesn't mean the code has to go public. In this case I think it is part because the original matlab code is patented, research is ongoing and as it is research code it is not necessarily public ready. It doesn't mean he will never release it. If you desperately need it, the matlab code is available, and has the same functionality, you are free to adapt it or use it directly (for research purposes). I must say I don't like that the original code is patented, but I see hundreds of reasons why the authors patented it even if I don't particularly agree :)

nolanhergert89 6y ago

For detecting a pulse from the face, I was able to get good results using pixel tracking and spatial averaging with no fancy math. However if illumination at all changed due to motion, it threw things off considerably without a more work or band passing around the heart rate frequency. (assuming too much in my book) More notes and code here: http://nhergert.ozeo.org/doku.php?id=projects:remoteppg http://nhergert.ozeo.org/doku.php?id=projects:remoteppg

djmips 6y ago

I saw a Steve Mould video on this recently. https://youtu.be/rEoc0YoALt0 https://youtu.be/rEoc0YoALt0 The guy who used it seems to imply his version had a bit of 'special sauce'

mark-r 6y ago
Every huckster in the world will claim "special sauce".

logotype 6y ago

I’d guess you get better results by using a long wave infrared camera instead

Firecracker 6y ago

My friend and I hacked on something like this for a while, and eventually got something pretty accurate for measuring heart rate. Shipped on iOS as [0]. It did indeed require a lot of additional tweaks and benchmarking. Honestly don't think any part of our original algorithm or the tricks in the paper made it into the final version. Just doing a Fourier type thing works rarely (~20%), but the principle does work; you can get it working ~90% of the time. That said, I'd be surprised irregular heartbeat were doable in a short recording period. But I'd love to be surprised! [0] Hands Free Heart Rate Monitor (free, private) https://apps.apple.com/app/id1457939511 https://apps.apple.com/app/id1457939511

ginko 6y ago

A colleague and me implemented it for a university lab course a while ago using a combination of C++ and Python. (Used C++ for the processing and Python for the UI and filter design) One important thing we learned is that the filter design is a very important part of the whole thing. We used scipy.signal[1] for filter design. One thing that was very noticeable was that even small IIR filters performed significantly better than FIR. I thought we put our code online a while ago but looks like we didn't. If you're still interested I might have a look if I can put my stuff online later today. Not that it's all that fancy anyways. The C++ part is pure processing and the python interface just puts a gui around it for filter design and calling the application. [1] https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/signal.html#filter-design https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/s...

cantagi 6y ago

I once tried to reproduce a webcam PPG paper, but not using Eulerian video magnification. The pipeline started with face detection, then found the mean colour value of a rectangle of forehead. It then did some DSP and returned the pulse frequency. It worked, but only when the person held extremely still very close to the camera, with auto exposure turned off. Disappointed with how badly mine worked, I took a look at some open source webcam PPG projects to find out how they were working. It turns out they were pretty similar, except most of them were aggressively filtering the pulse signal to be in the correct range, without really checking there was already a signal there. Of course by taking white noise signal and filtering it to a certain frequency range, you'll get a signal in that frequency range. I now believe that to make video PPG work, you need some of (1) a very good camera, i.e. not just a low end webcam (2) quite a long recording, i.e. 10s-1m (3) a controlled environment