5 ms·

Measuring heart rate with a smartphone camera

48 points by yati 13y ago

VladRussian2 13y ago
basically a smartphone size polygraph or a software version of Dr. Lightman ("Lie to me"). Heart rate, eyes expansion/contraction, pattern recognition on face muscle twitching, like of lips, eyelids, ... Could something like it speed up, for example, TSA lines at airports? Automated "checkout" lines like in grocery stores - face camera, hand palm on touch pad, "Are you a terrorist with a bomb? Choose Yes or No" (reminds a gun vending machine scene from South Park if i remember correctly :)
- dan1234 13y ago
  How would it tell between liars and nervous people?
  - Jare 13y ago
    They have to be nervous about something, right? Always err on the side of caution!
    - easy_rider 13y ago
      I'm always nervous at airport security, even though I won't have anything to hide. Not to mention all the real anxiety sufferers.
      - Jare 13y ago
        I was just being sarcastic. Crossing immigration while nervous today already triggers interesting reactions from border officers. If/when they start using this kind of tech, I doubt they will be more understanding and patient.
JazCE 13y ago
I had been wondering about this myself. Thank you for the breakdown. Most interesting.
tzs 13y ago
That talks about putting a finger right on the camera. You don't actually have to go anywhere near the camera to measure heart rate. See the iPhone app Cardiio [1] which gets your heart rate by looking at your face. The technique behind this, Eulerian Video Magnification, is pretty cool. There is a description and a very nifty video at [2]. [1] http://www.cardiio.com http://www.cardiio.com [2] http://people.csail.mit.edu/mrub/vidmag/ http://people.csail.mit.edu/mrub/vidmag/
- yati 13y ago
  This is amazing. Thanks for the links :)
- d0mine 13y ago
  You can even wear a mask http://web.mit.edu/newsoffice/2013/seeing-the-human-pulse-0620.html http://web.mit.edu/newsoffice/2013/seeing-the-human-pulse-06...
- Osmium 13y ago
  I think this is the technique used by Philips in their app too: https://itunes.apple.com/gb/app/vital-signs-camera-philips/id474433446?mt=8 https://itunes.apple.com/gb/app/vital-signs-camera-philips/i... [iTunes link] What would be ideal is if the app also took your heart rate using a combination of both methods to reduce error, but I don't think either do this.
- tagawa 13y ago
  I guess with the finger-on-the-camera version you don't have the risk of the camera auto-adjusting for slight variances in brightness.
gmac 13y ago
I've been toying with a similar setup. Most of the time, you can actually get a really good estimate of the heart rate by doing simple-minded peak detection on the raw averaged brightness signal (e.g. using the algorithm here: http://www.billauer.co.il/peakdet.html http://www.billauer.co.il/peakdet.html). However, the iPhone camera can't be prevented from periodically adjusting exposure parameters (as far as I've been able to tell), and I've not been able to stop this screwing things up once or twice a minute. In other projects I've had some success using the FFT and windowing functions of Apple's Accelerate.framework, so I may see now if the Matlab code shown here can be translated to run in real-time on the phone.
- objclxt 13y ago
  You should be able to lock the exposure on the iPhone camera by setting AVCaptureExposureMode to lock (you can do the same for the white balance as well). But I might be wrong! It's been a while since I did camera work on the SDK.
alx 13y ago
website is down, is there a mirror?
- cbolat 13y ago
  it works fine for me, but images seems broken. http://www.ignaciomellado.es/blog/Measuring-heart-rate-with-a-smartphone-camera http://www.ignaciomellado.es/blog/Measuring-heart-rate-with-...
- uavster 13y ago
  Sorry for that. It seems that it ended up at some social networks today and I'm having much more concurrent visits than often. My hosting provider thought that I was being hacked and closed access for some minutes but now it's up again. I just talked to them and they increased the connection limit, so things should get better shortly. Sorry again, I hope you can enjoy the article and subscribe to the RSS feed if interested (http://ignaciomellado.es/blog/feed http://ignaciomellado.es/blog/feed). You can also reach me on Twitter @uavster.
- uavster 13y ago
  It works fine now. Thank you for your patience.
DieBuche 13y ago
Mirror: http://www.readability.com/articles/jywwrz1h http://www.readability.com/articles/jywwrz1h
BrianPetro 13y ago
I've seen and used apps using this tech(or similar). Ideally this can be used to evaluate need of emergency service automatically. For instance, if someone collapsed while holding the phone and sensors were able to alert EMS. Idk, maybe wishful thinking.
sharpneli 13y ago
Great article. I just cannot contain my need to nitpick (I have background in Ultrasound signal processing). It's Hann window. Not Hanning window. There is a window function called Hamming window due to which some mistakenly add the 'ing' to Hann window too. Second of all "In summary, with a 6-second window, we get a tolerable 6-second startup delay that gives a fair time accuracy of 6 seconds and a fair frequency accuracy of 5 bpm (half the FFT resolution)" is bit incorrect. You do get far more accuracy in determining the frequency of that single signal. What the window length determines is the limit where you can separate 2 signals. So using that window you would mix up 2 signals that are 5bpm or closer to eachother and they would look like a single signal. Having 2 humans with similar heart rates in same video stream would thus be unseparable using that length. But you can determine the heart rate of a single individual with far more accuracy. The correct term would be "precision" as per http://en.wikipedia.org/wiki/File:Accuracy_and_precision.svg http://en.wikipedia.org/wiki/File:Accuracy_and_precision.svg Nonetheless this is an excellent article.

48 points by yati 13y ago

basically a smartphone size polygraph or a software version of Dr. Lightman ("Lie to me"). Heart rate, eyes expansion/contraction, pattern recognition on face muscle twitching, like of lips, eyelids, ... Could something like it speed up, for example, TSA lines at airports? Automated "checkout" lines like in grocery stores - face camera, hand palm on touch pad, "Are you a terrorist with a bomb? Choose Yes or No" (reminds a gun vending machine scene from South Park if i remember correctly :)

dan1234 13y ago
How would it tell between liars and nervous people?
- Jare 13y ago
  They have to be nervous about something, right? Always err on the side of caution!
  - easy_rider 13y ago
    I'm always nervous at airport security, even though I won't have anything to hide. Not to mention all the real anxiety sufferers.
    - Jare 13y ago
      I was just being sarcastic. Crossing immigration while nervous today already triggers interesting reactions from border officers. If/when they start using this kind of tech, I doubt they will be more understanding and patient.

JazCE 13y ago

I had been wondering about this myself. Thank you for the breakdown. Most interesting.

tzs 13y ago

That talks about putting a finger right on the camera. You don't actually have to go anywhere near the camera to measure heart rate. See the iPhone app Cardiio [1] which gets your heart rate by looking at your face. The technique behind this, Eulerian Video Magnification, is pretty cool. There is a description and a very nifty video at [2]. [1] http://www.cardiio.com http://www.cardiio.com [2] http://people.csail.mit.edu/mrub/vidmag/ http://people.csail.mit.edu/mrub/vidmag/

yati 13y ago
This is amazing. Thanks for the links :)
d0mine 13y ago
You can even wear a mask http://web.mit.edu/newsoffice/2013/seeing-the-human-pulse-0620.html http://web.mit.edu/newsoffice/2013/seeing-the-human-pulse-06...
Osmium 13y ago
I think this is the technique used by Philips in their app too: https://itunes.apple.com/gb/app/vital-signs-camera-philips/id474433446?mt=8 https://itunes.apple.com/gb/app/vital-signs-camera-philips/i... [iTunes link] What would be ideal is if the app also took your heart rate using a combination of both methods to reduce error, but I don't think either do this.
tagawa 13y ago
I guess with the finger-on-the-camera version you don't have the risk of the camera auto-adjusting for slight variances in brightness.

gmac 13y ago

I've been toying with a similar setup. Most of the time, you can actually get a really good estimate of the heart rate by doing simple-minded peak detection on the raw averaged brightness signal (e.g. using the algorithm here: http://www.billauer.co.il/peakdet.html http://www.billauer.co.il/peakdet.html). However, the iPhone camera can't be prevented from periodically adjusting exposure parameters (as far as I've been able to tell), and I've not been able to stop this screwing things up once or twice a minute. In other projects I've had some success using the FFT and windowing functions of Apple's Accelerate.framework, so I may see now if the Matlab code shown here can be translated to run in real-time on the phone.

objclxt 13y ago
You should be able to lock the exposure on the iPhone camera by setting AVCaptureExposureMode to lock (you can do the same for the white balance as well). But I might be wrong! It's been a while since I did camera work on the SDK.

alx 13y ago

website is down, is there a mirror?

cbolat 13y ago
it works fine for me, but images seems broken. http://www.ignaciomellado.es/blog/Measuring-heart-rate-with-a-smartphone-camera http://www.ignaciomellado.es/blog/Measuring-heart-rate-with-...
uavster 13y ago
Sorry for that. It seems that it ended up at some social networks today and I'm having much more concurrent visits than often. My hosting provider thought that I was being hacked and closed access for some minutes but now it's up again. I just talked to them and they increased the connection limit, so things should get better shortly. Sorry again, I hope you can enjoy the article and subscribe to the RSS feed if interested (http://ignaciomellado.es/blog/feed http://ignaciomellado.es/blog/feed). You can also reach me on Twitter @uavster.
uavster 13y ago
It works fine now. Thank you for your patience.

DieBuche 13y ago

Mirror: http://www.readability.com/articles/jywwrz1h http://www.readability.com/articles/jywwrz1h

BrianPetro 13y ago

I've seen and used apps using this tech(or similar). Ideally this can be used to evaluate need of emergency service automatically. For instance, if someone collapsed while holding the phone and sensors were able to alert EMS. Idk, maybe wishful thinking.

sharpneli 13y ago

Great article. I just cannot contain my need to nitpick (I have background in Ultrasound signal processing). It's Hann window. Not Hanning window. There is a window function called Hamming window due to which some mistakenly add the 'ing' to Hann window too. Second of all "In summary, with a 6-second window, we get a tolerable 6-second startup delay that gives a fair time accuracy of 6 seconds and a fair frequency accuracy of 5 bpm (half the FFT resolution)" is bit incorrect. You do get far more accuracy in determining the frequency of that single signal. What the window length determines is the limit where you can separate 2 signals. So using that window you would mix up 2 signals that are 5bpm or closer to eachother and they would look like a single signal. Having 2 humans with similar heart rates in same video stream would thus be unseparable using that length. But you can determine the heart rate of a single individual with far more accuracy. The correct term would be "precision" as per http://en.wikipedia.org/wiki/File:Accuracy_and_precision.svg http://en.wikipedia.org/wiki/File:Accuracy_and_precision.svg Nonetheless this is an excellent article.