11 ms·
Eulerian Video Magnification for Revealing Subtle Changes in the World
- samirahmed 14y agoi wonder if this could be used for lie detection
- tokenadult 14y agoi wonder if this could be used for lie detection I was tempted to downvote, but this is a legitimate question, and I guess you haven't had opportunity to learn about how bogus "lie detection" in general is. I happened to attend university where some of the earliest rigorous researchers on lie detection were active, http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the_polygraph.html http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the... http://images.salon.com/april97/news/news970410.html http://images.salon.com/april97/news/news970410.html http://antipolygraph.org/articles/article-018.shtml http://antipolygraph.org/articles/article-018.shtml http://www.psychologicalscience.org/observer/getArticle.cfm?id=2119 http://www.psychologicalscience.org/observer/getArticle.cfm?... so people in my town have known for a long time that purported lie detection tests don't work, and indeed our state has long banned the use of polygraphs or other purported lie detection devices for any courtroom purpose. Law enforcement investigators in our state use actual interviews of witnesses and suspects, and by state law all police interrogations must be videotaped. That helps immensely in reducing mistaken conclusions from improperly conducted interrogations.
- bane 14y agoI'm actually thinking if it might be a good tool to detect distress in a subject. Not lying exactly, but might provide insight into a subject's emotional state non-invasively during questioning or interrogation. People might become distressed for many reasons that may not just be simple lying. Perhaps even a Voight-Kampff-like test a la Blade Runner/Do Androids Dream of Electric Sheep?
- taliesinb 14y agoIt never made sense to me that an manufactured android would be easiest to detect using a psychology test instead of say, X-raying them.
- JoeAltmaier 14y agoAndroids in that case were fully human, just manufactured. The principle differences were improved physiology and durable mental processes, which could have occurred naturally.
- hikarudo 14y agoIn "Do Androids Dream of Electric Sheep", the androids are practically indistinguishable from humans in their physiology.
- Symmetry 14y agoIts quite true that polygraphs and similar means of telling whether a person is distressed are almost useless. However, there have recently been some advances in using fMRI to do direct lie detection by looking at what parts of a person's brain are being used at a given time. This doesn't seem to be reliable enough for courtroom use yet, but its getting there.
- tsumnia 14y agoI said it above, but to carry on with what tokenadult said; Lie detection is a little more complicated than detecting facial expressions. If you have a defensive, introverted person looking downward, they might be lying or that could be how he responds to whatever interrogation they are under. Conversely, a mentally distressed individual (or someone who has done a heavy amount of NLP) could lie straight to your face because they personally believe they are telling the truth. This type of technology will no doubt be used to attempt lie detection, but it won't work. The only thing we can do is use the context clues of the face to infer what we think is going on.
- fferen 14y agoI like how this kind of turns the "main idea" of computer vision on its head. Namely, instead of trying to emulate and catch up to human abilities, it augments them the way only a computer can; it plays to computers' strengths instead of their weaknesses. Also, who would have thought you can determine someone's pulse just by looking at the color change of the face due to blood rushing in and out? If asked beforehand, I wouldn't have even said it was theoretically (physically) possible.
- wxs 14y agoI also found it very surprising that cameras could pick these things up, there's actually an iPhone app from Philips that claims to be able to be able to read heart-rate, I assumed it was snake-oil at first : http://itunes.apple.com/us/app/vital-signs-camera-philips/id474433446?mt=8 http://itunes.apple.com/us/app/vital-signs-camera-philips/id...
- gmurphy 14y agoI was skeptical too, but when I tried this app a few months ago, I compared it to my Garmin heart rate monitor at rest and after exercise - they were usually within 1bpm of each other. I was very impressed.
- jasonwatkinspdx 14y agoThere is so much faint and indirect information in the world it's dumbfounding. We're crossing a threshold where computation can make this visible. It's really exciting.
- zher 14y agoYes, it's hard to believe, but it's really possible to sense the small color changes due to blood flow in the face using cameras. My earlier work described how you just need a simple webcam :)(http://web.mit.edu/newsoffice/2010/pulse-camera-1004.html http://web.mit.edu/newsoffice/2010/pulse-camera-1004.html). As a result, I'm working on Cardiio (http://cardiio.com http://cardiio.com) to bring this technology to both the web and mobile very soon. If you're interested in computer vision based biosensing, please sign up!
- wxs 14y agoIf people find this compelling, they may also like this earlier work from the same group: http://people.csail.mit.edu/celiu/motionmag/motionmag.html http://people.csail.mit.edu/celiu/motionmag/motionmag.html This other algorithm amplifies only motion, and the algorithm is less elegantly simple, however its results are very interesting and fun to watch as well. Make sure to watch the video at the bottom of the page http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2005.mp4 http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2...
- pbhjpbhj 14y agoI watched that second video http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2005.mp4 http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2... - that must be incredibly useful in engineering analyses.
- zemaj 14y agoWow, reading someone's heart rate from a video. Imagine that plugged into Google Glass. Overlay someone's heart rate next to their face while you're talking to them!
- DannoHung 14y agoHow about amplifying micro-motor motions in the eyes or something like that?
- jtchang 14y agoExcuse me while I book a trip to Vegas.
- reginaldo 14y agoI was thinking about the implications of using this technique to analyze e.g. political speeches and try to catch people lying on the act. Your application (winning on card games) seems very interesting too.
- tsumnia 14y agoCoincidentally so was I! With a person's microgestures amplified, you'd at least notice inherent changes in a person's physical behavior. Now, we need to remember that lying isn't as simple as "he's looking down and to the right" because that can just be a personality trait of the person. With public figures, however, if you collect enough video of various activities, you could determine a baseline physical demeanor. Then, if the figure in question radically steps out of that baseline, you can send off red flags. Ideally, this could be used to analyze facial dynamics to help determine focus, expression (for example, the SOEmote video that was recently posted), mood (though we now get back into personality traits), or, as the video showed, subtle changes in the physical body.
- taliesinb 14y agoI was under the impression that micro-expressions (I assume you're referring to them) were very short in duration rather than in magnitude. So this would be the wrong kind of amplification.
- carterschonwald 14y agoThis is pretty darn amazing! I'm so going to try to implement this set of algorithms later this week! :)
- theschwa 14y agoThe Is He/She flirting with me App is just around the corner. I wonder what new privacy issues will crop up with this kind of technology.
- btipling 14y agoAmazing for many reasons, but what I really find impressive is that messing with video encoding requires like a PHD in everything because of how complex things like H.264 are. At least that's my perspective after trying to grok the H.264 spec. There are all kinds of optimizations and algorithms for network latency and video sizes, etc. I don't consider myself a layman but I certainly do not have the knowledge and skill to do anything at all with video. Do you know what a Inverse sub-macroblock partition scanning process is? I don't, not even after reading the part about it in the spec.
- justinsb 14y agoNo need to mess with the encoding format - just use an existing utility (or library) to grab the raw frame data. e.g. ffmpeg
- ZenPsycho 14y agoa macroblock is a chunk of an image which is identified to have some constancy between frames. a sub-macroblock would be some subdivision of this for finer motion details. sub-Macroblock partitian scanning would be applying an algorithm to identify the sub-macroblocks in a set of images. Add the word "Inverse" though and now I have no idea what the hell that means.
- taliesinb 14y agoThis really doesn't have much to do with what makes computer vision difficult (or rather, interesting). I doubt very many computer vision researchers are conversent with the H.264 spec. Typically toolkits like OpenCV take care of codecs for you (which make them quite annoyingly difficult to compile on different platforms). (I speak as someone who had done commercial computer vision algorithm development.)
- joeconway 14y agoUnfortunately when they do (My dissertation is on motion analysis using H.264 compression artifacts) things get pretty complicated very quickly
- anigbrowl 14y agoI'm not seeing what the big deal is here. Extraction of differential information from video frames has been standard in video editing and compositing software for years. Grab a demo copy of After effects, folks.
- taliesinb 14y agoThis is kind of the stock-negative comment that I don't like about HN. It would be redeemed if it had a link, but instead its just a look-how-cool-I-am-that-I'm-not-impressed-by-this actually-pretty-awesome-technology-demo, dammit.
- tptacek 14y agoThe commenter you are replying to (and downvoting) has done (IIRC) professional digital video work, and written many detailed and helpful comments on HN about digital media. The comment you're replying to isn't snarky. Assume good faith. Maybe he's just mistaken? Or maybe he's right, and the thread just doesn't like that? Here are some best-of's to peruse before concluding that 'anigbrowl is part of the problem with HN: * http://news.ycombinator.com/item?id=1875412 http://news.ycombinator.com/item?id=1875412 (MIDI) * http://news.ycombinator.com/item?id=995096 http://news.ycombinator.com/item?id=995096 (3D film) * http://news.ycombinator.com/item?id=4011770 http://news.ycombinator.com/item?id=4011770 (Google Moog doodle) * http://news.ycombinator.com/item?id=1263289 http://news.ycombinator.com/item?id=1263289 (DV on DSLRs) A couple suggestions: * If a downvoted comment with no replies upsets you so much that you feel a need to pile on with a pointed negative comment, go to the bottom of the page, type the commenter's name in, click 'comments', then sort by 'score', and read the top 10 or so first. You may find the urge to pile on dissipates quickly. * While it certainly isn't going to cause drama here ('anigbrowl being one of the more levelheaded contributors to HN, way more so than me), I think you'll find that comments like yours tend to read as personalizing disputes, and lead to embarrassing spats you'd be better off not attaching yourself to.
- anigbrowl 14y agoblush
- emmett 14y agoIs there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.
- taliesinb 14y agoAccording to the paper, it is really quite a simple idea. Imagine a filter that one might use in an audio context that boosts the amplitude of frequencies falling in a certain range. But now imagine applying that filter to a timeseries that actually comes from the intensity of a single pixel over time. This has the effect of emphasizing motions that occur over timescales that correspond to the selected band of frequencies. The reason this is true actually comes back to the Fourier transform -- you can imagine the motion of part of a complex image to be composed out of the linear combination of many simpler 'sinusoidal motions', each of which is being amplified by the per-pixel filter.
- tomp 14y agoWhat about motion amplification? For example, the baby breathing in the bed... Aren't multiple pixels involved then?
- ClintTorres 14y agoDefinitely. There are two streams of information being analyzed. First is color, observed as one position changing properties from one frame of video to the next. Second is position, which is a change in location of things within the field of view from one frame of video to the next. Both color and position changes can happen at the same time, but it's easier to see how this information is used if you start off by treating them separately. For color changes, imagine video of a stationary person's face where the center pixel of every picture is the center of the face. The color at that center pixel changes from frame to frame. At frame 1 the red value (of an RGB triplet) is maybe 200 (out of 255), and by frame 5 the red value is 203. This is a temporal red shift of just 3 levels over 5 frames. For position changes, imagine video of a beach ball in a still pool of water, where the center of the ball changes from frame to frame. At frame 1, the position is (in x,y) (512,512), and at frame 5, the position is (512, 515). This is a spatial shift of just 3 pixels over 5 frames. In both of these cases, the observed change is probably too smal for humans to notice in casual viewing, so an exaggeration of some kind is desired. To make the change noticeable, we'll take the change of signal (3 values over 5 frames) and "amplify" it. These modified ("amplified") values will be shown on top of the original source video, resulting in an exaggerated change. If a color shifted by 3 red levels, we can show that by varying the color by 30 red levels. If a position changed by 3 pixels, we can show it changing by 12 pixels. We can choose the level of amplification best suited for observing the changes we've isolated. Now the really cool aspect of these data streams is that their resulting "amplifications" don't need to be applied to the same data stream that they were observed in. It's possible to take a position change, and represent it as a color change. It's possible to take a color change, and stretch it out over a longer time (over more video frames). Any combination of properties you like can be used to represent your observed set of changes. Some of the most interesting representations will be a mixture of different data streams, represented as another mixture of exaggerations to the original video.
- riemannzeta 14y agoThese guys are going to make millions on baby monitors alone.
- pbhjpbhj 14y agoMaybe, does it work well in a darkened room? Baby monitors should probably be IR based; then this would appear to be a great addition allowing a pulse to be determined from the video feed. Current baby monitors use a pad underneath to detect movement including breathing. Personally, we used the "you can hear them screaming half-a-mile away" method of baby monitoring.
- RockofStrength 14y agoThis type of technology would be a nice novelty item to implement into a futuristic zombie story. For example, there could be a large battle between humans and zombies, and the humans are equipped with pulse-magnifying glasses, allowing them to identify friend vs. foe.
- vibragiel 14y agoInstant Heart Rate for Android uses something similar to check the pulse using the phone's camera. https://play.google.com/store/apps/details?id=si.modula.android.instantheartrate&hl=en https://play.google.com/store/apps/details?id=si.modula.andr...
- stefantalpalaru 14y agoEven more similar: http://www.whatsmyheartrate.com/ http://www.whatsmyheartrate.com/
- joelthelion 14y agoDoes anyone have a link to the paper?
- apu 14y agoIt's on the page: http://people.csail.mit.edu/mrub/papers/vidmag.pdf http://people.csail.mit.edu/mrub/papers/vidmag.pdf
- twp 14y agoFinally, you can "enchance" videos, just like they do in the movies: http://www.youtube.com/watch?v=Vxq9yj2pVWk http://www.youtube.com/watch?v=Vxq9yj2pVWk
- eridius 14y agoThat was fairly ridiculous, but where on earth did that clip of rotating the video come from?
- newbie12 14y agoThis technology is badly needed today for skin cancer screenings. We could save tens of thousands of lives a year with better detection of cancers and pre-cancerous growths. It is crazy that most screenings in 2012 are still 3 minutes a year with a harried doctor.
- JacobIrwin 14y agoIf anybody could help clarify a concept (a specific filtering method), from video: "we filter the sequence temporally using a "???" ...and boost the temporal signal to generate a video..." (at 2min. 45sec in video) What type of filter is being described (missing from quotes; "???") ? Bonus upvotes for providing a URL to definition (detailed description of the missing term).
- noisebeuter 14y agoHe talks about a "Temporal Bandpass Filter". Not sure about the "temporal" part, but in DSP a bandpass filter is a filter (think Audio-EQ) that passes frequencies within a specific range and rejects the rest. See https://en.wikipedia.org/wiki/Bandpass_filter https://en.wikipedia.org/wiki/Bandpass_filter
- JacobIrwin 14y agoThanks much noisebeuter!
- 0x09 14y agoIt would seem to me from perusing the spec that they are simply taking a video, doing a 1D DCT on the time axis and amplifying a "frame" in the result corresponding to the desired period. Hopefully someone can correct me if there's more to it. I'd like to get at one of their sample videos before filtering because it would be pretty easy to toss together a workalike based on the above.
- jasonkester 14y agoThere's a whole business to be made off that five second clip where they amplify the motion of the sleeping baby's chest. Anybody who's ever been a parent can remember going up to check on the baby at night, standing over the crib for a full minute thinking "move! Move! MOVE!" at the lifeless body of their baby, running all the horror stories they've heard about cot death over and over in their head until finally the kid's chest moves enough to breathe a sigh of relief and go back to bed. The first company that adds this feature to a video baby monitor will make a mint.
- wcoenen 14y agoThat business already exists, though the existing sensors typically come as a pressure-sensitive mat (placed under the mattress) instead of a camera.
- JoeAltmaier 14y agoA business preying off the fears of new parents, marvelous.
- elemeno 14y agoThere's a bit difference between building a business that preys on the fears of a new parent and a business that acknowledges that new parents already have fears and tries to reduce them. This I would say falls firmly into the later category. It's the difference between a tool which helps you make sure that your baby is ok and one which tries to persuade you that if you don't use it you're creating unconscionable risks to your baby which this product could save you from. The former is a tool which is good, the later is one which preys on parents.
- JoeAltmaier 14y agoTotally disagree. The two devices are one and the same. Checking on your baby in the middle of the night because you woke with a sudden fear - normal parent anxiety, makes a funny story later. Providing a device to allow constant electronic monitoring - validating an irrational fear, reinforces anxiety, feeds into an obsessive loop, invents issues for new parents just when they need to be calm and rational.
- psb 14y agoIf someone wants to do a kickstarter to try and use this technology to build a glucose meter, I will be the first contributor
- th0ma5 14y ago:( unfortunately might not be entirely applicable to this problem... unless some measure of blood density and hence probably glucose mix... but at least at this layer of filtering and reasoning, it appears to be about minute amounts of motion... is there something about motion that could denote glucose? trembling maybe? maybe a score could be derived using something that borrows from this technique, but you'd have to put more into it than it seems this technique specifically shows.
- psb 14y agoI was thinking more skin color or possibly changes in the eye (if I remember right eyes get swollen when blood sugar is high) Honestly, I don't know if there are enough outwardly visible changes for this to even be possible. But there's a lot of money out there waiting on the person who can figure out a way to use a cell phone do something like this.
- nhergert 14y agoGood idea, but apparently much money has been spent on it already. Excellent article on the physics of all things non-invasive (but especially glucose): http://www.mendosa.com/noninvasive_glucose.pdf http://www.mendosa.com/noninvasive_glucose.pdf
- mistercow 14y agoThe simplicity and elegance of this algorithm is really breathtaking. I'm mostly thinking out loud here, but I think you could use a related process to do inter-frame interpolation for smooth time-scaling of video.
- 0x09 14y agoYou can! http://ompldr.org/vYWh5eA/test2.mp4 http://ompldr.org/vYWh5eA/test2.mp4 This was made by resampling in time with a DCT. Problems with this: 1. practically, it needs the entire movie in memory. 2. gibbs phenomenon in time is really weird. These could both be addressed by using short overlapping windows, and then the process becomes just like typical sinc/lanczos interpolation in time.
- mistercow 14y agoWell, yes, you can interpolate in the frequency domain like that, which works pretty well for resampling 2D data as well. However, it does not transition shapes from location to location; instead, it looks like one frame is fading into another. What's fascinating about this multiscale algorithm is that it appears to actually exaggerate the motions rather than merely increasing the contrast of the changes.
- siavosh 14y agoFrom my experience with CV, one of the biggest unsolved problems and rarely publicized aspect of the field is how much variable tuning is often required to get good results. I'm curious how sensitive these demos are to different environmental conditions and the robustness of the algorithm.
- Hitchhiker 14y agoThe team's initial work on topic: http://people.csail.mit.edu/celiu/motionmag/motionmag.html http://people.csail.mit.edu/celiu/motionmag/motionmag.html
- RockofStrength 14y agoI have a tangential question: What's the state of technology that would allow one to see through the eyes of another? This would be useful for piano tutorials, etc. I think it could be done with a separate eye-tracking device and a camera located between the eyes.
- bergerj 14y agoWouldn't a keyboard with keys that light up be better and easier?
- regularfry 14y agoYou don't get to play with proprioception that way :-)
- groby_b 14y agoUh, what? If you play piano, you're looking at the score, not the keys.
- RockofStrength 14y agoNot necessarily; I usually don't use a score when I play. It would also be insightful to see the 1st person perspective of a jazz musician playing from a lead sheet, to see how his eyes jump between page and piano.
- apu 14y agoFrom 8 years ago: http://www.cs.columbia.edu/CAVE/projects/world_eye/ http://www.cs.columbia.edu/CAVE/projects/world_eye/ http://www.cs.columbia.edu/CAVE/projects/eyes_relight/ http://www.cs.columbia.edu/CAVE/projects/eyes_relight/
- MPSimmons 14y agoExcellent. Integrate this into the Tricorder projct: http://tricorderproject.org/ http://tricorderproject.org/
- lurker14 14y agoI just downloaded an app[1] that measures heartbeat rate by taking video of a fingertip. I was stunned by the brilliance of the application, how ubiquitous video sensors can replace other senses. There's something else to discuss here: We have a recently published paper from MIT, including mathematical details of the process; and at the same time a stack of closed-source commercial applications already in the market with slick interfaces. Are they using the same technique? Is one app-maker trawling preprints for app ideas? Are many app-makers copying one pioneering app-maker? [1] https://play.google.com/store/apps/details?id=com.macropinch.hydra.android&feature=related_apps https://play.google.com/store/apps/details?id=com.macropinch... but there are many similar apps.
- lurker14 14y agoThis is a similar idea to the temporal arterty thermometer, which uses an infrared camera: https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_artery https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_a... The mathematical model for the sensor is probably different, though.