16 ms·
Extracting audio from visual information
- tferraz 12y agoThis algorithm is really incredible, I can only imagine the improvements to this in 20 years
- supahfly_remix 12y agoBecause of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. Can anyone explain what this quirk is?
- thrownaway2424 12y agoIt's explained further on in the press release. The "quirk" is the same one that causes rolling shutter artifacts in videos: reading the camera's sensor one row at a time, instead of all at once.
- saticmotion 12y agoCameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's frame rate.
- kaoD 12y agoSo it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows. So having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.
- lifeformed 12y agoBetween the time the first and last row are read, the object might've moved a little bit. So if you take a picture with your phone from the side window of a moving car, the picture will appear stretched.
- kaoD 12y agoSure, I meant it specifically as a guess of how it's applied to sound extraction and how it means you have ROW samples per frame.
- darkmighty 12y agoWhat's at play is the sampling theorem. Using instantaneous shutter, considering the speed of sound is very high*, every part of the image is resonating essentially the same sound; the sampling theorem says you can only perfectly reconstruct up to F/2 hertz if you sample at F frames per second. The rolling shutter introduces regular variations at at much higher frequency, allowing much better sampling. - Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.
- dokem 12y agoRolling shutter effect. http://en.wikipedia.org/wiki/Rolling_shutter http://en.wikipedia.org/wiki/Rolling_shutter
- ColinDabritz 12y agoThis is a beautiful hack in best tradition of the term.
- petercooper 12y agoAnd now think of how much high definition video, CCTV, and other forms of recordings already exist.. and then think about running large collections of pre-existing video through such algorithms, along with the best speech-to-text in the business. You could have a whole new Wikileaks on your hands :-)
- deleted 12y ago[deleted]
- imaginenore 12y agoThankfully most CCTVs are so crappy and so low framerate, I doubt they can get much out of it. If you want to record a particular person through a window, for instance, you can get a laser microphone that catches the vibrations of the glass.
- petercooper 12y agoYeah, I've heard that's what the spies use, although that does require specific effort. What I find more intriguing about this development is how pre-existing footage could be used. While HD CCTV is certainly not popular, I suspect enough has been said in the presence of existing HD video to incriminate a few people :-)
- protopete 12y agoI think it unlikely that pre-existing footage can be used, because HD video is almost always compressed, thus masking the minute vibrations in the pixels. The algorithm described in the article work best for uncompressed video directly from the image sensor, and they can run it in real-time without needing to store the video.
- anigbrowl 12y agoYou'd be better off learning to lip read.
- jbaiter 12y agoAm I the only one who isn't getting any audio from the video at all? I tried in two different browsers, downloaded the video with youtube-dl and tried to play it with mpv, everything to no avail, it's just a video with no sound.
- mrfusion 12y agoPretty ironic ... I wonder if they could run their algorithm on the video on their page ...
- mzs 12y agoYou probably only downloaded the DASH video, this should work: $ youtube-dl -f 18 http://newsoffice.mit.edu/2014/algorithm-recovers-speech-from-vibrations-0804 http://newsoffice.mit.edu/2014/algorithm-recovers-speech-fro... You can use the -F option to list the available formats.
- adriancooney 12y agoThis is amazing. Imagine taking high definition video of a crowd of people. You could pick out objects nearby and hear what individuals are saying. You could nearly produce a 3D auditorium by sampling different points in a video. Couple this with a 3D camera and an Oculus Rift, you could have something incredible.
- atomicfiredoll 12y agoWhile I'm not sure how closely this matches up, it sounds like you may at least be interested in exploring these concepts: http://en.wikipedia.org/wiki/Cocktail_party_effect http://en.wikipedia.org/wiki/Cocktail_party_effect http://en.wikipedia.org/wiki/Independent_component_analysis http://en.wikipedia.org/wiki/Independent_component_analysis Edit: Spacing.
- lucb1e 12y agoIt doesn't seem to mention what kind of camera you'd need to do this at the mentioned distance (15 feet). I'm assuming it's at least a couple thousand bucks, or maybe even so expensive that most professional photographers won't have it, but does anyone actually know? Or did I miss it in the article?
- tferraz 12y agoit says about the frame rate necessary, we can only infer the price of the camera.
- randomdrake 12y agoThe video goes into a bit more detail and shows a couple of things that address your question. 1) A lot of the tests were done with a camera that costs thousands of dollars. You can see images of the camera used in one of the experiments. 2) They were able to use consumer-grade cameras to also capture sound. Even frequencies up to 5x higher than the actual 60 FPS of the captured video.
- nl 12y agoI wonder how well a GoPro would do? They can do 120fps at 720p or 240fps at WVGA.
- ygra 12y agoNot very well, at least for the initial approach with a high-speed camera. With 240 fps you're limited to 240 samples per second. If I'm not terribly mistaken that would limit you to frequencies up to 120 Hz in the reconstruction. But the GoPro has a rolling shutter as well, so their second approach would be applicable. However, that effectively relies on rows per second and while you have a higher frame rate you have a lower resolution. In the end they could cancel each other out.
- wmf 12y agoI think this is the cheapest (relatively speaking) high-speed camera: http://edgertronic.com/ http://edgertronic.com/ There are also some consumer cameras that can capture almost 1,000 fps from a small sensor window. I wonder if those would work.
- FatalLogic 12y agoIt's reminiscent of the laser microphone, which measures sound-induced vibrations in objects by bouncing a laser off them, and reconstructs the original sound waves http://en.wikipedia.org/wiki/Laser_microphone http://en.wikipedia.org/wiki/Laser_microphone
- __m 12y agoor active and passive sonar in their difference.
- sp332 12y agoThis is from the movie Eagle Eye, right? The evil computer watches the vibrations in a cup of coffee while someone is speaking.
- ChintanGhate 12y agoThe first thought to cross my mind.
- vitd 12y agoThey also did something like this in the recent remake of "Robocop". Some thugs are in a diner having a discussion, and it's caught on CCTV. Later, the main character goes back and analyzes the video and extracts the audio from it, presumably based on some vibrations somewhere in the scene. They don't say, but I got the impression it was from the window vibrations, similar to the technique of shooting a laser at it and watching the vibrations of the reflected light.
- shangxiao 12y agolol that is the first thing that I thought as well. I just passed it off as a typical Hollywood trope, but now I'm quite delighted to see it become a reality.
- TeMPOraL 12y ago> I just passed it off as a typical Hollywood trope I often wonder what makes people to pass off things like that as "typical tropes", where they are obviously realistic and doable.
- ctdonath 12y ago"Any sufficiently advanced technology is indistinguishable from magic." Those who view it as magic will express it as a trope, even if it is merely advanced technology. Just yesterday I chuckled when recalling a James Bond movie involving Bond driving a car in reverse by viewing a back-up camera. At the time, it formed an instant "trope" because it was so cool and novel. Yesterday, I was doing exactly that with the backup camera on my car - obviously realistic and doable technology.
- qzxvwt 12y agoReminds me of Alvin Lucier's "I am sitting in a room" audio experiment.
- infiniteri 12y agoIncredible and terrifying at the same time. If they're doing this kind of stuff right now with consumer cameras, imagine how effective this technology will be in just a few decades. Privacy is fading quickly with the advent of exciting technology like this.
- platz 12y agoyou need a camera that captures at least 2,000 frames per second.
- prutschman 12y agoThey also took advantage of rolling-shutter sensors running at conventional frame rates to get a "murky" audio signal.
- pavel_lishin 12y ago> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.
- Joeboy 12y agoThe audio from the 60fps video sounds pretty bad though, which I suspect is mostly because of inherent maths/physics limitations rather than anything that software can improve. Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt you'd get intelligible speech from current consumer hardware using this technique.
- mnw21cam 12y agoThere is some small possibility of improvement through software techniques, such as maybe data assimilation, which can use information from surrounding time-frames to improve the measurement. This is assuming that the magnitude of vibrations changes a lot slower than the vibrations themselves, which is usually true, and how most audio compression works. It may be able to clean up the sound a little. However, I would say that the results they have obtained so far are very impressive.
- atuladhar 12y agoThe first thing I thought of: a story called "The Extractor" that recently featured on The Truth podcast: http://thetruthpodcast.com/Story/Entries/2014/3/9_The_Extractor.html http://thetruthpodcast.com/Story/Entries/2014/3/9_The_Extrac...
- VanillaCafe 12y agoFrom watching the video, I get the impression that there is a very large amplitude of the input audio -- taking advantage of the "loud" in loudspeaker.
- CWuestefeld 12y agoA good question. I too got that impression from the first example. In the second example (chips bag through glass), the "control" audio had a lot of reverb, which might have been introduced by the phone they acquired it through, but may also suggest that it wasn't just a person talking, but a reproduction through some kind of amplification equipment.
- DINKDINK 12y agoA loudspeaker is a term for a speaker that you traditionally think of when you think of a speaker: https://en.wikipedia.org/wiki/Loudspeaker https://en.wikipedia.org/wiki/Loudspeaker
- ErikRogneby 12y agoI am amazed this wasn't funded by DARPA.
- IvyMike 12y agoI would bet money that this technique is already well known and has been used by certain intelligence agencies.
- MichailP 12y agoI am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.
- jrockway 12y agoThey put the camera behind soundproof glass for one of their tests.
- MichailP 12y agoI know, but I still wonder about level of soundproof/mechanical vibration and the level of mic and camera sensor isolation VS level of optical vibration. In addition sound can travel many different paths since it is mechanical vibration, and traveling through sound proof glass is not the only path. Anw cool project.
- aidos 12y agoI'm sure it's such a tiny amount that it isn't really relevant. Easy enough to test. Remove the bag of chips and just film the floor. See how much signal you can derive from that. Not exactly perfect science but gives a quick indication of "signal" that's reaching the system via other means.
- jrockway 12y agoActually, what I think would be even better would be to record the bag of chips through the window, but play the sound outside where the camera is.
- prutschman 12y agoThe subtitle of the article is "Algorithm recovers speech from the vibrations of a potato-chip bag filmed through soundproof glass."
- abedavis 12y ago
- uptown 12y agoReminds me a little of "Dual Photography" as presented at Siggraph in 2005. All the data you need to construct a new view is available if you know where and how to look: https://www.youtube.com/watch?v=p5_tpq5ejFQ https://www.youtube.com/watch?v=p5_tpq5ejFQ
- moron4hire 12y agoHah! I have the book that is used in the example picture. Though, I suppose everyone who studied CG in the early to mid 2000s has that book.
- silus151 12y agoI think this will be a good help for visually challenged people. I wish to see if this goes in that direction.
- keerthiko 12y agoWouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?
- panzi 12y agoI had the exact same idea.
- panzi 12y agoWhy did I got down voted for that? It's true. Proof: http://www.crackajack.de/2014/08/04/visual-microphone-audio-extracted-from-vibrations-of-a-potato-chip-bag/#comment-1526903732 http://www.crackajack.de/2014/08/04/visual-microphone-audio-...
- ctdonath 12y agoBecause posting "I had the exact same idea" without further interesting commentary is uninteresting.
- panzi 12y agoI only down vote when something is demonstrably wrong or offensive. I thought that's how everyone is doing it, so I wondered what could I have written that was wrong or offensive?
- ctdonath 12y agoDownvotes also apply to "demonstrably content-free". BTW: not everyone uses exactly your own personal guidelines. BTW: HN Guidelines include "Resist complaining about being downmodded. It never does any good, and it makes boring reading."
- guelo 12y agoYou missed the part where the video's frames per second has to be higher than the audio frequency.
- cagataycouk 12y agoBinoculars as bugging devices...
- deleted 12y ago[deleted]
- jballanc 12y agoWhile it's cool to see this technique applied to consumer video, the general idea has been used in international espionage for a while now. In fact, Léon Theremin (yes, of the musical instrument) invented an early device capable of eavesdropping based on window vibrations: http://en.wikipedia.org/wiki/L%C3%A9on_Theremin#Espionage http://en.wikipedia.org/wiki/L%C3%A9on_Theremin#Espionage . I also recall that this is why all of the windows in the White House are fitted with tiny devices that vibrate the windows randomly as a countermeasure.
- clueless123 12y agoAs a kid (in the early 80's!) I accidentally figured this out while "hacking" a sound based "walkie talkie" with a photo cell and a LED pointed at each other through a little telescope, from my house to my neighbors across the bock. The device worked, but had a bug. a hummmm on the receiver. For the longest time I could not figure out what was the "hummm", till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off! Then I realized the humm was the reflection of incandescent light on the window, and the voices where the people in the room making the window vibrate too.. Ahh.. those where great fun days early on "hacker" life as a hacker. :)
- enraged_camel 12y ago>>till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off! ...and that is when little Johnny knew he had to lay off the shrooms. :P I love hearing stories like yours though. I, too, miss the child-like wonder of making discoveries. We take the physical world we live in for granted, but there's just so much out there to see and learn from.
- ranran876 12y agoWell this system would in theory bypass the White House countermeasures
- AndrewKemendo 12y agoThe technologies involved are quite different and this new approach radically simplifies the logistics.
- thisjepisje 12y agoReminds me of how deaf people can "listen" to music with a sheet of paper or a balloon in their hands.
- imjustsaying 12y ago*deaf people ;)
- ProAm 12y agoEither way it's a superpower.
- thisjepisje 12y agodamn it! lol
- mgulaid 12y agoNow I am gonna be suspicious of chip bags and plants everywhere. We will need to invent telepathic communication (mind to mind) to preserve privacy.
- TeMPOraL 12y ago> We will need to invent telepathic communication (mind to mind) to preserve privacy. Or maybe it's time to think how to adapt to a world without privacy?
- deleted 12y ago[deleted]
- johnydepp 12y agoTry noise cancellation on it. And results may be more clear. I did a project on active noise cancellation using neural networks. It should apply here as well.
- anigbrowl 12y agoThey did already, by the sound of the audio.
- yarone 12y agoReminds me of the technique used to uncover "invisible" motion by amplifying the tiny movements found in a video: http://bits.blogs.nytimes.com/2013/02/27/scientists-uncover-invisible-motion-in-video/ http://bits.blogs.nytimes.com/2013/02/27/scientists-uncover-...
- skywhopper 12y agoThis is fun research, but it's basically just a microphone (which works by turning these vibrations into magnetic pulses that generate electric pulses) using a much, much more complex signal processing mechanism (observing photons bouncing off the vibrating surface, translating the digital representation of those photons into a digital simulation of the electric pulses a microphone would generate, and turning that digital simulation into real electric pulses).
- crucialfelix 12y agoReminds me of this great hoax / video piece from a few years ago about French scientists decoding voices that were accidentally encoded into vases in ancient Pompeii http://www.youtube.com/watch?v=ZbpwBTDvXrI http://www.youtube.com/watch?v=ZbpwBTDvXrI It's in French, but this makes it even funnier.
- aus_ 12y agoWhile a different technique, this reminds me of how researchers were able to extract keystrokes based on the sound from each key press. Here is the original paper: http://rakesh.agrawal-family.com/papers/ssp04kba.pdf http://rakesh.agrawal-family.com/papers/ssp04kba.pdf
- quasiconvex 12y agoAlso reminds me of this: http://cnx.org/content/m13224/latest/?collection=col10380/latest http://cnx.org/content/m13224/latest/?collection=col10380/la...
- jbuzbee 12y agoReminds me of the attempts to extract audio from the groves on ancient spun pots: http://en.wikipedia.org/wiki/Archaeoacoustics#Past_interpretations_controversy http://en.wikipedia.org/wiki/Archaeoacoustics#Past_interpret...
- Houshalter 12y agoHow much vibration does sound actually cause an object. I tried humming near various objects and noticed nothing. I used to be really interested in early mechanical microphones and sound recording, but I couldn't find much information on how they work either.
- anigbrowl 12y agoGently put your fingers on a (cheap) speaker cone and then play back some music.
- PeterisP 12y agoYour eyes wouldn't be able to see a 2000 Hz vibration, most vibrations that we perceive as sounds (except from the very low vibrations) are so fast that even with a huge amplitude they'd be just a motion blur for our eyes. Now, a 2000 fps camera can see things that a naked eye can not.
- Houshalter 12y agoBut I should see a blur or something, unless it happens to be vibrating at exactly the frame rate of both of my eyes (and even that would cause some weird effects, see helicopter blades spinning.) Almost missed your comment btw.
- arh68 12y agoWait, so what would be the best reflector for this type of thing? If I could host the ultimate cocktail party, placing ferns/etc all around the room, could I record every conversation with a single camera? What would those ferns look like? What would they be made of? I'm imagining christmas trees made of cellophane fibers or something.
- chadzawistowski 12y agoIf you're allowed to place the ferns, why not simply hide microphones in them?
- JoeAltmaier 12y agoBecause microphones can be detected. When ferns are detected, they are assumed to be innocuous.
- arh68 12y agoI don't know, you may be very far away. The camera could be in space, or maybe on a plane. I mean, if you can hide microphones, the ferns would be irrelevant anyways, apart from being a good distraction. Kind of off topic but just last night I was watching some (silent) lightning in the clouds, thinking how much more localized sound energy is when compared to light. In other words, I can see for light-years but can't hear much anything 100ft away. Or perhaps it's just our sensors are more sensitive to light waves than air pressure waves. Now I realize sound isn't so localized! It leaks. /rant
- e12e 12y agoIf only I hadn't been wearing my tinfoil hat, they'd never have known my plans!
- NAFV_P 12y agoThe watery sounds you find when listening to low bit-rate mp3, there was a bit of that in the video, especially the "Mary had a little lamb..." skit.
- pessimizer 12y agoIf this is something that commodity hardware is now capable of, a total surveillance society is now very cost-effective. Is the stability required from the camera a dealbreaker when it comes to outdoor mounted cameras in moving air, or would it be pretty easy to algorithmically filter that out? i.e. are winds and drafts predictable enough that they could be removed accurately enough for smaller vibrations to remain?
- DarkIye 12y agoCan somebody executive summary this big lump of wooly garbage?
- ctdonath 12y agoTL;DR - sound is a physical phenomenon, pushing on objects (more pronounced on wide thin lightweight things like bags). A fast enough video camera with image enhancement can "see" the sound affecting the object, which then can be translated to recreating the sound from the video image. Not long ago there was a spate of HN articles about apps that could measure your heart rate via the camera (it watches for & measures subtle changes in your skin color which occur during the pulse cycle). This is exactly the same idea, just with a much faster "pulse". I expect the researchers will next discover the "rolling shutter" (a "that's not a bug, it's a feature!" of cell phone cameras) and discover how to extract the audio info without the need for high-framerate cameras. atomatica found a perfect example: http://youtu.be/TKF6nFzpHBU?t=10s http://youtu.be/TKF6nFzpHBU?t=10s