11 ms·
Algorithm recovers speech from a potato-chip bag filmed through glass (2014)
- emilfihlman 8y ago[2014] https://news.ycombinator.com/item?id=8131785 https://news.ycombinator.com/item?id=8131785
- mirimir 8y agoRule of thumb is never having sensitive conversations where such exploits are feasible.
- toomuchtodo 8y agoWith enough processing power, you can listen to anyone as long as you can get an object in frame that’s vibrating with speech (the UK and Chinese CCTV networks comes to mind). Depending on resolution, I’d expect this to be the case for any footage already stored. It’s just a matter of having a distributed computing job kicked off to comb through video data and add the additional audio metadata. Will you only speak of sensitive subjects in rooms with no windows? Will we be silent in public? These are issues where technology, politics, and human rights intersect.
- vonmoltke 8y ago> Will you only speak of sensitive subjects in rooms with no windows? That is exactly why almost all DOD secure areas are windowless.
- toomuchtodo 8y agoDon’t disagree. Scale that to an entire populace though.
- PeterisP 8y agoIn the brave new world that's being built, the entire population (most of it) doesn't get to have true confidentiality if somebody is sufficiently interested in them.
- mirimir 8y agoWell, windowless bathrooms are not uncommon. And most modern construction has walk-in closets, which typically lack windows. Even so, a source of white noise is always prudent.
- dragonwriter 8y ago> That is exactly why almost all DOD secure areas are windowless. I'm pretty sure that a bigger factor in why secure facilities have limitations on windows (and especially on ground floor windows) is physical security.
- vonmoltke 8y agoNot really. Non-operator windows do not degrade physical security that much, or really at all if they are not accessible from the ground. It is primarily danger of surveillance that drives the lack of windows.
- PeterisP 8y ago"Will you only speak of sensitive subjects in rooms with no windows" - yes, the building standards for secure facilities (i.e. a room where discussing classified information above a certain grade is permitted) mandate no external windows, and much more than that.
- PowerfulWizard 8y agoI tried to look this up, info here: https://www.dni.gov/files/NCSC/documents/Regulations/Technical-Specifications-SCIF-Construction.pdf https://www.dni.gov/files/NCSC/documents/Regulations/Technic... (Ch. 3 sec. F, pg. 13) F. SCIF Window Criteria 1. Every effort should be made to minimize or eliminate windows in the SCIF, especially on the ground floor. 2. Windows shall be non-opening. 3. Windows shall be protected by security alarms in accordance with Chapter 7 when they are within 18 feet of the ground or an accessible platform. 4. Windows shall provide visual and acoustic protection. 5. Windows shall be treated to provide RF protection when recommended by the CTTA. 6. All windows less than 18 feet above the ground or from the nearest platform affording access to the window (measured from the bottom of the window), shall be protected against forced entry and meet the standard for the perimeter.
- kryogen1c 8y agoBeat me to it. https://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility?wprov=sfla1 https://en.wikipedia.org/wiki/Sensitive_Compartmented_Inform...
- nickpsecurity 8y agoThere's also the room in a room concept where one is suspended inside the other. You might add things like ultrasound masking or blocking, too, since it was a known attack vector. From there, I thought about, but can't recall if implemented, some double doorway with buffer in between so opening a door didn't leak sounds/signals. Open one, go in, close it, and then go through other.
- mirimir 8y ago> Will you only speak of sensitive subjects in rooms with no windows? Of course. I mean, that's been obvious for at least a decade. > Will we be silent in public? There's no need to be silent. However, one must be aware of surveillance risks, and act accordingly. > These are issues where technology, politics, and human rights intersect. For sure. But just wanting privacy doesn't work. And you must always deal with what's so.
- maxerickson 8y agoDon't sweat the CCTV or recordings: Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.
- zeta0134 8y agoRight, which is why the researchers were reliant on video artifacts caused by the rolling shutter effect. Even at NTSC resolution, that could get you 262.5 scanlines per frame at 60 FPS (525 at ~29.75 FPS, because of interlacing... NTSC is complicated), which could potentially get you ~15 KHz as a baseline sample rate, depending on how prominent the rolling shutter effect actually is for that image sensor, and how much of the frame your vibrating object consumes vertically. This isn't perfect and would require all sorts of calibration, but I could see it being used to recover windows of time at much higher sample rates, and make inferences about the inbetween data with some FFT analysis on what you recovered. Basically, it's more the rate of _scanlines_ that matters for this technique, and the quality of the image sensor used. The rate of full frames isn't the limiter.
- RugnirViking 8y agoThe article quite literally is about a technique to bypass that limitation and do it with a regular 60fps camera.
- maxerickson 8y agoIt sounds like that technique doesn't quite recover the audio though, just a portion of it: While this audio reconstruction wasn’t as faithful as that with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities. Saying it may still be good enough pretty strongly implies that the quality is quite low.
- LukeShu 8y agoBack when this was new, I remember discussing it on Hackaday or somewhere. Someone remarked that they were amazed that the video could pick up on the motion at all, the video looked still to them. I responded that it really was still--the demo video had been lossfully encoded for the web (re-encoded by YouTube?), and the encoder took out the subtle changes. I suspect that most footage already stored is similarly lossfully encoded, and that this technique isn't possible on it.
- jdietrich 8y ago>the UK and Chinese CCTV networks comes to mind Minor nitpick, but the UK doesn't have a CCTV network. It has a huge number of privately owned CCTV cameras and a relatively small number of CCTV cameras operated by individual local authorities and police forces. The privately-owned cameras aren't joined up in any useful way and are often of very poor quality; the publicly owned cameras are overwhelmingly used for real-time monitoring of busy city centre locations. Installing CCTV cameras is cheap and easy, but usefully monitoring them is expensive and difficult, even with whizz-bang CV algorithms. I'm deeply sceptical as to how useful any state-level CCTV network would be for mass surveillance. 20 million 4K/30fps cameras would produce something in the region of two exabytes per day; just storing that data would cost about $7bn per month.
- sbierwagen 8y agoRun the algo on any reflective objects in the field of view, feed the outputted audio to a transcription NN. Save all high confidence transcribed voice as compressed txt. Save the source video if you pass a threshold of danger keywords. ("bomb", "kill", "prime minister", "didn't pay my tv licence")
- jdietrich 8y agoBest case scenario, you'll need a nuclear power station and a few dozen data centers full of ASICs. Oh, and an internet's worth of extra bandwidth.
- gh0zt 8y ago“I’m sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it’s cool and then somebody turns around and uses it for something you never imagined. It’s really nice to have this type of creative stuff.” Thinking atomic bombs :(
- fooker 8y ago>Thinking atomic bombs :( The same ones which have prevented large scale open military conflicts involving superpowers for the last 60+ years? Atom bombs have likely saved more lives than they have taken, if we had conventional wars with modern technology without MAD.
- freewilly1040 8y agoThat’s one read on history. Another is that the world has been tremendously lucky to not have suffered a nuclear holocaust since WWII. Many powerful people have done everything they could to advocate for nuclear bombings in the past 60 years, and we’ve had some extremely close calls with near accidental launches.
- fooker 8y agoYou are right. It might be too soon to definitively say which take is correct. It might be the case that a disaster (natural, economic, alien invasion, etc) might spark yet another world war.
- grkvlt 8y agoI think the thing that saved the world was Stalin's stroke, if he had held onto power for longer he had the right kind of volatile personality to go 'fuck it' and start something..
- JoshuaDavid 8y agoT5
- PhasmaFelis 8y ago
- rjplatte 8y agoAh, yes, I saw this in 2010. They didn't have high enough resolution to detect actual vibrations, so they used an algorithm based on per-pixel color shifts. Really clever.
- devoply 8y agoNow imagine doing this with a high res satellite anywhere on earth.
- zerb 8y agoThat would only be useful if everyone were outside, and you wanted a split second of many different people's conversations. Satellites move rather quickly relative to the Earth's surface.
- devoply 8y agoSome of them do, there are geosynchronous spy satellites. https://www.popsci.com/gaofen-4-worlds-most-powerful-geo-spy-satellite-continues-chinas-great-leap-forward-into-space#page-3 https://www.popsci.com/gaofen-4-worlds-most-powerful-geo-spy... "It may also have a lower resolution video streaming capacity."
- lifeisstillgood 8y agoOk I was coming to say "spy satellites at 40,000 km up - I doubt they can see anything. And if the linked article is correct the Chinese satellites up there have a resolution of 50m - good luck finding a crisp packet. But the new generation "might" have a resolution of 1m. which is insane. Then again, good luck knowing which square meter of the 1/3 of the earths surface you can see, has the crisp packet in. I still think there will be a place for good old bribery corruption and sex spy techniques for a while yet.
- angry_octet 8y agoRecent Chinese optical satellites are thought to have 10cm resolution. All these are low earth orbit. Depends on cloud cover, atmospheric turbulence and look angle. Still impractical to get sound vibrations from that. But a drone with a laser would work for windows. Think listening in on a conversation in a car.
- 8y ago
- an4rchy 8y agoDefinitely some interesting applications. Sort of orthogonal but is there a similar way to get sound from seeing someone's lip's move in a video -- would it then be possible to recover audio/conversations from old video only files? Edit: Guess they can read lips via AI https://www.techemergence.com/machine-learning-that-learns-more-like-humans-an-ai-lip-reading-machine-and-more-this-week-in-artificial-intelligence-11-11-16/ https://www.techemergence.com/machine-learning-that-learns-m... Now getting the high quality sound from just video would be amazing.
- mhb 8y agohttps://news.ycombinator.com/item?id=17681442 https://news.ycombinator.com/item?id=17681442
- Bromskloss 8y ago> Definitely some interesting applications. Eavesdropping is the obvious one. What others do you have in mind?
- spyder 8y agoWhat about lip-reading with reflected Wi-Fi signals :) : https://people.eecs.berkeley.edu/~guanhua/paper/WiHear_MobiCom14.pdf https://people.eecs.berkeley.edu/~guanhua/paper/WiHear_MobiC... "WiHear aims to detect human speech by analyzing radio reflections from mouth movements. It requires individual user to train the system extensively, and can recognize only a limited number of words (6 words) with high accuracy."
- cma 8y agoAt some point it is (controversially) hypothesized that we may be able to pull imprinted recordings off of ancient artifacts: https://en.wikipedia.org/wiki/Archaeoacoustics#Past_interpretations_controversy https://en.wikipedia.org/wiki/Archaeoacoustics#Past_interpre...
- MrQuincle 8y agoI also once thought of that, but maybe there are other ways in which large vibrations have left fingerprints on materials. I don't mean gravitational waves, but really acoustic phenomena. Like a comet impact or a vulcano outburst. What's a delicate material which would be able to record sound but that doesn't get destroyed by time? Clay that dries up is a logical one.
- eganist 8y agoOld glass or porcelain might be the most viable candidates for a first look as they would be cooling off from a heated, more malleable state, potentially capturing sound from the immediate vicinity as they cooled off. I have no science to back this up. It's just a hunch.
- ianai 8y agoThat may be complicated from old glass never fully hardening. But it sounds plausible enough. Possibly mortar in walls could record the workers taking. All sorts of pottery start out malleable, so they might be candidates. Cave paintings might be an option - wonder whether being finger painted would leave biological fingerprints behind - heart rate, for instance. Pretty wild idea. Here’s hoping it has some legs. Maybe someday we’ll learn about places like Stonehenge this way.
- ghusbands 8y agoOld glass doesn't flow and is solid. Google it to see plenty of debunking.
- 8y ago
- Digit-Al 8y agoForgive me, as someone with zero actual knowledge in this field. This may be a naive question, but wouldn't this be fairly easily defeated by playing songs in the background?
- justwalt 8y agoI think that would just add music to the sound recovered.
- nikanj 8y agoA fan would probably work too, as it adds more white noise to the already noisy signal
- seandougall 8y agoAnd might blow the chip bag away entirely :)
- mvid 8y agoIt might be possible to cancel that noise out. But something like a pink noise generator or an office grey noise generator would probably fool it, the same way it would fool a microphone at distance
- joewee 8y agoThis doesn’t work anymore, if it ever did. There is free software capable of isolating audio like a single voice from background noise.
- grkvlt 8y agoNot sure how this can't work, it's just audio jamming, same as for EM waves. You just need to broadcast sufficient noise to cover the signal, i.e. the S:N ratio is such that the receiver (crisp packet, microphone, etc.) can no longer see the signal above the noise floor. Now, this might mean broadcasting a loud noise signal, which could overwhelm your (or other friendly) receivers (ears, etc.) so optimal placement of the jamming noise source becomes an issue. Generally, you want it closer to the threat than you, so that distance attenuation keeps it bearable for you but still jams possible recording devices. Or, introduce high bandwidth noise vibrations into the surfaces of the area you are in, such as windows or walls. Anyway, it's very possible and in use today in secure facilities that must be protected from audio eavesdropping.
- booleandilemma 8y agoBecause of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. That’s a pretty convenient quirk if you ask me.
- zeta0134 8y agoIt's known as the Rolling Shutter effect. It's most commonly observed when attempting to take still imagery of very fast moving objects like rotors on airplanes, but affects pretty much any image sensor that isn't designed specifically to work around it. The following Wikipedia article goes into far more depth, but the basic problem is that an image sensor doesn't sample every pixel instantly, but instead reads them out serially, and thus there's a tiny, tiny fraction of a second in-between each pixel being sampled. It's small enough to not matter for most purposes, but it's still there, and can be useful / detremental depending on the application. https://en.wikipedia.org/wiki/Rolling_shutter https://en.wikipedia.org/wiki/Rolling_shutter
- Sharlin 8y agoAnd it is why mechanical shutters are still a thing in dedicated cameras. It is still much faster to move two pieces of metal or carbon fiber over the sensor than it is to read out a multi-megapixel CMOS sensor. When shooting video mechanical shutters cannot be used so the rolling shutter effect can cause artifacts such as fast-moving subjects appear slanted.
- im3w1l 8y agoBorrowing from rendering, it seems like a solution would be for every pixel to have "double buffers", one for the measurement in progress and one for the past measurement. Then you just need to make sure you can read all the past values in less than a frame and that all the buffer swaps happen in sync.
- zeta0134 8y ago
- jwilk 8y agoDiscussed 6 months ago: https://news.ycombinator.com/item?id=16201745 https://news.ycombinator.com/item?id=16201745
- joewee 8y agoMakes me wonder why this is even possible...video of Oval Office argument... https://m.youtube.com/watch?v=Ija-VZwcznE https://m.youtube.com/watch?v=Ija-VZwcznE
- wiz21c 8y agopushing the argument to its limit : what happens if we can recover any speech from this or that material (moving leaves, bags, lips movements (AI would be good at that), changes in magnetic fields whatever). should we assume that there is no such thing as a "safe place" anymore ? One could argue there never was (because, for example, police can eavesdrop) but, this article implies it can reach a whole new level... How is that kind of technology controlled by those in powers ? Considering Apple is valued at gazillions of dollars, I guess those technologies will be worth a lot and be available to some others very powerful people... James Bond looks quite outdated to me now :-/
- xapata 8y agoMicrophones are already quite good. If you can be seen, you can be "heard". That has been true for many years. I think there was a scene in the movie Enemy of the State involving cone mics. The difference with this technology is that someone can "hear" you even if you're around a corner.
- femto 8y agoIf interested in this, you might also be interested in the lead author's other research papers http://abedavis.com/research.html http://abedavis.com/research.html and his 2016 PhD thesis, which includes the "chip bag" research. http://abedavis.com/thesis.pdf http://abedavis.com/thesis.pdf I'm interested in this research, from the angle of "Can passive surface measurements tell us what is happening inside the human body?"
- StanislavPetrov 8y ago>In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass. Its hard enough to discern intelligible speech from many people who are standing right in front of you.
- test6554 8y agoSo if I eat my sour cream chedar ruffles in a one-party consent state could I get in trouble?
- dbg31415 8y agoAre there any applications for this tech that aren't a massive violation of privacy? Seriously... why develop this? It's super creepy.
- Jaruzel 8y agoTangential... but why do the US call them 'Chips' whereas in the UK we call them 'Crisps'?
- xtrapolate 8y agoA different approach to decoding vibrations into speech, involving a laser microphone: https://en.wikipedia.org/wiki/Laser_microphone https://en.wikipedia.org/wiki/Laser_microphone
- Patient0 8y ago"If I'm sitting next to a swimming pool, and somebody dives in - and she's not too pretty, so I can think of something else - I think of the waves and things that have formed in the water. And, uh, when there's lots of people have dived in the pool there's a very great choppiness of all these waves all over the water and to think that it's possible, maybe, that in those waves there's a clue as to what's happening in the pool. That some sort of insect or something with sufficient cleverness could sit in the corner of the pool and just be disturbed by the waves, and by the nature of the irregularities and bumping of the waves have figured out who jumped in where and when and where what's happening all over the pool. And that's what we're doing when we're looking at something. Uh, the light that comes out is ... is waves, just like in the swimming pool except in three dimensions instead of the two dimensions of the pool it's they're going in all directions. And we have a eighth of an inch black hole into which these things go ... which, uh, is particularly sensitive to the parts of the waves that are coming in a particular direction it's not particularly sensitive when they're coming in at the wrong angle which we say is from the corner of our eye. And if we want to get more information from the corner of our eye we swivel this ball about so that the hole moves from place to place. Then ... uh, it's quite wonderful that we can see ... figure out so easy. That's really because the light waves are easier than the ... the waves in the water are a little bit more complicated it would have been harder for the bug than for us but it's the same idea. Figure out what the thing is that we're looking at at a distance." Transcribed from footage included in the documentary "The Last Journey of a Genius" (1989) by Christopher Sykes, a BBC TV production in association with WGBH Boston and Coronet/MTI Film and Video. https://www.youtube.com/watch?v=1qQQXTMih1A https://www.youtube.com/watch?v=1qQQXTMih1A
- anonu 8y agoI think this has shown up on HN previously... Cant find the link though.
- jordache 8y agoTIL don't leave snack bags out, and the window shades open.
- adrianhel 8y agoThis would also be really cool for recording vocals. I liked the glitchiness of potato chips!