16 ms·
DIY Acoustic Camera
- onion2k 5y agoThis could be really useful for finding birds in trees when I'm out with my camera...
- loxias 5y agoWould work! One of the first applications I made was a processing layer returning, in spherical coordinates centered at an arbitrary reference point, what the system determined as the "primary sound source". In demo, the two angles drove a pair of servos steering a laser pointer. Followed the loudest object around the room :) IME, finding a way to communicate the information to the user is often non-intuitive. That is to say, once a device has located birds in trees, how would you like it to inform you?
- onion2k 5y agoMy first thought is for something like an arrow visible through the viewfinder, a bit like a damage indicator in an FPS game like Halo.
- loxias 5y agoHm. Doable. I think the hard part for that then might be getting the real time information about the position and orientation of the viewfinder in high enough resolution. Keep in mind "the black box" can output the position in 3 space (x,y,z, measured in mm) of coherent sound sources, but to know where those are relative to the camera, so that once can draw a little arrow, can be hard. I'd like to try hooking it up to a VR/AR headset, since I imagine those already handle the task of knowing precisely where my head is and where it's looking.
- zlsa 5y agoI think this might be possible with a phone that has AR support - you'd scan a QR code on the sound camera to capture its position relative to the world, then the phone could display a 3D view through the camera of where the sound source(s) are.
- loxias 5y agoOh that's interesting! Is your thinking something like: 1. mount the array on a tripod somewhere in the frame of the camera 2. the array is covered with an assortment of fiducials, 3. software uses the known intrinsics and extrinsics of the camera to figure out the array position relative to the camera 4. do the obvious thing with chaining transforms until you get the sound source position relative to the camera If so, I think that would work, but would be a lot of coding to do all that CV... > phone that has AR support I take it cell phones now do much of this work?
- transistor-man 5y agoAwesome work! How computationally intensive is Acoular / how complex would doing this from a live feed instead of recorded files be? Thanks for posting your project.
- loxias 5y agoI'm not familiar with Acoular*, but the math involved in computing the sound coherence function over a large space is quite involved! In my implementation, there are multiple stages using a dataflow approach with lots of compile time optimization. In 2011 I could image a roughly 2m^3 space using 8 microphones at ~10fps in real time on 3 desktop computers, 2015 I was able to do 12mics, 3m^3 space, on 2 laptops, but that involved a LOT of custom numeric programming to shave cycles. If I had access, I'd love to see what could be done given a well tuned implementation and modern GPUs. An efficient scatter gather OP (like what AVX3 has) would increase performance by an order of magnitude. *OK, I've skimmed Acoular.
- deleted 5y ago[deleted]
- xattt 5y agoCoupled with a VR headset, this would recreate the sound goggles that were featured in the halloween Magic School Bus special 30 or so years ago.
- jack_pp 5y agoIs this sensitive enough to find flying insects in a room?
- mshockwave 5y agovisualize that would be super cool
- davidb_ 5y agoI think with some filtering it definitely could be.
- dghughes 5y agoThey usually find me.
- loxias 5y agoNot speaking to OP's device, but yes, I was able to track a loud fly buzzing through a room in real-time. ~cm accuracy, but that can be improved on.
- ekingr 5y agoIf so, it’d definitely be worth coupling it with lasers to kill mosquitoes!
- davidb_ 5y agoCombining this with Motion Amplification/Video Magnification [1] could result in some very interesting visuals and applications for factory equipment. [1] https://www.youtube.com/watch?v=rEoc0YoALt0 https://www.youtube.com/watch?v=rEoc0YoALt0 Explainer Youtube video about Motion Amplification
- elfchief 5y agoI don't suppose you have any idea if there are publicly available motion amplification tools, yet?
- johndough 5y agoThe authors of the predecessor method released some of their code: https://people.csail.mit.edu/mrub/vidmag/#code https://people.csail.mit.edu/mrub/vidmag/#code
- noodlesUK 5y agoA starting point for the MIT research in question can be found here https://people.csail.mit.edu/mrub/vidmag/ https://people.csail.mit.edu/mrub/vidmag/
- Wistar 5y agoThat is a great video.
- epmaybe 5y agoMotion and color amplification from wu et Al are underused in my opinion. Maybe because under patent?
- johndough 5y agoPatents will expire starting from 2035 up to 2040 depending on the method used.
- 5faulker 5y agoThus another surveillance tool is born.
- Someone1234 5y agoStupid idea(?): Back-project onto some sunglasses (or corners/edge for behind), and give deaf people some basic level of sound-based situational awareness. Combine with some voice -> text tech, and you could have something pretty interesting.
- ampdepolymerase 5y agoThere is a long trail of dead startups attempting this. But don't let this dissuade you, please do a Show HN when you launch.
- marginalia_nu 5y agoProbably requires the technology to reach some tipping point. It was the same with VR and motion tracking. We've been able to do those things for nearly half a century, but it hasn't been anywhere near commercially viable until recently.
- fragmede 5y agoThe tipping point is the availability of quality, affordable AR glasses. Until recently, AR has been too niche, so even if the acoustic camera technology is fine, the company also has to build AR glasses to go with the other part of the system. Whenever Apple comes out with AR glasses, then writing an acoustic camera app is almost trivial in comparison to having to also having to design some AR glasses. Not having to design the glasses makes acoustic camera technology overlay software way more commercially viable.
- vdqtp3 5y ago> availability of quality, affordable AR glasses And since Intel, Google, Facebook etc keep buying startups that produce cool things and preventing them from producing more cool things (North Focals being the most recent I'm aware of) it's gonna be a while
- 5y ago
- londons_explore 5y agoI'd like to see this done with a single microphone and a moving 'sound mirror' like a fan. The fan blades should cause doppler shift and changing amplitude that varies based on the location of the sound. I suspect that after just a few seconds, this would give better information than an array of 16 microphones.
- jcims 5y agoThat would work for amplitude-based location but this is using phase correlation to find time of flight difference to each microphone. With a fan idea you would get a lot of phase drop outs and smearing that I think would make that difficult. Not to say wouldn’t work, you would get results, but they will be based in a different strategy.
- londons_explore 5y agoBut the phase smearing is useful information... Sure, the maths is complex... But there is only one source soundwave and location which causes a given smearing. The challenge is to find it...
- thatsadude 5y agoNot a single microphone but there is acoustic vector sensor which can also give you the sound's direction. Very expensive though, several Israel companies use them for detecting gunshot's direction. https://www.ese.wustl.edu/~nehorai/research/avs/avssensor.htm.sedbak https://www.ese.wustl.edu/~nehorai/research/avs/avssensor.ht...
- loxias 5y agoDo you know anything about these "acoustic vector sensors"? When I first saw a popular science article about them, I got excited about incorporating them into an array, but couldn't find any technical details, just a lot of what looked like vaporware. Is it anything more than three orthogonal pressure sensors? (aka.... 3 microphones?)
- 6gvONxR4sf7o 5y agoAm I understanding it correctly that this is not using anything analogous to a lens? How does this not need a lens when optical cameras need them?
- thatsadude 5y agoIn fact, you can use some lens such as parabolic reflector but it will make the problem very complicated to solve.
- seiferteric 5y agoAudio is low enough frequency that you can process the signal directly. The time delay/phase information between each mic allows you to know which direction the sound is coming from. This is essentially the opposite of beam forming. Theoretically you could do it with visible light and not need a lens if you had a computer and sensors that could operate fast enough. But optical sensors typically only tell you the amplitude of light and not phase for example. Edit: To clarify, the "opposite" of beam forming means using processing you can choose which direction you want to listen at any one time, like a beam. Then you can scan the beam across x,y and make an image.
- 6gvONxR4sf7o 5y agoThanks!
- _Microft 5y agoFun fact: we manage to record amplitude and phase of radiowaves, though. That allows us to record them at different points on Earth, ship the recorded data to a datacenter and computationally merge them to get a planet-wide virtual telescope dish with a much better angular resolution that a single telescope dish ever could have.
- ddalex 5y agowe don't record phase, there is no way to recover the phase from a single signal what we do we make sure that all receivers are synchronized, i.e take samples at the exact same time then you can correlate the signal received between dishes (which will arrive at different times due to delays in propagation), and find out the time difference of the signal which then points out to signal origin (beam forming) - this is how phased radar works once you align the signals you can use the minute differences in the signals to compute a synthetic aperture, i.e improving the angular resolution
- awinter-py 5y agofinally hardware to nail the guy who leans on his car horn outside my place
- achn 5y agoIt would be interesting to see how well this could detect non incident sounds - for instance detecting reflective/resonant hotspots in an audio mixing/recording room.
- kabla 5y agoWorks very well for that! At least on high enough frequencies. Source: have done something similar.
- loxias 5y agoI found that quite hard! Curious if there's anything public about your approach :) With high enough frequencies I can see reflections, but not at any distance, and the sound source has to be loud. Of course, I'm relying on line of sight, and perfect reflection. Any bumps in the wall would add some phase error I think. If I ever get a chance to work on the problem again I'd love to see if anything interesting can be done with multipath.
- maxcan 5y agoIs it possible to tune this to specific frequencies to detect mosquitos? Their audio signal is pretty weak but its also a very specific frequency. This would definitely help in the hunting and killing of the little bastards.
- wcfields 5y agoFollowing Someone1234's comment[1] Stupid idea(?): Attach a mid-power laser to zap bugs, could even be a DIY project with an arduino and an PTZ mount. [1] https://news.ycombinator.com/item?id=29015202 https://news.ycombinator.com/item?id=29015202
- mLuby 5y agoJust be careful not to blind yourself. Even a narrow flashlight cone could be enough for you to find and kill the bug by hand. Link to a previous post about this mosquito turret concept: https://news.ycombinator.com/item?id=27552516 https://news.ycombinator.com/item?id=27552516
- maxcan 5y agoExactly.. the laser introduces all kinds of safety issues. Simply shining a light and pointing me in the right direction vastly increases my lethality, especially combined with one of those fly swatter electric rackets. My cat would serve a similar role for larger insects. Her eyes would track them for me so I could locate and destroy them. Unfortunately she either cannot, or more likely chooses not to, track mosquitos.
- dylan604 5y agoNo need for the hi-tech equipment for finding mosquitoes. Just take me to the spot you're looking, and they will find me.
- potatoman22 5y agoIf the mosquito frequency is less than half of the sampling rate of these mics, then yes. Very basically, these algorithms work by looking at the delay between each mic picking up a certain frequency, and calculating the direction of the sound wave from that.
- supermatt 5y agoAre there any inexpensive microphone arrays? I was interested in making my own alexa-like device, but it seems mic arrays are sooooo expensive - more than the cost of an alexa device for the least expensive one i can find :/
- ghostly_s 5y agoThe post links to an inexpensive array at the end. I don't really get why the 16-mic one he used is so expensive, those smd mics can't be more than $1 or so each...
- supermatt 5y agoYeah, the respeaker ones i have looked at - but at $25+ they still feel very expensive to me. Are they that complicated/expensive to make?
- qazxcvbnmlp 5y agoIt’s probably a niche enough product that the engineering / marketing work that goes into it is a higher fraction of the cost than the raw components.
- atoav 5y agoThe mics are probably cheap but 16x ADC in decent quality with decent power supply and low time offset between channels? How much is a 16chn audio interface with 16 mic preamps these days?
- awinter-py 5y agofwiw they did this in world war I with microphone arrays + seismometer tape (picture of tape on p5) https://acousticstoday.org/wp-content/uploads/2020/06/Battlefield-Acoustics-in-the-First-World-War-Artillery-Location-Richard-Daniel-Costley-Jr..pdf https://acousticstoday.org/wp-content/uploads/2020/06/Battle...
- CamperBob2 5y agoAwesome article. Deserves a spot on the front page by itself! Makes me want to learn a lot more about W. L. Bragg's physics exploits, along with those of his father.
- geokon 5y agoActual information on how it's done is here: http://www.acoular.org/literature/index.html http://www.acoular.org/literature/index.html
- garyfirestorm 5y agowow this is amazing. i directly work in automotive and we use super expensive stuff which does exactly this (for 500k) lol
- loxias 5y ago> $500k Are you kidding me???! It costs so... so... so... much less. I thought automotive might be a good application, considering all the doors opened by using more DSP tricks layered in addition to source localization. (I can localize coherent sound patterns s well as coherent sound) I would love to chat with you, happy to buy a coffee or beer for your time. My email's on my profile.
- dendrite9 5y agoSome of these devices for automotive are large enough to surround a car on 3-4 sides, with several hundreds of microphones and the associated cables and positioning arms. Depending on where the devices he mentioned are being used, there are things like mannequins with heads and models of how humans hear for identifying sources inside a car. Here's an article about a large installation at Porsche. https://www.azom.com/article.aspx?ArticleID=18378 https://www.azom.com/article.aspx?ArticleID=18378
- loxias 5y ago> there are things like mannequins with heads and models of how humans hear for identifying sources inside a car. HRTF stuff is fun, if that's what you're referring to! :) I've worked with some of that stuff before, including the stupidly overpriced mannequin heads. > Some of these devices for automotive are large enough to surround a car on 3-4 sides, with several hundreds of microphones and the associated cables and positioning arms. Depending on where the devices he mentioned are being used, there are things like mannequins with heads and models of how humans hear for identifying sources inside a car. Do you work in a field that would benefit from the same results, for a fraction of the cost? Or, if not, do you have any advice on how to find and talk to these mythical industries that could pay me? It looks like Porsche wanted to build their own, in house, but I'm hoping if it costs less than a tenth as much, maybe more people would want one.
- jcims 5y agoOne application for those that I think might be interesting is to record a scene and retain all of the raw audio. On playback, allow people to click on parts of the image and use beamforming to focus on that part of the audio. Does anyone know if the array used here supports timestamped samples and/or clock sync to support multiple arrays? Or is it a single 16-channel stream? Having done some very primitive dabbling with this stuff, the DSP programming is always the most interesting part to me. These folks are killing it with some really cool 3D scanning integration to the acoustic analysis https://youtube.com/user/gfaitechgmbh https://youtube.com/user/gfaitechgmbh
- loxias 5y ago>On playback, allow people to click on parts of the image and use beamforming to focus on that part of the audio. You can do that, but the gain isn't as pronounced as you'd like. A 12-16dB gain doesn't sound that dramatic. Now, combined with some other newfangled math, like neural source separation, you might be able to do something spectacular...
- rapjr9 5y agoCould you do the inverse and use Melodyne to isolate a particular sound, click on it, and then have the camera scan for and isolate its location? https://www.celemony.com/en/start https://www.celemony.com/en/start
- jcims 5y agoI was thinking the inverse and transmit beamformed noise cancellation and create the 'shut-up gun' lol
- acjohnson55 5y agoThat would be a lot of data. Instead of a few bytes of color data per pixel per frame, you'd need a thousands of samples per second per unit of spatial resolution. Another approach to this is the Ambisonics method of capturing the directional soundfield at a point. But you'd need to use a high degree multipole expansion to get resolution anywhere closer to video. https://en.m.wikipedia.org/wiki/Ambisonics https://en.m.wikipedia.org/wiki/Ambisonics
- loxias 5y agoI'm not the original author, however, ages ago I invented what's now being called "acoustic camera". (Specifically, the SOTA on the math side for precision, accuracy.) The resolution is fine enough that with COTS parts, I can record my signature simply by sketching it out with my fingernail on a table. Every few years I dust this off and play with it, wondering if there's some application or other way to "turn this into money" (an increasing concern in the coming months...<tiny>PLZ HIRE ME</>), but I'm not a "product guy". I'll answer some questions about the technology, but would really love to know if anyone here has advice on somehow using this achievement to pay rent. :)
- _trampeltier 5y agoIn industrial maintenace are such cameras in use to find air leaks in pneumatic. The model from Fluke is something like 20k USD. Because of this super high price, I don't know anybody who has a such model. So ask another company, if they are interested.
- paulirwin 5y agoNot saying it's exactly the same thing, as I don't think a video overlay was involved, but I know someone that got their PhD in this area in the 1970s and had a long career working for a U.S. military contractor doing this. The U.S. military has a significant interest in acoustic beamforming, both in the air and underwater, for obvious reasons.
- loxias 5y agoOh absolutely! Sorry, I was excited in my typing. It's not every day you see "your baby" on HN :) Everything I built was on the shoulders of giants, and lol, I didn't invent beamforming itself (of course). The problem I have trying to find the niche application is that there's not much (at least, that I could think of) where you can have high quality audio data, but where a simple camera wouldn't work. Also, full imaging (as opposed to just tracking the largest/loudest source via TDOA) is quite different math, stupendously more computationally intensive.
- 5y ago
- causi 5y agoI feel like there has to be a cheaper way to do this than a $275 acoustic array. It's only 16 elements. You couldn't do this with 16 cheap microphones?
- fragmede 5y agoHow cheap do you want them? $275/16 = $17 per microphone, or $15 per + $35 worth of additional materials to make it into an array. Or $10 mics + $115 of metal and plastic. $275 doesn't really seem exorbitant for niche hardware given than you need 16 decently high quality microphones. I eagerly await a ShowHN using $2 mics and cardboard instead!
- loxias 5y ago> given that you need 16 decently high quality microphones... But... you don't. :) The challenge I find is getting the data into the computer. That's what always costs the most. I've done it with 8x $1 mics and a used $100 sound card.
- cozzyd 5y agoCan't you buy a bunch of i2s microphones and use a cheap FPGA dev board with a USB interface?' I may just order a bunch of i2s microphones...
- loxias 5y agoCan one? Probably. Can I? Never done so before. Can you? :) I know enough EE to do simple things, about the amount you'd expect someone who's worked as a firmware engineer to know. The fact that you're saying "Can't you ... cheap..."? make me think this must be a viable path though. My hands shake too much for anything but the simplest soldering; is there a cheap FPGA board you'd recommend? And getting all the data into the computer... it could easily be ~70Mbps. (16 mics, 192/24) Making a custom USB class could be a mess, I wonder how hard it would be to just dump ethernet.
- IshKebab 5y agoFYI if you put your microphones in a random pattern you can reduce aliasing artefacts. It's basically the same as dithering / noise shaping.
- loxias 5y agoThis took me years to figure out. :) Even cooler is that you can put them in a random pattern AND have the system determine its own geometry w/o measurements.
- nyanpasu64 5y agoBased on my experience building corrscope, I feel this is the kind of project that will outgrow Python once you want to implement your own low-level algorithms, make it embeddable or shippable as an application, or parallelize it. I wonder what's the easiest way to port Python DSP code and UIs to a compiled/etc. language.
- jcun4128 5y agoThat's pretty cool as far as adding to a sensor fusion stack
- herval 5y agoI imagine this + AR glasses can become quite the lifesaver for deaf folks. Throw in some voice recognition and you can have real-life speech bubbles!
- modeless 5y agoThis is super cool. I was thinking about making a 4x4 mems mic array on pcb exactly like that one. I had no idea you could just buy one off the shelf these days. Has anyone put four together to make a 64 mic 3D acoustic camera?
- deleted 5y ago[deleted]
- amelius 5y agoI'd like to see a video with an acoustic mirror.
- pabs3 5y agoCould this work underwater? Would be interesting to have a acoustic tracking system for diver safety.