23 ms·

Phased Array Microphone (2023)

600 points by bglazer 2y ago

amelius 2y ago
Nice. It would be cool if this project could cleanly separate sources based on location. That would be a bit like a lightfield camera, where you can edit the focusing parameters after the image has already been taken, but now with sound. https://en.wikipedia.org/wiki/Light_field_camera https://en.wikipedia.org/wiki/Light_field_camera
- hinkley 2y ago
  I’m still sad these didn’t become a thing. I don’t need a 48MP camera phone. No seriously. I do not.
  - sitkack 2y ago
    If you can get a microlens array infront of that 48MP imager, you can have the light field camera you seek.
    - hinkley 2y ago
      Right. I don’t need a 48MP image but a 48MP sensor behind a depth of field image would be pretty useful. The thing about people taking pictures is they interrupt the flow of a live experience. My ex specifically bought a digital camera with a quiet boot up (low noise focus motor) so she could get shots of our friends before they knew she was taking pictures. Being able to take the shot and work on focal length later would be less disruptive.
      - sitkack 2y ago
        You can pickup the Lytro cameras for cheap on ebay. It is a fun piece of tech to play with. There really should be a microlens array for modern cameras since the pixel counts are so high, and with AI able to do all these crazy convolutions, it could make sense out of even garbage optics. https://www.edmundoptics.com/f/microlens-arrays/13812/ https://www.edmundoptics.com/f/microlens-arrays/13812/ http://lightfield-forum.com/2012/05/how-to-make-your-own-microlense-array-diy-lightfield-camera/ http://lightfield-forum.com/2012/05/how-to-make-your-own-mic...
- miloignis 2y ago
  I believe it can, there's a demo under the "Directional Audio" section, unless I misunderstand you.
dllu 2y ago
I once did a project to do multilateration of bats (the flying mammal) using an array of 4 microphones arranged in a big Y shape on the ground. Using the time difference of arrival at the four microphones, we could find the positions of each bat that flew over the array, as well as identify the species. It was used for an environmental study to determine the impact of installing wind turbines. Fun times.
- ryandvm 2y ago
  Honestly, that sounds like amazing work. I wish I could afford to get out of enterprise software engineering and just do academic software development like that.
- jessetemp 2y ago
  What were the results of your study? I’ve heard that bat lungs are so sensitive that when they fly across the pressure differential of large turbines their capillaries basically explode
  - dllu 2y ago
    Yes basically. Bird lungs are relatively rigid, open at both ends like a tube, and have a one-way flow of air, so they are less prone to pressure-related injuries. Bat lungs are mammalian lungs that expand and contract as they breathe just like us, so they are particularly vulnerable to barotrauma near wind turbines. After writing a bunch of MATLAB code to find the bats, I handed it off and haven't heard back about whether they actually built the wind turbines or not.
- mywacaday 2y ago
  I would love to do something like that to track the bats in my garden, how feasible would it be for an amateur to do as a personal project? Any good references on where to start.
- FredPret 2y ago
  I had no idea they were mammals until this comment. I thought they were furry birds!
crote 2y ago
I'm a bit surprised by those long "arm" PCBs. They are already doing calibration to account for some relatively large offsets: why not place each sensor on its own PCB, mount them to some carrier structure, and let calibration deal with the rest?
- elictronic 2y ago
  Pcb manufacturing is cheap. I put 20 parts 1.5 inch by 24 inch into pcbway and ended up with final delivered cost of 240 dollars. Not having to deal with wiring that many individual boards and all days of headaches tracking down issues is well worth it in my book.
  - crote 2y ago
    Huh, you're right. I expected 24-inch-long PCBs to be quite a bit more expensive, but even 4-layer boards at those sizes are still available at discount prices. I guess such thin boards could be used to fill in edges of mixed-order panels? It does make me wonder why they say "the array" was $700. Maybe assembly was extremely expensive It doesn't seem they weren't really able to benefit from it all that much, though: half of them arrived defective, and they had to do quite a lot of debugging to fix them.
    - kindiana 2y ago
      (OP here) the $700 was for 50 arm boards and 5 hub boards, fully assembled and shipped including all the parts (enough for 2 full arrays, with some spares). $350 @ qty 2 is pretty good, considering just the microphones is ~$100 for each array! Unfortunately the assembly/DFM didn't work out well, but with some better design and foresight it should be much less work/wiring compared to wiring them manually.
brunosan 2y ago
Armchair comment. I would LOVE to be a grad student again and try to pair it with ultrasound speaker arrays, for medical applications. Essentially a super HIFU (High-Intensity Focused Ultrasound) with live feedback. https://en.wikipedia.org/wiki/Focused_ultrasound https://en.wikipedia.org/wiki/Focused_ultrasound
- brudgers 2y ago
  Then, why not be a grad student again?
  - 01100011 2y ago
    Maybe they want to afford dinner?
    - polishdude20 2y ago
      Hey saw your message a while back in a thread talking about continuous glucose meters and feeling tired and fatigued etc. Mind contacting me? I'd love to chat. My email is in my profile
    - brudgers 2y ago
      TANSTAAFL, but student loans too.
- always_swapping 2y ago
  I may be the FUS grad student you seek. Reach out via profile email if you want to chat. Cheers!
- etrautmann 2y ago
  Medical applications would presumably require contact coupling and not through air?
- zipy124 2y ago
  I do my PhD in in-air ultrasound with phased arrays and talk to the medical guys at conferences/labs that we talk to and it's soooo much harder in solids/liquids. The frequency is significantly higher, think 1-10MHz instead of like 40khz, so any normal electronics are out the window.
proee 2y ago
What is the most practical application for this technology? Could you use it to pinpoint sounds coming from a car like a squeak?
- Salmonfisher11 2y ago
  A similar technique is very popular in industrial automation to spot leaks in compressed air pipes and their connections from far away. These leaks are extremely loud in the ultrasonic range. It's overlayed with a camera picture. That's ultra expensive gear.
  - pletnes 2y ago
    I’ll guess «ultra expensive» partly due to very low volumes made for a customer that can afford a lot.
- spankalee 2y ago
  I've always wanted this for videoconferencing room. A microphone array around the screen should be able to dynamically focus on the active talkers and cancel out background noise and echos to get much better sound quality that the muddy crap we usually get. If there were a speaker array around the screens too, you might be able to localize the audio for each person so that it seems like the sound is coming from where their head is on the screen.
  - icegreentea2 2y ago
    Shure sells a variety of array microphones (and the software) that handles similar things. I've never used one, but heh. https://www.shure.com/en-US/products/microphones/mxa710 https://www.shure.com/en-US/products/microphones/mxa710 https://www.shure.com/en-US/products/microphones/mxa920 https://www.shure.com/en-US/products/microphones/mxa920
  - Salmonfisher11 2y ago
    Beamforming is standard in modern conference room gear. It's being used for making a video focus on the active speaker and optimizing his audio. Have a look at the "Meeting Owl" for example. It works great up to a limit (around 5m) then you will need additional microphones closer to the speaker.
deleted 2y ago
[deleted]
cushychicken 2y ago
This is more or less the same principle of how Amazon Echo devices work, but on steroids. Very neat. I would be surprised if you aren’t seeing some diminished marginal returns from all those extra mics, but I guess you’re trying to capture azimuth deltas that Echo devices don’t really care about.
Salmonfisher11 2y ago
If somebody wants to play around with Zynq 7010's - have a look at the EBAZ4205 board. They can be bought from Aliexpress (20-30€). These are former Bitcoin Mining controllers. Some people reverse engineered the entire thing. It can be found in GitHub. And there's an adapter plate available for getting to the GPIOs. For a less complex entry there are also Chinese FPGAs ("Sipeed" boards which use a GoWin FPGA. They are quite capable and the IDE is free.
- telgareith 2y ago
  Xilinx tool chain is also no-cost.
  - scottapotamas 2y ago
    For some/smaller parts. Once you start going higher than Artix or the token Kintex parts you need to pay up.
frankus 2y ago
"As part of the calibration, the speed of sound is also a parameter which is optimized to obtain the best model of the system, which allows this whole procedure to act as a ridiculously overengineered thermometer." Reminds me of the electronics adage: "all sensors are temperature sensors, some measure other things as well."
- Marthinwurer 2y ago
  I love these kind of inadvertent measurements. One of my favorite examples is that a sufficiently accurate IMU can get you relatively accurate longitude measurements from the Coriolis effect.
  - emptiestplace 2y ago
    https://en.m.wikipedia.org/wiki/Inertial_measurement_unit https://en.m.wikipedia.org/wiki/Inertial_measurement_unit
  - adolph 2y ago
    Slight correction, latitude, not longitude. The earth’s surface closer to the poles has less distance to travel for any rotation than the surface closer to the equator. As a result the inertial navigation systems of long distance systems must be adjusted. Iirc, this is also the case for artillery firing computations. https://www.oxts.com/blog/going-round-circles-earth-rotation-inertial-navigation/ https://www.oxts.com/blog/going-round-circles-earth-rotation... https://www.britannica.com/science/latitude https://www.britannica.com/science/latitude
    - billyjmc 2y ago
      Coriolis corrections are thrown into sniper ballistic calculations, too. Not a huge effect in most conditions, but not zero, and there have been a lot of long shots in the past two decades.
      - psunavy03 2y ago
gravypod 2y ago
I was just doing research and landed on this exact page last night! I was wondering if anyone knows how someone could mic a room and record audio from only a specific area. For my use case I want to record a couch so I can watch TV with my friends online and remove their speech + show noise from the audio. Setting up some array of mics and using them for beam steering would probably work but there's not a lot of examples I could find on GitHub with code that works in real time.
- imbusy111 2y ago
  From the article "The simplest method of beamforming is delay-and-sum (DAS)". Measure distance from a point (couch) to each microphone, delay the signal in time domain by the time the sound takes to travel from point (couch) to microphone, and add up the signals. Pretty trivial. Basically you want the microphones receive the couch signal at the same time, even though they are different distances away. Make sure there is enough variation in microphone distances for this method to be effective.
- crazygringo 2y ago
  Loud show noise and your online friends' nearby audio is going to be reflected around the room as well as off of your bodies. What you want isn't microphone or beamforming tech, it's echo cancellation the same as every videoconferencing software uses. You just need to feed the show audio and friend audio in, and apply echo cancellation to each.
- aspenmayer 2y ago
  You might look into OBS and/or VoiceMeeter to see how streamers selectively route audio while livestreaming/recording video/audio streams. https://obsproject.com/ https://obsproject.com/ https://voicemeeter.com/ https://voicemeeter.com/
killjoywashere 2y ago
I wonder how well this would work with laser microphones on a pane of glass. Can you infer keystrokes with near infrared laser? That is, can you identify the heatmap of keystroke events to infer which keyboard they're using, then replay the tape to identify the strings of characters being typed? Can you localize the turning of pages with UV?
- Salmonfisher11 2y ago
  Didn't Israeli students show that you can recover audio from the vibrations of bulb filament with a fast photo diode? I'd test that with a CCD line sensor plus a wide aperture lens and reading it out with 8kHz. Then you have 128 audio pixels that can cover an entire city.
  - killjoywashere 2y ago
    Line of sight might be an issue there. I'm thinking more high-end clandestine eavesdropping. Fun fact: curtains are a pretty good defeat for laser microphones, but if the building is really old and made of solid stone, you can point at the rock instead!
    - 0cf8612b2e1e 2y ago
      The rock?! That’s incredible. I would have guessed it was too dense to pick up normal speaking volume. Then again, even the window glass vibration seems pretty magical to me.
- quantadev 2y ago
  This beamforming effect only works well when each sensor is getting a dramatic enough "different angle" on the signal that each one can use phase shifting to cancel out other noise, but with a laser there's not really any noise to cancel out (i mean you're just monitoring a vibrational spot on a window), and you also don't have a far enough "different angle" to shine from, if you're monitoring from one spot. However having multiple lasers from multiple different locations might be able to create an improved signal if all signals are averaged, but it wouldn't really be due to the phase shifting that's used in beamforming.
adamcharnock 2y ago
I would love to see this come to our various mobile devices in a nicely packaged form. I think part of what is holding back assistants, universal-translators, etc, is poor audio. Both reducing noise and being able to detect direction has a huge potential to help (I want to live-translate a group conversation around a dining table, for example). Firstly it would be great if my phone + headphones could combine the microphones to this end. But what if all phones in the immediate vicinity could cooperate to provide high quality directional audio? (Assuming privacy issues could be addressed).
- hatsunearu 2y ago
  It's already kind of implemented.
- abecedarius 2y ago
  For the hard of hearing like me the killer application would be live transcription in a noisy setting like a meetup or party, with source separation and grouping of speech from different speakers. Could be life-changing. (Android's Live Transcribe is very good now but doesn't even try to separate which words are from different speakers.)
  - adolph 2y ago
    * Automatic speech recognition (ASR) systems have progressed to the point where humans can interact with computing devices using speech. However, the distance between a device and the speaker will cause a loss in speech quality and therefore impact the effectiveness of ASR performance. As such, there is a greater need to have reliable voice capture for far-field speech recognition. The launch of Amazon Echo devices prompted the use of far-field ASR in the consumer electronics space, as it allows its users to interact with the device from several meters away by using microphone array processing techniques.* https://assets.amazon.science/da/c2/71f5f9fa49f585a4616e49d52749/sir-beam-selector-for-amazon-echo-devices-audio-front-end.pdf https://assets.amazon.science/da/c2/71f5f9fa49f585a4616e49d5...
- MVissers 2y ago
  I believe modern macbook pro’s already have multiple microphones that probably do some phase-array magic.
gizajob 2y ago
What about a soundfield microphone? Does about the same thing and the electronics can be done in the analogue domain.
- radiowave 2y ago
  At a rough guess from the audio samples, that array is producing an acceptance angle much narrower than any Soundfield mic is capable of. The noise source is only 45 degrees off-axis; I'd say any first-order microphone polar pattern (i.e. those a Soundfield mic is capable of) would capture more of the noise than is demonstrated here. Of course, you can improve on the rejection of off-axis sound by instead using a microphone with a more specialized polar patten (e.g. a shotgun mic), but then you lose the property of the pattern being steerable merely by signal processing. Lastly, such an array of dirt cheap pressure sensitive mic capsules with some clever computation behind them strikes me as the sort of thing you could throw Moore's law at, if you could justify the quantity. Whereas, Soundfield mics don't make much sense unless you're working with very precisely machined pressure-gradient capsules. Still, I get the feeling it'll be a while yet before this technique starts looking viable for audio production work, but it's very interesting.
hinkley 2y ago
Boeing ginned up a spherical version of these and used it on 787 prototypes to identify candidates for sound deadening material. Apparently in loud situations like airplanes, audio illusions can make a sound appear to come from a different spot than it really is. And when you have a weight budget for sound dampening material it matters if you hit the 80/20 sweet spot or not.
jcims 2y ago
Look up acoustic cameras on YouTube, there are some pretty impressive demonstrations of their capability. This is one of the companies I've been watching for a while, but it looks like FLIR and some other big names are getting into it: https://www.youtube.com/@gfaitechgmbh https://www.youtube.com/@gfaitechgmbh The one use case that is both creepy and interesting to me is recording a public space and then after the fact 'zooming in' to conversations between individuals.
- sipjca 2y ago
  I am very interested in how small these arrays can be. From talking with a friend with cochlear implants, I would assume this could help dramatically with the right signal processing to help him hear.
pftburger 2y ago
I wonder if there is a meaningful limit to number of listening zones. I’m imagining a 3d grid of virtual mics in a space, each with an AI behind it Heck, train the model on the raw sensor data and you get the most awesome conference mics
beambot 2y ago
Starting to see more & more of this with drones. In some cases, it's for military to detect drones nearby. In others, it's being used by drone delivery companies to detect other planes in the sky in a way that is cheaper, works in low-visibility, and doesn't use the same power requirements as radar.
dchichkov 2y ago
I'm curious, why haven't you used TDM I2S microphones for your array and used PDM? I understand that ICS-52000 is a relatively low cost ($2/100pcs) and there are even breakout boards available with 4 microphones, which can be chained to 8 or 16, like https://www.cdiweb.com/datasheets/notwired/ds-nw-aud-ics52000.pdf https://www.cdiweb.com/datasheets/notwired/ds-nw-aud-ics5200... Then you can take Jetson (or any I2S capable hardware with DSP or GPU on it) and chain 16 microphones per I2S port. It would seem a lot easier to assemble and probgam, if comared to FPGA setup.
- morcheeba 2y ago
  Not OP, but I looked in to this a few years ago. It was more expensive then, and only went to 20 kHz. Higher frequencies are helpful if you're listening for the hiss of leaking gas, or corona discharge of an electric arc. The Orin has 6xI2S ports internally, so that would work up to 16*6 = 96 microphones, which is a good number. But it looks like maybe only 3 are brought out & on different dev board connectors [1]? As with a lot of design, the devil is in the details. An FPGA could be easier to configure if you need more than 96 microphones. My notes: ICS-52000 $3.50, 20 kHz ICS-41350 $1.05, 40 kHz SPH0641LU4H-1 $1.45, 80 kHz+ [1] https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/Communications/AudioSetupAndDevelopment.html#board-interfaces https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide...
- tverbeure 2y ago
  I've considered making a phased array myself, but never got around to sending out the PCB. But here are two reasons by I2S is not the best option: * I2S requires 3 instead of the 2 pins of PDM. However, in the datasheet that you provided, it shows how you can daisy-chain microphones which is really cool (even if not standard I2S.) So that argument goes away. * PDM gives you access to way higher sample rates which in turns gives you more flexibility in choosing the delay for a delay-and-sum operation. For example, if the PDM clock is 2MHz, you could theoretically delay with a precision of 0.5us. In practice, you'll do that with lower precision, but with I2S, the clock will typically max out at 192kHz. * PDM microphones then do be cheaper.
  - 2y ago
djmips 2y ago
This has been on my to-do list since forever! Nice work Ben Wang.
cma 2y ago
Could this be combined with a smaller number of high quality mics and then machine learning or something else incorporating them to boost the overall quality while maintaining all the other features?
- markhahn 2y ago
  afaik, it really depends on the spatial structure of the audio field. think nyquist sampling rates, applied to space, and you can't apply a low-pass filter just because you don't care about higher-order signals. that means that for any given audio environment, there will be some "spatial spectrum" of signal, and you need to sample it densely enough to avoid aliasing.
jensenbox 2y ago
Why a radial pattern and not a grid?
- quantadev 2y ago
  Because the distance between the mics needs to be 1) large and 2) consistent. It would work with a grid but the mics near the middle would be "underutilized" (not maximally taken advantage of), and also in a grid the mathematics is horrendous, but with a circle it's simple.
- kindiana 2y ago
  (OP here) Primary reason is that you can make a big array with only 2 boards, a small board in the middle and a bunch of long boards around it. Radial pattern of linear arrays with exponential spacing should also be pretty close to optimal for the distribution of pairwise microphone distances to maximize the gain with a fixed number of microphones.
jsharf 2y ago
Wow, you can refocus the direction after the audio is recorded! This would be cool to mix with VR, so you could hear different conversations as you move around a virtual room
jojobas 2y ago
Using crab rave for demo is top notch.
holyknight 2y ago
damn, this is so cool
kindiana 2y ago
OP here, cool to see so many people are interested in this project! Happy to answer any questions (and I'll go around to reply to any questions already here)
peter_retief 2y ago
Is this what is used to find direction of sound or where sound is coming from. Like gunshots?
sfelicio 2y ago
Very very cool.

600 points by bglazer 2y ago

Nice. It would be cool if this project could cleanly separate sources based on location. That would be a bit like a lightfield camera, where you can edit the focusing parameters after the image has already been taken, but now with sound. https://en.wikipedia.org/wiki/Light_field_camera https://en.wikipedia.org/wiki/Light_field_camera

hinkley 2y ago
I’m still sad these didn’t become a thing. I don’t need a 48MP camera phone. No seriously. I do not.
- sitkack 2y ago
  If you can get a microlens array infront of that 48MP imager, you can have the light field camera you seek.
  - hinkley 2y ago
    Right. I don’t need a 48MP image but a 48MP sensor behind a depth of field image would be pretty useful. The thing about people taking pictures is they interrupt the flow of a live experience. My ex specifically bought a digital camera with a quiet boot up (low noise focus motor) so she could get shots of our friends before they knew she was taking pictures. Being able to take the shot and work on focal length later would be less disruptive.
    - sitkack 2y ago
      You can pickup the Lytro cameras for cheap on ebay. It is a fun piece of tech to play with. There really should be a microlens array for modern cameras since the pixel counts are so high, and with AI able to do all these crazy convolutions, it could make sense out of even garbage optics. https://www.edmundoptics.com/f/microlens-arrays/13812/ https://www.edmundoptics.com/f/microlens-arrays/13812/ http://lightfield-forum.com/2012/05/how-to-make-your-own-microlense-array-diy-lightfield-camera/ http://lightfield-forum.com/2012/05/how-to-make-your-own-mic...
miloignis 2y ago
I believe it can, there's a demo under the "Directional Audio" section, unless I misunderstand you.

dllu 2y ago

I once did a project to do multilateration of bats (the flying mammal) using an array of 4 microphones arranged in a big Y shape on the ground. Using the time difference of arrival at the four microphones, we could find the positions of each bat that flew over the array, as well as identify the species. It was used for an environmental study to determine the impact of installing wind turbines. Fun times.

ryandvm 2y ago
Honestly, that sounds like amazing work. I wish I could afford to get out of enterprise software engineering and just do academic software development like that.
jessetemp 2y ago
What were the results of your study? I’ve heard that bat lungs are so sensitive that when they fly across the pressure differential of large turbines their capillaries basically explode
- dllu 2y ago
  Yes basically. Bird lungs are relatively rigid, open at both ends like a tube, and have a one-way flow of air, so they are less prone to pressure-related injuries. Bat lungs are mammalian lungs that expand and contract as they breathe just like us, so they are particularly vulnerable to barotrauma near wind turbines. After writing a bunch of MATLAB code to find the bats, I handed it off and haven't heard back about whether they actually built the wind turbines or not.
mywacaday 2y ago
I would love to do something like that to track the bats in my garden, how feasible would it be for an amateur to do as a personal project? Any good references on where to start.
FredPret 2y ago
I had no idea they were mammals until this comment. I thought they were furry birds!

crote 2y ago

I'm a bit surprised by those long "arm" PCBs. They are already doing calibration to account for some relatively large offsets: why not place each sensor on its own PCB, mount them to some carrier structure, and let calibration deal with the rest?

elictronic 2y ago
Pcb manufacturing is cheap. I put 20 parts 1.5 inch by 24 inch into pcbway and ended up with final delivered cost of 240 dollars. Not having to deal with wiring that many individual boards and all days of headaches tracking down issues is well worth it in my book.
- crote 2y ago
  Huh, you're right. I expected 24-inch-long PCBs to be quite a bit more expensive, but even 4-layer boards at those sizes are still available at discount prices. I guess such thin boards could be used to fill in edges of mixed-order panels? It does make me wonder why they say "the array" was $700. Maybe assembly was extremely expensive It doesn't seem they weren't really able to benefit from it all that much, though: half of them arrived defective, and they had to do quite a lot of debugging to fix them.
  - kindiana 2y ago
    (OP here) the $700 was for 50 arm boards and 5 hub boards, fully assembled and shipped including all the parts (enough for 2 full arrays, with some spares). $350 @ qty 2 is pretty good, considering just the microphones is ~$100 for each array! Unfortunately the assembly/DFM didn't work out well, but with some better design and foresight it should be much less work/wiring compared to wiring them manually.

brunosan 2y ago

Armchair comment. I would LOVE to be a grad student again and try to pair it with ultrasound speaker arrays, for medical applications. Essentially a super HIFU (High-Intensity Focused Ultrasound) with live feedback. https://en.wikipedia.org/wiki/Focused_ultrasound https://en.wikipedia.org/wiki/Focused_ultrasound

brudgers 2y ago
Then, why not be a grad student again?
- 01100011 2y ago
  Maybe they want to afford dinner?
  - polishdude20 2y ago
    Hey saw your message a while back in a thread talking about continuous glucose meters and feeling tired and fatigued etc. Mind contacting me? I'd love to chat. My email is in my profile
  - brudgers 2y ago
    TANSTAAFL, but student loans too.
always_swapping 2y ago
I may be the FUS grad student you seek. Reach out via profile email if you want to chat. Cheers!
etrautmann 2y ago
Medical applications would presumably require contact coupling and not through air?
zipy124 2y ago
I do my PhD in in-air ultrasound with phased arrays and talk to the medical guys at conferences/labs that we talk to and it's soooo much harder in solids/liquids. The frequency is significantly higher, think 1-10MHz instead of like 40khz, so any normal electronics are out the window.

proee 2y ago

What is the most practical application for this technology? Could you use it to pinpoint sounds coming from a car like a squeak?

Salmonfisher11 2y ago
A similar technique is very popular in industrial automation to spot leaks in compressed air pipes and their connections from far away. These leaks are extremely loud in the ultrasonic range. It's overlayed with a camera picture. That's ultra expensive gear.
- pletnes 2y ago
  I’ll guess «ultra expensive» partly due to very low volumes made for a customer that can afford a lot.
spankalee 2y ago
I've always wanted this for videoconferencing room. A microphone array around the screen should be able to dynamically focus on the active talkers and cancel out background noise and echos to get much better sound quality that the muddy crap we usually get. If there were a speaker array around the screens too, you might be able to localize the audio for each person so that it seems like the sound is coming from where their head is on the screen.
- icegreentea2 2y ago
  Shure sells a variety of array microphones (and the software) that handles similar things. I've never used one, but heh. https://www.shure.com/en-US/products/microphones/mxa710 https://www.shure.com/en-US/products/microphones/mxa710 https://www.shure.com/en-US/products/microphones/mxa920 https://www.shure.com/en-US/products/microphones/mxa920
- Salmonfisher11 2y ago
  Beamforming is standard in modern conference room gear. It's being used for making a video focus on the active speaker and optimizing his audio. Have a look at the "Meeting Owl" for example. It works great up to a limit (around 5m) then you will need additional microphones closer to the speaker.

deleted 2y ago

[deleted]

cushychicken 2y ago

This is more or less the same principle of how Amazon Echo devices work, but on steroids. Very neat. I would be surprised if you aren’t seeing some diminished marginal returns from all those extra mics, but I guess you’re trying to capture azimuth deltas that Echo devices don’t really care about.

Salmonfisher11 2y ago

If somebody wants to play around with Zynq 7010's - have a look at the EBAZ4205 board. They can be bought from Aliexpress (20-30€). These are former Bitcoin Mining controllers. Some people reverse engineered the entire thing. It can be found in GitHub. And there's an adapter plate available for getting to the GPIOs. For a less complex entry there are also Chinese FPGAs ("Sipeed" boards which use a GoWin FPGA. They are quite capable and the IDE is free.

telgareith 2y ago
Xilinx tool chain is also no-cost.
- scottapotamas 2y ago
  For some/smaller parts. Once you start going higher than Artix or the token Kintex parts you need to pay up.

frankus 2y ago

"As part of the calibration, the speed of sound is also a parameter which is optimized to obtain the best model of the system, which allows this whole procedure to act as a ridiculously overengineered thermometer." Reminds me of the electronics adage: "all sensors are temperature sensors, some measure other things as well."

Marthinwurer 2y ago
I love these kind of inadvertent measurements. One of my favorite examples is that a sufficiently accurate IMU can get you relatively accurate longitude measurements from the Coriolis effect.
- emptiestplace 2y ago
  https://en.m.wikipedia.org/wiki/Inertial_measurement_unit https://en.m.wikipedia.org/wiki/Inertial_measurement_unit
- adolph 2y ago
  Slight correction, latitude, not longitude. The earth’s surface closer to the poles has less distance to travel for any rotation than the surface closer to the equator. As a result the inertial navigation systems of long distance systems must be adjusted. Iirc, this is also the case for artillery firing computations. https://www.oxts.com/blog/going-round-circles-earth-rotation-inertial-navigation/ https://www.oxts.com/blog/going-round-circles-earth-rotation... https://www.britannica.com/science/latitude https://www.britannica.com/science/latitude
  - billyjmc 2y ago
    Coriolis corrections are thrown into sniper ballistic calculations, too. Not a huge effect in most conditions, but not zero, and there have been a lot of long shots in the past two decades.
    - psunavy03 2y ago

gravypod 2y ago

I was just doing research and landed on this exact page last night! I was wondering if anyone knows how someone could mic a room and record audio from only a specific area. For my use case I want to record a couch so I can watch TV with my friends online and remove their speech + show noise from the audio. Setting up some array of mics and using them for beam steering would probably work but there's not a lot of examples I could find on GitHub with code that works in real time.

imbusy111 2y ago
From the article "The simplest method of beamforming is delay-and-sum (DAS)". Measure distance from a point (couch) to each microphone, delay the signal in time domain by the time the sound takes to travel from point (couch) to microphone, and add up the signals. Pretty trivial. Basically you want the microphones receive the couch signal at the same time, even though they are different distances away. Make sure there is enough variation in microphone distances for this method to be effective.
crazygringo 2y ago
Loud show noise and your online friends' nearby audio is going to be reflected around the room as well as off of your bodies. What you want isn't microphone or beamforming tech, it's echo cancellation the same as every videoconferencing software uses. You just need to feed the show audio and friend audio in, and apply echo cancellation to each.
aspenmayer 2y ago
You might look into OBS and/or VoiceMeeter to see how streamers selectively route audio while livestreaming/recording video/audio streams. https://obsproject.com/ https://obsproject.com/ https://voicemeeter.com/ https://voicemeeter.com/

killjoywashere 2y ago

I wonder how well this would work with laser microphones on a pane of glass. Can you infer keystrokes with near infrared laser? That is, can you identify the heatmap of keystroke events to infer which keyboard they're using, then replay the tape to identify the strings of characters being typed? Can you localize the turning of pages with UV?

Salmonfisher11 2y ago
Didn't Israeli students show that you can recover audio from the vibrations of bulb filament with a fast photo diode? I'd test that with a CCD line sensor plus a wide aperture lens and reading it out with 8kHz. Then you have 128 audio pixels that can cover an entire city.
- killjoywashere 2y ago
  Line of sight might be an issue there. I'm thinking more high-end clandestine eavesdropping. Fun fact: curtains are a pretty good defeat for laser microphones, but if the building is really old and made of solid stone, you can point at the rock instead!
  - 0cf8612b2e1e 2y ago
    The rock?! That’s incredible. I would have guessed it was too dense to pick up normal speaking volume. Then again, even the window glass vibration seems pretty magical to me.
quantadev 2y ago
This beamforming effect only works well when each sensor is getting a dramatic enough "different angle" on the signal that each one can use phase shifting to cancel out other noise, but with a laser there's not really any noise to cancel out (i mean you're just monitoring a vibrational spot on a window), and you also don't have a far enough "different angle" to shine from, if you're monitoring from one spot. However having multiple lasers from multiple different locations might be able to create an improved signal if all signals are averaged, but it wouldn't really be due to the phase shifting that's used in beamforming.

adamcharnock 2y ago

I would love to see this come to our various mobile devices in a nicely packaged form. I think part of what is holding back assistants, universal-translators, etc, is poor audio. Both reducing noise and being able to detect direction has a huge potential to help (I want to live-translate a group conversation around a dining table, for example). Firstly it would be great if my phone + headphones could combine the microphones to this end. But what if all phones in the immediate vicinity could cooperate to provide high quality directional audio? (Assuming privacy issues could be addressed).

hatsunearu 2y ago
It's already kind of implemented.
abecedarius 2y ago
For the hard of hearing like me the killer application would be live transcription in a noisy setting like a meetup or party, with source separation and grouping of speech from different speakers. Could be life-changing. (Android's Live Transcribe is very good now but doesn't even try to separate which words are from different speakers.)
- adolph 2y ago
  * Automatic speech recognition (ASR) systems have progressed to the point where humans can interact with computing devices using speech. However, the distance between a device and the speaker will cause a loss in speech quality and therefore impact the effectiveness of ASR performance. As such, there is a greater need to have reliable voice capture for far-field speech recognition. The launch of Amazon Echo devices prompted the use of far-field ASR in the consumer electronics space, as it allows its users to interact with the device from several meters away by using microphone array processing techniques.* https://assets.amazon.science/da/c2/71f5f9fa49f585a4616e49d52749/sir-beam-selector-for-amazon-echo-devices-audio-front-end.pdf https://assets.amazon.science/da/c2/71f5f9fa49f585a4616e49d5...
MVissers 2y ago
I believe modern macbook pro’s already have multiple microphones that probably do some phase-array magic.

gizajob 2y ago

What about a soundfield microphone? Does about the same thing and the electronics can be done in the analogue domain.

radiowave 2y ago
At a rough guess from the audio samples, that array is producing an acceptance angle much narrower than any Soundfield mic is capable of. The noise source is only 45 degrees off-axis; I'd say any first-order microphone polar pattern (i.e. those a Soundfield mic is capable of) would capture more of the noise than is demonstrated here. Of course, you can improve on the rejection of off-axis sound by instead using a microphone with a more specialized polar patten (e.g. a shotgun mic), but then you lose the property of the pattern being steerable merely by signal processing. Lastly, such an array of dirt cheap pressure sensitive mic capsules with some clever computation behind them strikes me as the sort of thing you could throw Moore's law at, if you could justify the quantity. Whereas, Soundfield mics don't make much sense unless you're working with very precisely machined pressure-gradient capsules. Still, I get the feeling it'll be a while yet before this technique starts looking viable for audio production work, but it's very interesting.

hinkley 2y ago

Boeing ginned up a spherical version of these and used it on 787 prototypes to identify candidates for sound deadening material. Apparently in loud situations like airplanes, audio illusions can make a sound appear to come from a different spot than it really is. And when you have a weight budget for sound dampening material it matters if you hit the 80/20 sweet spot or not.

jcims 2y ago

Look up acoustic cameras on YouTube, there are some pretty impressive demonstrations of their capability. This is one of the companies I've been watching for a while, but it looks like FLIR and some other big names are getting into it: https://www.youtube.com/@gfaitechgmbh https://www.youtube.com/@gfaitechgmbh The one use case that is both creepy and interesting to me is recording a public space and then after the fact 'zooming in' to conversations between individuals.

sipjca 2y ago
I am very interested in how small these arrays can be. From talking with a friend with cochlear implants, I would assume this could help dramatically with the right signal processing to help him hear.

pftburger 2y ago

I wonder if there is a meaningful limit to number of listening zones. I’m imagining a 3d grid of virtual mics in a space, each with an AI behind it Heck, train the model on the raw sensor data and you get the most awesome conference mics

beambot 2y ago

Starting to see more & more of this with drones. In some cases, it's for military to detect drones nearby. In others, it's being used by drone delivery companies to detect other planes in the sky in a way that is cheaper, works in low-visibility, and doesn't use the same power requirements as radar.

dchichkov 2y ago

I'm curious, why haven't you used TDM I2S microphones for your array and used PDM? I understand that ICS-52000 is a relatively low cost ($2/100pcs) and there are even breakout boards available with 4 microphones, which can be chained to 8 or 16, like https://www.cdiweb.com/datasheets/notwired/ds-nw-aud-ics52000.pdf https://www.cdiweb.com/datasheets/notwired/ds-nw-aud-ics5200... Then you can take Jetson (or any I2S capable hardware with DSP or GPU on it) and chain 16 microphones per I2S port. It would seem a lot easier to assemble and probgam, if comared to FPGA setup.

morcheeba 2y ago
Not OP, but I looked in to this a few years ago. It was more expensive then, and only went to 20 kHz. Higher frequencies are helpful if you're listening for the hiss of leaking gas, or corona discharge of an electric arc. The Orin has 6xI2S ports internally, so that would work up to 16*6 = 96 microphones, which is a good number. But it looks like maybe only 3 are brought out & on different dev board connectors [1]? As with a lot of design, the devil is in the details. An FPGA could be easier to configure if you need more than 96 microphones. My notes: ICS-52000 $3.50, 20 kHz ICS-41350 $1.05, 40 kHz SPH0641LU4H-1 $1.45, 80 kHz+ [1] https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/Communications/AudioSetupAndDevelopment.html#board-interfaces https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide...
tverbeure 2y ago
I've considered making a phased array myself, but never got around to sending out the PCB. But here are two reasons by I2S is not the best option: * I2S requires 3 instead of the 2 pins of PDM. However, in the datasheet that you provided, it shows how you can daisy-chain microphones which is really cool (even if not standard I2S.) So that argument goes away. * PDM gives you access to way higher sample rates which in turns gives you more flexibility in choosing the delay for a delay-and-sum operation. For example, if the PDM clock is 2MHz, you could theoretically delay with a precision of 0.5us. In practice, you'll do that with lower precision, but with I2S, the clock will typically max out at 192kHz. * PDM microphones then do be cheaper.
- 2y ago

djmips 2y ago

This has been on my to-do list since forever! Nice work Ben Wang.

cma 2y ago

Could this be combined with a smaller number of high quality mics and then machine learning or something else incorporating them to boost the overall quality while maintaining all the other features?

markhahn 2y ago
afaik, it really depends on the spatial structure of the audio field. think nyquist sampling rates, applied to space, and you can't apply a low-pass filter just because you don't care about higher-order signals. that means that for any given audio environment, there will be some "spatial spectrum" of signal, and you need to sample it densely enough to avoid aliasing.

jensenbox 2y ago

Why a radial pattern and not a grid?

quantadev 2y ago
Because the distance between the mics needs to be 1) large and 2) consistent. It would work with a grid but the mics near the middle would be "underutilized" (not maximally taken advantage of), and also in a grid the mathematics is horrendous, but with a circle it's simple.
kindiana 2y ago
(OP here) Primary reason is that you can make a big array with only 2 boards, a small board in the middle and a bunch of long boards around it. Radial pattern of linear arrays with exponential spacing should also be pretty close to optimal for the distribution of pairwise microphone distances to maximize the gain with a fixed number of microphones.

jsharf 2y ago

Wow, you can refocus the direction after the audio is recorded! This would be cool to mix with VR, so you could hear different conversations as you move around a virtual room

jojobas 2y ago

Using crab rave for demo is top notch.

holyknight 2y ago

damn, this is so cool

kindiana 2y ago

OP here, cool to see so many people are interested in this project! Happy to answer any questions (and I'll go around to reply to any questions already here)

peter_retief 2y ago

Is this what is used to find direction of sound or where sound is coming from. Like gunshots?

sfelicio 2y ago

Very very cool.