4 ms·

One application for those that I think might be interesting is to record a scene and retain all of the raw audio. On playback, allow people to click on parts of

by jcims 5y ago

One application for those that I think might be interesting is to record a scene and retain all of the raw audio. On playback, allow people to click on parts of the image and use beamforming to focus on that part of the audio. Does anyone know if the array used here supports timestamped samples and/or clock sync to support multiple arrays? Or is it a single 16-channel stream? Having done some very primitive dabbling with this stuff, the DSP programming is always the most interesting part to me. These folks are killing it with some really cool 3D scanning integration to the acoustic analysis https://youtube.com/user/gfaitechgmbh https://youtube.com/user/gfaitechgmbh

loxias 5y ago
>On playback, allow people to click on parts of the image and use beamforming to focus on that part of the audio. You can do that, but the gain isn't as pronounced as you'd like. A 12-16dB gain doesn't sound that dramatic. Now, combined with some other newfangled math, like neural source separation, you might be able to do something spectacular...
- rapjr9 5y ago
  Could you do the inverse and use Melodyne to isolate a particular sound, click on it, and then have the camera scan for and isolate its location? https://www.celemony.com/en/start https://www.celemony.com/en/start
  - jcims 5y ago
    I was thinking the inverse and transmit beamformed noise cancellation and create the 'shut-up gun' lol
acjohnson55 5y ago
That would be a lot of data. Instead of a few bytes of color data per pixel per frame, you'd need a thousands of samples per second per unit of spatial resolution. Another approach to this is the Ambisonics method of capturing the directional soundfield at a point. But you'd need to use a high degree multipole expansion to get resolution anywhere closer to video. https://en.m.wikipedia.org/wiki/Ambisonics https://en.m.wikipedia.org/wiki/Ambisonics
- Aachen 5y ago
  No, you need a few audio channels. If the algorithm can filter out what's where based on this 16 or even 4 microphone array, so can your client when provided with those sound tracks which are just a few kilobytes each. Probably all audio streams together are still less data than the video stream (given typical audio and video track quality combinations). You don't need a preprocessed stream for every point. Even if you want to keep the algorithm secret, you can have the client send the desired position to the server and do the magic there.
  - acjohnson55 5y ago
    I think you're mostly right. If you want to be able to preserve enough bandwidth for music, you're talking about kilobytes per second of audio, but yeah, you could just store the streams from each mic in the array, so multiply the bitrate by the array size. And it's probably possible to find ways to save bandwidth with joint encoding.

by jcims 5y ago

loxias 5y ago

>On playback, allow people to click on parts of the image and use beamforming to focus on that part of the audio. You can do that, but the gain isn't as pronounced as you'd like. A 12-16dB gain doesn't sound that dramatic. Now, combined with some other newfangled math, like neural source separation, you might be able to do something spectacular...

rapjr9 5y ago
Could you do the inverse and use Melodyne to isolate a particular sound, click on it, and then have the camera scan for and isolate its location? https://www.celemony.com/en/start https://www.celemony.com/en/start
- jcims 5y ago
  I was thinking the inverse and transmit beamformed noise cancellation and create the 'shut-up gun' lol

acjohnson55 5y ago

That would be a lot of data. Instead of a few bytes of color data per pixel per frame, you'd need a thousands of samples per second per unit of spatial resolution. Another approach to this is the Ambisonics method of capturing the directional soundfield at a point. But you'd need to use a high degree multipole expansion to get resolution anywhere closer to video. https://en.m.wikipedia.org/wiki/Ambisonics https://en.m.wikipedia.org/wiki/Ambisonics

Aachen 5y ago
No, you need a few audio channels. If the algorithm can filter out what's where based on this 16 or even 4 microphone array, so can your client when provided with those sound tracks which are just a few kilobytes each. Probably all audio streams together are still less data than the video stream (given typical audio and video track quality combinations). You don't need a preprocessed stream for every point. Even if you want to keep the algorithm secret, you can have the client send the desired position to the server and do the magic there.
- acjohnson55 5y ago
  I think you're mostly right. If you want to be able to preserve enough bandwidth for music, you're talking about kilobytes per second of audio, but yeah, you could just store the streams from each mic in the array, so multiply the bitrate by the array size. And it's probably possible to find ways to save bandwidth with joint encoding.