5 ms·

Yay, a sound-field camera! I've been wondering if such things are made. Should I take the existence of this project as an indication that they aren't? The use

Yay, a sound-field camera! I've been wondering if such things are made. Should I take the existence of this project as an indication that they aren't? The use case I imagine is finding what part of a machine is emitting a noise, much like how you would use a thermal camera for finding what part of it is running hot.

visakanv 10y ago
That's a very sensible use case. I was thinking more like, super-trippy live music visuals.
- joshu 10y ago
  http://www.gizmag.com/seesv-s205-sound-camera-locates-noises/27447/ http://www.gizmag.com/seesv-s205-sound-camera-locates-noises...
- Bromskloss 10y ago
  Yours is sensible too.
Artlav 10y ago
Not exactly unheard of, but quite rare. I found a company that made large arrays like that, but they seem to operate in near field, measuring only the sound volume. Imagine a wall of microphones next to a truck - you can tell which part of the wheel makes the sound. That sort of stuff. As for an actual "camera" that turns sound into an image - i've never heard of one.
- nxzero 10y ago
  Isn't this exactly what sonar does? SOund Navigation And Ranging: https://en.m.wikipedia.org/wiki/Sonar https://en.m.wikipedia.org/wiki/Sonar
  - Artlav 10y ago
    Not exactly. A sonar maps a surface using sound, while my camera looks at the sound itself. Think laser rangefinder on a scanner platform vs a camera.
    - deleted 10y ago
      [deleted]
    - benjamincburns 10y ago
      Have you seen light field photography [1]? It would seem that your array is quite similar in concept. I'd imagine that getting images similar to those produced by various SONAR imaging technologies is a similar sort of deconvolution problem. If you're looking for textbooks or scientific papers to review, I think the field of tomography is also ripe with methods to explore here. I'd challenge the notion however that this isn't 'just' a passive SONAR device. Keeping the camera analogy, active SONAR use spread spectrum pings, much like the white light of a camera's flash. Your camera lacks a flash function, and its lens is maybe not yet as in focus as it could be. If you did want to add a flash, you're likely a $0.02 piezo device and maybe $20 in other various parts (and heaps of numerical processing on a PC) away from having the ability to produce a nice 3D reconstruction of the physical scene. If you get direct data capture going, please do post some raw data online. I'm sure others would have heaps of fun playing around with it. 1: http://graphics.stanford.edu/projects/lightfield/ http://graphics.stanford.edu/projects/lightfield/
      - Artlav 10y ago
        Hm, why not - here is the raw data (after PDM decoding) for the 16x16 video from the article: http://orbides.org/etc/snd_16x16_raw.tar.gz http://orbides.org/etc/snd_16x16_raw.tar.gz (4 Mb) One file per cell, 16bit integer samples. Stored row by row, 128 slices per frame. Not sure where to post it, however - here it would be buried.
  - yathern 10y ago
    No. SONAR sends out an ultrasonic ping and times the response of its echo to get a distance. This project is similar in that it involves visualization from audio, but apart from that, it's quite difference. This is the ability to see a sound. To derive the positions of arbitrary sounds in a viewing plane, and render it like you would a picture or video.
- ProAm 10y ago
  Doesn't the army use something like this in the field to determine enemy sniper locations when they are taking pot shots?
  - sbierwagen 10y ago
    https://en.wikipedia.org/wiki/Gunfire_locator https://en.wikipedia.org/wiki/Gunfire_locator
- xlayn 10y ago
  Congrats, beautiful history and nice hacking. I hope you get a really big bag of money one of these days so you can keep doing this kind of things!!! kudos!
- femto 10y ago
  > As for an actual "camera" that turns sound into an image - i've never heard of one. I have. His name is Daniel Kish [1, 2]. Apparently his visual cortex has learned to process sound rather then light, so he literally sees, based on the sounds he is hearing. [1] http://www.abc.net.au/catalyst/stories/4441831.htm http://www.abc.net.au/catalyst/stories/4441831.htm [2] https://en.wikipedia.org/wiki/Daniel_Kish https://en.wikipedia.org/wiki/Daniel_Kish
  - duaneb 10y ago
    Unfortunately only he can appreciate his own experience.
    - femto 10y ago
      He gives a pretty decent explanation of what he is seeing in the Catalyst transcript. It's fascinating, as from his description it's a low resolution (due to the longer wavelength) of what you or I would see. Presumably he also sees things such as colours, but based on audio reflectivity rather than optical reflectivity? Your point about only him appreciating his own experience is interesting, as the same question can be asked of all of us [1]. In audio, maybe apples and oranges are the same colour and strawberries are different? It would be an interesting application of a sonic camera, to render the image so that equivalent audio colours are consistently mapped to the same light colours. [1] https://www.quora.com/Visual-Perception-Is-everyones-experience-of-color-the-same-1 https://www.quora.com/Visual-Perception-Is-everyones-experie...
- skrebbel 10y ago
  I commented it elsewhere in this thread, but I a company called Sorama has been doing sound camera stuff for quite a while. Afaicr it's based on the phd dissertation of their CEO Rick Scholte so that name might yield some articles you might like. Cool stuff!
aeling 10y ago
http://visisonics.com/apps/#nsi http://visisonics.com/apps/#nsi I'm unaffiliated, but saw one of their demos.
jo-m 10y ago
There is a Swiss startup called Distran (http://www.distran.ch/ http://www.distran.ch/) which develops ultrasound cameras. They also have a spherical version which looks really cool and allows 360deg sight, unfortunately I cannot find anything about that one on the website anymore.
- Jommi 10y ago
  A finnish one too: http://www.nlacoustics.com/ http://www.nlacoustics.com/ Can someone with more expertise explain the differences between the many companies in this field?
  - kabla 10y ago
    We all have a bit of a different way of approaching the problem and our target market differs somewhat. You can use this technology for a lot of different things, not just the obvious use cases. We're still looking at a lot of applications, so the clearer product differentiation is something that will form as time goes by. - Kai @ nlacoustics.com
gene-h 10y ago
Well as it happens there are special sonar cameras out there capable of providing real-time video images[0][1]. The use case here is being able for a diver or ROV to see in murky water [0]http://www.soundmetrics.com/image-gallery http://www.soundmetrics.com/image-gallery [1]http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS.html http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS....
- space_fountain 10y ago
  Not quite the same thing though. These systems are active. They have sound being produces most likely at a very specific frequency and then listened for so that depth information can be gathered. This instead images the sound that's already out there in the light. It's the difference between lidar and a camera
  - moheeb 10y ago
    Sounds to me more like the difference between using the flash or not.
    - eric_h 10y ago
      Sort of, except the "camera" can't make any use of "light" generated by anything other than the flash.
  - wbeaty 10y ago
    Nope, theirs is the sound-camera, detector array and sound-lens, but with an acoustic floodlight. I think they do use ranging though, so the 2D image is "sound camera" in one raster dimension, and "sonar range" in the other.
ChuckMcM 10y ago
No, not common, but as the OP noted the evolution of MEMs based microphones might make them a lot more common. As the author notes, they are more like holographic cameras than what many people think of as "regular" cameras. By converting the imaging into the FFT domain the resolution gets better "everywhere" the more sensors you add. (If you have ever cut up a hologram into smaller and smaller pieces you can see this in action, as the full image is still there, just lower resolution). It would also be interesting to illuminate a non-active source with sound beam. By using two, slightly out of phase, ultrasonic sources he might be able to reveal depth detail of the surface being imaged.
wtvanhest 10y ago
Visisonics had a device that did something like that.
wbeaty 10y ago
Search: acoustic camera. They sell phase-array microphones for mapping noise sources. Rather than a plane array, theirs is a sphere.
hjnilsson 10y ago
Squarehead makes exactly this product: http://www.sqhead.com http://www.sqhead.com It's used my military & events for drone tracking by sound, and concert venues for seeing where sound leaks out to the surrounding neighborhood. (Among other uses) Edit: Their example screenshot is even showing an engine block overlaid with acoustic output of the components: http://static1.1.sqspcdn.com/static/f/1431009/18268254/1337339694793/AcousticCam_screenshot_450px.png?token=8gWfc9vn92Pg2czimKMvquvmTvs%3D http://static1.1.sqspcdn.com/static/f/1431009/18268254/13373...

visakanv 10y ago

That's a very sensible use case. I was thinking more like, super-trippy live music visuals.

joshu 10y ago
http://www.gizmag.com/seesv-s205-sound-camera-locates-noises/27447/ http://www.gizmag.com/seesv-s205-sound-camera-locates-noises...
Bromskloss 10y ago
Yours is sensible too.

Artlav 10y ago

Not exactly unheard of, but quite rare. I found a company that made large arrays like that, but they seem to operate in near field, measuring only the sound volume. Imagine a wall of microphones next to a truck - you can tell which part of the wheel makes the sound. That sort of stuff. As for an actual "camera" that turns sound into an image - i've never heard of one.

nxzero 10y ago
Isn't this exactly what sonar does? SOund Navigation And Ranging: https://en.m.wikipedia.org/wiki/Sonar https://en.m.wikipedia.org/wiki/Sonar
- Artlav 10y ago
  Not exactly. A sonar maps a surface using sound, while my camera looks at the sound itself. Think laser rangefinder on a scanner platform vs a camera.
  - deleted 10y ago
    [deleted]
  - benjamincburns 10y ago
    Have you seen light field photography [1]? It would seem that your array is quite similar in concept. I'd imagine that getting images similar to those produced by various SONAR imaging technologies is a similar sort of deconvolution problem. If you're looking for textbooks or scientific papers to review, I think the field of tomography is also ripe with methods to explore here. I'd challenge the notion however that this isn't 'just' a passive SONAR device. Keeping the camera analogy, active SONAR use spread spectrum pings, much like the white light of a camera's flash. Your camera lacks a flash function, and its lens is maybe not yet as in focus as it could be. If you did want to add a flash, you're likely a $0.02 piezo device and maybe $20 in other various parts (and heaps of numerical processing on a PC) away from having the ability to produce a nice 3D reconstruction of the physical scene. If you get direct data capture going, please do post some raw data online. I'm sure others would have heaps of fun playing around with it. 1: http://graphics.stanford.edu/projects/lightfield/ http://graphics.stanford.edu/projects/lightfield/
    - Artlav 10y ago
      Hm, why not - here is the raw data (after PDM decoding) for the 16x16 video from the article: http://orbides.org/etc/snd_16x16_raw.tar.gz http://orbides.org/etc/snd_16x16_raw.tar.gz (4 Mb) One file per cell, 16bit integer samples. Stored row by row, 128 slices per frame. Not sure where to post it, however - here it would be buried.
- yathern 10y ago
  No. SONAR sends out an ultrasonic ping and times the response of its echo to get a distance. This project is similar in that it involves visualization from audio, but apart from that, it's quite difference. This is the ability to see a sound. To derive the positions of arbitrary sounds in a viewing plane, and render it like you would a picture or video.
ProAm 10y ago
Doesn't the army use something like this in the field to determine enemy sniper locations when they are taking pot shots?
- sbierwagen 10y ago
  https://en.wikipedia.org/wiki/Gunfire_locator https://en.wikipedia.org/wiki/Gunfire_locator
xlayn 10y ago
Congrats, beautiful history and nice hacking. I hope you get a really big bag of money one of these days so you can keep doing this kind of things!!! kudos!
femto 10y ago
> As for an actual "camera" that turns sound into an image - i've never heard of one. I have. His name is Daniel Kish [1, 2]. Apparently his visual cortex has learned to process sound rather then light, so he literally sees, based on the sounds he is hearing. [1] http://www.abc.net.au/catalyst/stories/4441831.htm http://www.abc.net.au/catalyst/stories/4441831.htm [2] https://en.wikipedia.org/wiki/Daniel_Kish https://en.wikipedia.org/wiki/Daniel_Kish
- duaneb 10y ago
  Unfortunately only he can appreciate his own experience.
  - femto 10y ago
    He gives a pretty decent explanation of what he is seeing in the Catalyst transcript. It's fascinating, as from his description it's a low resolution (due to the longer wavelength) of what you or I would see. Presumably he also sees things such as colours, but based on audio reflectivity rather than optical reflectivity? Your point about only him appreciating his own experience is interesting, as the same question can be asked of all of us [1]. In audio, maybe apples and oranges are the same colour and strawberries are different? It would be an interesting application of a sonic camera, to render the image so that equivalent audio colours are consistently mapped to the same light colours. [1] https://www.quora.com/Visual-Perception-Is-everyones-experience-of-color-the-same-1 https://www.quora.com/Visual-Perception-Is-everyones-experie...
skrebbel 10y ago
I commented it elsewhere in this thread, but I a company called Sorama has been doing sound camera stuff for quite a while. Afaicr it's based on the phd dissertation of their CEO Rick Scholte so that name might yield some articles you might like. Cool stuff!

aeling 10y ago

http://visisonics.com/apps/#nsi http://visisonics.com/apps/#nsi I'm unaffiliated, but saw one of their demos.

jo-m 10y ago

There is a Swiss startup called Distran (http://www.distran.ch/ http://www.distran.ch/) which develops ultrasound cameras. They also have a spherical version which looks really cool and allows 360deg sight, unfortunately I cannot find anything about that one on the website anymore.

Jommi 10y ago
A finnish one too: http://www.nlacoustics.com/ http://www.nlacoustics.com/ Can someone with more expertise explain the differences between the many companies in this field?
- kabla 10y ago
  We all have a bit of a different way of approaching the problem and our target market differs somewhat. You can use this technology for a lot of different things, not just the obvious use cases. We're still looking at a lot of applications, so the clearer product differentiation is something that will form as time goes by. - Kai @ nlacoustics.com

gene-h 10y ago

Well as it happens there are special sonar cameras out there capable of providing real-time video images[0][1]. The use case here is being able for a diver or ROV to see in murky water [0]http://www.soundmetrics.com/image-gallery http://www.soundmetrics.com/image-gallery [1]http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS.html http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS....

space_fountain 10y ago
Not quite the same thing though. These systems are active. They have sound being produces most likely at a very specific frequency and then listened for so that depth information can be gathered. This instead images the sound that's already out there in the light. It's the difference between lidar and a camera
- moheeb 10y ago
  Sounds to me more like the difference between using the flash or not.
  - eric_h 10y ago
    Sort of, except the "camera" can't make any use of "light" generated by anything other than the flash.
- wbeaty 10y ago
  Nope, theirs is the sound-camera, detector array and sound-lens, but with an acoustic floodlight. I think they do use ranging though, so the 2D image is "sound camera" in one raster dimension, and "sonar range" in the other.

ChuckMcM 10y ago

No, not common, but as the OP noted the evolution of MEMs based microphones might make them a lot more common. As the author notes, they are more like holographic cameras than what many people think of as "regular" cameras. By converting the imaging into the FFT domain the resolution gets better "everywhere" the more sensors you add. (If you have ever cut up a hologram into smaller and smaller pieces you can see this in action, as the full image is still there, just lower resolution). It would also be interesting to illuminate a non-active source with sound beam. By using two, slightly out of phase, ultrasonic sources he might be able to reveal depth detail of the surface being imaged.

wtvanhest 10y ago

Visisonics had a device that did something like that.

wbeaty 10y ago

Search: acoustic camera. They sell phase-array microphones for mapping noise sources. Rather than a plane array, theirs is a sphere.

hjnilsson 10y ago

Squarehead makes exactly this product: http://www.sqhead.com http://www.sqhead.com It's used my military & events for drone tracking by sound, and concert venues for seeing where sound leaks out to the surrounding neighborhood. (Among other uses) Edit: Their example screenshot is even showing an engine block overlaid with acoustic output of the components: http://static1.1.sqspcdn.com/static/f/1431009/18268254/1337339694793/AcousticCam_screenshot_450px.png?token=8gWfc9vn92Pg2czimKMvquvmTvs%3D http://static1.1.sqspcdn.com/static/f/1431009/18268254/13373...