4 ms·

This is awesome, great job, I think you've just given humans echolocation. If someone was given a similar device at an early age that was semi-permanently atta

by dm2 11y ago

This is awesome, great job, I think you've just given humans echolocation. If someone was given a similar device at an early age that was semi-permanently attached to them, would their brain possibly be able to create a map of the room? There have been previous attempts but the Tango device didn't exist then so the hardware was bulky and usually required a backpack.

DanAndersen 11y ago
I definitely think it would be possible. I find it interesting to think about eyesight in the same way -- even though an image is projected onto our retinas, there's not a little homunculus looking at our retinas to see the image; it gets translated into electrical signals that our brain interprets. There seems to be a great amount of plasticity in the brain that lets us remap senses and view tools as extensions of our bodies. There has been some prior work on using depth cameras for navigation for the visually impaired. For example, a smart cane can detect objects beyond its reach and give haptic feedback. Microsoft Research did some work with putting the Kinect on a helmet and giving audio cues for navigation (http://research.microsoft.com/pubs/184208/VisionForTheBlind.pdf http://research.microsoft.com/pubs/184208/VisionForTheBlind....). What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. Maybe if someone was using it from birth, but for relatively fast training I doubt that approach. Other approaches do more interpretation -- Microsoft's work detected faces, walls, and floors, giving each a distinct sound for greater recognition.
- bramd 11y ago
  > What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. Have you tested this approach with blind users? I think building a picture of an environment is a good task to offload to the brain and a good skill to have/develop for blind people. > One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. I think this is a quite good approach, but agree it has a high learning curve. However, that high learning curve might reward the end-user with a system that is more flexible. By preprocessing the input and generating audio based on the detected patterns you limit the applicability of such a system. That being said, a generic system that gives "unfiltered" output and has additional cues you can set for example for fast approaching objects might be useful.
- callil 11y ago
  This is a great project - to address your last point, I dont think it would just be noise if the user habituated to it. Check out this project [0][1] that maps audible data to vibrations and seems to have successfully re-mapped sense data taking advantage of the elasticity of the human brain. Another similar project lets people "see" with their tongues [2] I definitely think using binaural (3d) audio could give users a much more complete and useful idea of what they are seeing so I wish you luck. Great Idea. [0] http://www.ted.com/talks/david_eagleman_can_we_create_new_senses_for_humans http://www.ted.com/talks/david_eagleman_can_we_create_new_se... [1] http://www.eagleman.com/research/sensory-substitution http://www.eagleman.com/research/sensory-substitution [2] http://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ http://www.scientificamerican.com/article/device-lets-blind-...
extra88 11y ago
Humans can already learn echolocation [1]. Still, there are many possibilities for machine-assisted perception/translation. I think the post correctly identifies finding good ways to aurally represent the information to be one of the challenges. [1] http://en.wikipedia.org/wiki/Daniel_Kish http://en.wikipedia.org/wiki/Daniel_Kish
- dm2 11y ago
  Somewhat, but our brains haven't had millions of years to develop the ability to "see" sound, like bats cat. A human generated "click" is much different than a computer generated series of sounds which represent an accurate scan of the objects in front of the user. Change in pitch representing changes in depth is much more able to be processed by the human brain verses trying to hear how the sound waves bounce off of objects. Plus, this will work in public while I'd guess that the human clicking noises require a quiet environment and have significant limitations. Imagine a blind person walking freely down the sidewalk and the device would not be making any sounds until there is a sign or building within 20 feet, with a scanning (or single point) tone that gets progressively louder. Effortless echolocation. The device also has several GPS, WiFi, gyroscope, and cellular geolocation abilities so it would know when the user has reached the end of the sidewalk, if outside. I remember this story from a few years ago: https://www.youtube.com/watch?v=YBv79LKfMt4 https://www.youtube.com/watch?v=YBv79LKfMt4

by dm2 11y ago

DanAndersen 11y ago

I definitely think it would be possible. I find it interesting to think about eyesight in the same way -- even though an image is projected onto our retinas, there's not a little homunculus looking at our retinas to see the image; it gets translated into electrical signals that our brain interprets. There seems to be a great amount of plasticity in the brain that lets us remap senses and view tools as extensions of our bodies. There has been some prior work on using depth cameras for navigation for the visually impaired. For example, a smart cane can detect objects beyond its reach and give haptic feedback. Microsoft Research did some work with putting the Kinect on a helmet and giving audio cues for navigation (http://research.microsoft.com/pubs/184208/VisionForTheBlind.pdf http://research.microsoft.com/pubs/184208/VisionForTheBlind....). What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. Maybe if someone was using it from birth, but for relatively fast training I doubt that approach. Other approaches do more interpretation -- Microsoft's work detected faces, walls, and floors, giving each a distinct sound for greater recognition.

bramd 11y ago
> What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. Have you tested this approach with blind users? I think building a picture of an environment is a good task to offload to the brain and a good skill to have/develop for blind people. > One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. I think this is a quite good approach, but agree it has a high learning curve. However, that high learning curve might reward the end-user with a system that is more flexible. By preprocessing the input and generating audio based on the detected patterns you limit the applicability of such a system. That being said, a generic system that gives "unfiltered" output and has additional cues you can set for example for fast approaching objects might be useful.
callil 11y ago
This is a great project - to address your last point, I dont think it would just be noise if the user habituated to it. Check out this project [0][1] that maps audible data to vibrations and seems to have successfully re-mapped sense data taking advantage of the elasticity of the human brain. Another similar project lets people "see" with their tongues [2] I definitely think using binaural (3d) audio could give users a much more complete and useful idea of what they are seeing so I wish you luck. Great Idea. [0] http://www.ted.com/talks/david_eagleman_can_we_create_new_senses_for_humans http://www.ted.com/talks/david_eagleman_can_we_create_new_se... [1] http://www.eagleman.com/research/sensory-substitution http://www.eagleman.com/research/sensory-substitution [2] http://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ http://www.scientificamerican.com/article/device-lets-blind-...

extra88 11y ago

Humans can already learn echolocation [1]. Still, there are many possibilities for machine-assisted perception/translation. I think the post correctly identifies finding good ways to aurally represent the information to be one of the challenges. [1] http://en.wikipedia.org/wiki/Daniel_Kish http://en.wikipedia.org/wiki/Daniel_Kish

dm2 11y ago
Somewhat, but our brains haven't had millions of years to develop the ability to "see" sound, like bats cat. A human generated "click" is much different than a computer generated series of sounds which represent an accurate scan of the objects in front of the user. Change in pitch representing changes in depth is much more able to be processed by the human brain verses trying to hear how the sound waves bounce off of objects. Plus, this will work in public while I'd guess that the human clicking noises require a quiet environment and have significant limitations. Imagine a blind person walking freely down the sidewalk and the device would not be making any sounds until there is a sign or building within 20 feet, with a scanning (or single point) tone that gets progressively louder. Effortless echolocation. The device also has several GPS, WiFi, gyroscope, and cellular geolocation abilities so it would know when the user has reached the end of the sidewalk, if outside. I remember this story from a few years ago: https://www.youtube.com/watch?v=YBv79LKfMt4 https://www.youtube.com/watch?v=YBv79LKfMt4