3 ms·

I definitely think it would be possible. I find it interesting to think about eyesight in the same way -- even though an image is projected onto our retinas, th

I definitely think it would be possible. I find it interesting to think about eyesight in the same way -- even though an image is projected onto our retinas, there's not a little homunculus looking at our retinas to see the image; it gets translated into electrical signals that our brain interprets. There seems to be a great amount of plasticity in the brain that lets us remap senses and view tools as extensions of our bodies. There has been some prior work on using depth cameras for navigation for the visually impaired. For example, a smart cane can detect objects beyond its reach and give haptic feedback. Microsoft Research did some work with putting the Kinect on a helmet and giving audio cues for navigation (http://research.microsoft.com/pubs/184208/VisionForTheBlind.pdf http://research.microsoft.com/pubs/184208/VisionForTheBlind....). What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. Maybe if someone was using it from birth, but for relatively fast training I doubt that approach. Other approaches do more interpretation -- Microsoft's work detected faces, walls, and floors, giving each a distinct sound for greater recognition.

bramd 11y ago
> What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. Have you tested this approach with blind users? I think building a picture of an environment is a good task to offload to the brain and a good skill to have/develop for blind people. > One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. I think this is a quite good approach, but agree it has a high learning curve. However, that high learning curve might reward the end-user with a system that is more flexible. By preprocessing the input and generating audio based on the detected patterns you limit the applicability of such a system. That being said, a generic system that gives "unfiltered" output and has additional cues you can set for example for fast approaching objects might be useful.
callil 11y ago
This is a great project - to address your last point, I dont think it would just be noise if the user habituated to it. Check out this project [0][1] that maps audible data to vibrations and seems to have successfully re-mapped sense data taking advantage of the elasticity of the human brain. Another similar project lets people "see" with their tongues [2] I definitely think using binaural (3d) audio could give users a much more complete and useful idea of what they are seeing so I wish you luck. Great Idea. [0] http://www.ted.com/talks/david_eagleman_can_we_create_new_senses_for_humans http://www.ted.com/talks/david_eagleman_can_we_create_new_se... [1] http://www.eagleman.com/research/sensory-substitution http://www.eagleman.com/research/sensory-substitution [2] http://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ http://www.scientificamerican.com/article/device-lets-blind-...

bramd 11y ago

> What I'm interested in is taking that sensory input and making it less immediate by giving it a memory -- letting it build up a picture of an environment rather than needing to point a device at something in order to know something about it. Have you tested this approach with blind users? I think building a picture of an environment is a good task to offload to the brain and a good skill to have/develop for blind people. > One big issue is figuring out how to sonify depth information so it's useful. One simple approach is to do a sort of sweep across each frame from left to right, letting each row of an image correspond to a certain pitch. I don't think this is a good approach, as it seems very vision-oriented and is likely to sound just like noise. I think this is a quite good approach, but agree it has a high learning curve. However, that high learning curve might reward the end-user with a system that is more flexible. By preprocessing the input and generating audio based on the detected patterns you limit the applicability of such a system. That being said, a generic system that gives "unfiltered" output and has additional cues you can set for example for fast approaching objects might be useful.

callil 11y ago

This is a great project - to address your last point, I dont think it would just be noise if the user habituated to it. Check out this project [0][1] that maps audible data to vibrations and seems to have successfully re-mapped sense data taking advantage of the elasticity of the human brain. Another similar project lets people "see" with their tongues [2] I definitely think using binaural (3d) audio could give users a much more complete and useful idea of what they are seeing so I wish you luck. Great Idea. [0] http://www.ted.com/talks/david_eagleman_can_we_create_new_senses_for_humans http://www.ted.com/talks/david_eagleman_can_we_create_new_se... [1] http://www.eagleman.com/research/sensory-substitution http://www.eagleman.com/research/sensory-substitution [2] http://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ http://www.scientificamerican.com/article/device-lets-blind-...