3 ms·

I also thought this might be useful for people who have problems perceiving some frequencies (maybe all?), to just see the sound. I am not sure what is the corr

by avaku 6y ago

I also thought this might be useful for people who have problems perceiving some frequencies (maybe all?), to just see the sound. I am not sure what is the correct way to call people who have trouble hearing, but I am really wondering if this could help someone, and I'd like to help make it better. (edit: typos)

andai 6y ago
I had the same idea. I was looking at some spectrograms of a podcast in Audacity and I began to recognize different people's voices and frequently used words. So I realized that with enough effort it would be possible to learn to read in this way. I also like looking at spectrograms of birdsong. I had a setup where I would see the spectrogram of sound coming through my microphone in realtime. I would place it on the windowsill and a few times I would see very faint birdsong I hadn't noticed with my ears, and then I would "tune in" my ears to that frequency range and I would hear it. Thinking about it the ideal form factor for a realtime spectrogram for the hard of hearing (or the curious) would be something like Google Glass. (Originally thought of a smartwatch but that doesn't make much sense for an additional sense). There's a project that's the opposite of this where a colorblind man attached a sensor to his head that converts color into sine waves he can hear. Adam Montandon. Eventually this electronic sense became a natural part of his experience and he began to dream in "color".
- avaku 6y ago
  I am not sure it would be possible to visually recognise speech, but might help people with hearing difficulties to perceive a concert better or something. Funny think about birds! Yesterday, as I was preparing to release the app, birds were singing outside of my window (coronavirus lockdown, so quiet), and I noticed how fun to look at their songs on my app! I've spent literally 30 minutes looking at that :) I've posted a link to birdwatchers reddit :)
- avaku 6y ago
  On the topic of recognition though, I think it would be much easier to implement speech recognition using the core algorithm used here, rather than from FFT for example. I'm only at the beginning of trying to use the frequency detection algo for something else apart from visualisation... The difficulty is that the dat it generates is much more detailed for higher frequencies, so it's hard to adapt modern ML algorithm inputs to it, I need to think more...
- avaku 6y ago
  Interesting idea about transforming colour to sound! P.S. I am red-green colourblind too :) Although, I don't find it a problem.
nmfisher 6y ago
One of my favourite (electronic music) producers mentioned recently that he has been developing tinnitus. Normally this would affect his ability to hear certain frequencies, but he’s so accustomed to working with raw waveforms, he’s not too concerned. He knows exactly what the sound he wants should look like.
- avaku 6y ago
  Thanks for comment. I think this can be useful to music producers (in some way), but the market for plugins is so tough, basically hard to survive even for one developer on revenue from one plugin. I’ll be thinking about this.

by avaku 6y ago

andai 6y ago

I had the same idea. I was looking at some spectrograms of a podcast in Audacity and I began to recognize different people's voices and frequently used words. So I realized that with enough effort it would be possible to learn to read in this way. I also like looking at spectrograms of birdsong. I had a setup where I would see the spectrogram of sound coming through my microphone in realtime. I would place it on the windowsill and a few times I would see very faint birdsong I hadn't noticed with my ears, and then I would "tune in" my ears to that frequency range and I would hear it. Thinking about it the ideal form factor for a realtime spectrogram for the hard of hearing (or the curious) would be something like Google Glass. (Originally thought of a smartwatch but that doesn't make much sense for an additional sense). There's a project that's the opposite of this where a colorblind man attached a sensor to his head that converts color into sine waves he can hear. Adam Montandon. Eventually this electronic sense became a natural part of his experience and he began to dream in "color".

avaku 6y ago
I am not sure it would be possible to visually recognise speech, but might help people with hearing difficulties to perceive a concert better or something. Funny think about birds! Yesterday, as I was preparing to release the app, birds were singing outside of my window (coronavirus lockdown, so quiet), and I noticed how fun to look at their songs on my app! I've spent literally 30 minutes looking at that :) I've posted a link to birdwatchers reddit :)
avaku 6y ago
On the topic of recognition though, I think it would be much easier to implement speech recognition using the core algorithm used here, rather than from FFT for example. I'm only at the beginning of trying to use the frequency detection algo for something else apart from visualisation... The difficulty is that the dat it generates is much more detailed for higher frequencies, so it's hard to adapt modern ML algorithm inputs to it, I need to think more...
avaku 6y ago
Interesting idea about transforming colour to sound! P.S. I am red-green colourblind too :) Although, I don't find it a problem.

nmfisher 6y ago

One of my favourite (electronic music) producers mentioned recently that he has been developing tinnitus. Normally this would affect his ability to hear certain frequencies, but he’s so accustomed to working with raw waveforms, he’s not too concerned. He knows exactly what the sound he wants should look like.

avaku 6y ago
Thanks for comment. I think this can be useful to music producers (in some way), but the market for plugins is so tough, basically hard to survive even for one developer on revenue from one plugin. I’ll be thinking about this.