4 ms·

Hey. I've been working on an audio recognition tool that is remarkably similar to this one, and there are a few areas I strongly recommend exploring if you want

by cnity 6y ago

Hey. I've been working on an audio recognition tool that is remarkably similar to this one, and there are a few areas I strongly recommend exploring if you want to improve the classification performance. For waveform similarity, you may achieve better results by comparing the spectrogram (computed by taking the FFT of each "clack") instead of comparing the time domain signal. Next, the best performing sound classification algorithms (at least in music note classification, which is a very similar problem to yours) utilise non-negative matrix factorisation[0] to do the comparison against a bank of note templates. I recommend reading "Real-time Polyphonic Music Transcription with Non-negative Matrix Factorization and Beta-divergence" (Dessein et. al.) for a more comprehensive description of the algorithm[1]. I actually tried applying my algorithm to detect taps on different parts of my laptop for fun, so that I could send keyboard events when e.g. I tap the bottom right corner of my laptop, and it was quite effective. The approach is incredibly accurate for classification of short spectrograms. Feel free to reach out to me if you want to discuss a bit more. [0] https://en.wikipedia.org/wiki/Non-negative_matrix_factorization https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat... [1] https://www.researchgate.net/publication/220723421_Real-time_Polyphonic_Music_Transcription_with_Non-negative_Matrix_Factorization_and_Beta-divergence https://www.researchgate.net/publication/220723421_Real-time... Edit: Also, this is great work. I don't mean to denigrate it at all.

ggerganov 6y ago
Thank you for these suggestions! The idea for comparing sounds in the frequency domain was something that I was considering, but couldn't figure out a proper way to do it. I will definitely check the resources that you provided and see if I can utilize this approach in Keytap.
waffletower 6y ago
I don't recommend using DFT (FFT) analysis unless you can constrain N to small sizes (<= 256 at 44.1KHz sampling rate perhaps) to reduce time aliasing. Use of wavelets may be a better fit for this application.
- ggerganov 6y ago
  I have found that I need to use about 75-100 ms of audio around the keystroke-down peak when comparing waveforms. At the sample rate that I am using - 16kHz, I would probably need samples of size about 1024 - 2048. Not sure if this is feasible, just giving some additional info.
  - cnity 6y ago
    If you want to compute an FFT with fewer bins you can also use STFT and configure the bin count and hop size (essentially computing many FFTs of smaller regions of the signal and computing the average). A quick side note, 16kHz is quite a low sample rate for your use case, since you won't be able to capture frequencies over 8kHz (the Nyquist frequency), and key clicks at least intuitively I'd imagine to contain a lot of detail at higher frequencies. I have no data to back this up though!
    - ggerganov 6y ago
      Regarding the sampling rate: I switched down to 16kHz from 48kHz, because I found that with my approach there wasn't much of a difference in the key-matching performance. At the same time, I gained computational performance. At some point, I was looking at the spectrum of the key sounds and I think that almost the entire signal is in the lower few kHz. It might be just my keyboard - not sure. I still have the tool, so I can easily check if there is some high-frequency portion that I am missing.
      - cnity 6y ago
        I believe you, it was only a hunch. Not backed up by evidence! I've been way off about my intuition of frequencies, especially with non-harmonic sounds like key presses :)
    - waffletower 6y ago
      I agree, 44.1KHz is a commonly available sampling rate and could provide better temporal resolution regardless of time doamin and frequency domain analysis technique you decide to use.
randomcarbloke 6y ago
I've been working on something similar to both but also distinct.

by cnity 6y ago

ggerganov 6y ago

Thank you for these suggestions! The idea for comparing sounds in the frequency domain was something that I was considering, but couldn't figure out a proper way to do it. I will definitely check the resources that you provided and see if I can utilize this approach in Keytap.

waffletower 6y ago

I don't recommend using DFT (FFT) analysis unless you can constrain N to small sizes (<= 256 at 44.1KHz sampling rate perhaps) to reduce time aliasing. Use of wavelets may be a better fit for this application.

ggerganov 6y ago
I have found that I need to use about 75-100 ms of audio around the keystroke-down peak when comparing waveforms. At the sample rate that I am using - 16kHz, I would probably need samples of size about 1024 - 2048. Not sure if this is feasible, just giving some additional info.
- cnity 6y ago
  If you want to compute an FFT with fewer bins you can also use STFT and configure the bin count and hop size (essentially computing many FFTs of smaller regions of the signal and computing the average). A quick side note, 16kHz is quite a low sample rate for your use case, since you won't be able to capture frequencies over 8kHz (the Nyquist frequency), and key clicks at least intuitively I'd imagine to contain a lot of detail at higher frequencies. I have no data to back this up though!
  - ggerganov 6y ago
    Regarding the sampling rate: I switched down to 16kHz from 48kHz, because I found that with my approach there wasn't much of a difference in the key-matching performance. At the same time, I gained computational performance. At some point, I was looking at the spectrum of the key sounds and I think that almost the entire signal is in the lower few kHz. It might be just my keyboard - not sure. I still have the tool, so I can easily check if there is some high-frequency portion that I am missing.
    - cnity 6y ago
      I believe you, it was only a hunch. Not backed up by evidence! I've been way off about my intuition of frequencies, especially with non-harmonic sounds like key presses :)
  - waffletower 6y ago
    I agree, 44.1KHz is a commonly available sampling rate and could provide better temporal resolution regardless of time doamin and frequency domain analysis technique you decide to use.

randomcarbloke 6y ago

I've been working on something similar to both but also distinct.