3 ms·

Interesting. Figuring out the phase problem is one of my biggest TODO items. How did you compute FFT of the phase? The thing is, phase is discontinuous or mult

Interesting. Figuring out the phase problem is one of my biggest TODO items. How did you compute FFT of the phase? The thing is, phase is discontinuous or multivalued function if we represent phase as a real number. We could also represent phase as a complex number of unit magnitude: exp(i phi). It would be continuous, but complex-valued. And phase is indeed important for hearing: https://auditoryneuroscience.com/vocalizations-speech/speech-modulated-signal https://auditoryneuroscience.com/vocalizations-speech/speech... I didn't quite get the trick with uninterleaving the lines.

sillysaurusx 6y ago
You're in luck -- I managed to dig up my WIP notes from a year ago. https://imgur.com/xLcvLIm https://imgur.com/xLcvLIm As you can see, the raw phase waveform is very "wavy", as might be expected. It oscillates rapidly, making it hard to see the patterns. But if you go to the tweets I linked above, you'll see the phase is much smoother in those images. How did I do it? The key is to focus on every other line. Notice that if you simply pay attention to every odd row, it will be smooth. I think I simply did "row 0, row 2, row 4, ... row n" followed by "row 1, row 3, row 5, ... row n + 1" As for the fft of the fft trick for phase, I'm rsync'ing all of my old code and demo images to here: https://battle.shawwn.com/sdb/voicecloning/ https://battle.shawwn.com/sdb/voicecloning/ You may be interested in the png images, in particular the ones with "phase" in the names. You can probably ignore all the code except repl2.py. Those images were generated via unknown methods -- sadly my repl sessions weren't saved. But, I happened to write down in repl2.py how the tweet images were generated: cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs.png"), np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png"))))) cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs2.png"), -1+2.0*np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png"))))) So, input: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro... Then, using the code above, the result: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase-fft-abs2.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro... I've verified that it still works. I think you can wget those images and copy-paste that code into a python repl. So the only remaining question is, how was mel-phase-spectrogram-phase.png generated? Unfortunately that seems to be lost with the sands of time. But, as a hint, I think it was simply a matter of turning the phase component into x,y using atan2, then turning it into blue and red. Also, completely unrelated, but I once made a super high resolution mel spectrogram that looked way cool and I can't resist showing it off: https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png I did all this when making 'Dr Kleiner sings "I Am the Very Model of a Modern Major General"' around a year ago. https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=ShawnPresser https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=Shawn... Kinda funny that all of this visualization work was just to make memes, but the quest to meme turns out to be surprisingly motivating. https://battle.shawwn.com/sdb/voicecloning/demo_output_101.wav https://battle.shawwn.com/sdb/voicecloning/demo_output_101.w... https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wav https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wa... Anyway, I think there's a lot left to discover in terms of audio visualization! I would definitely encourage you to play around with the phase component. The results can be pretty striking, as you can see from the "Result" image above (https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase-fft-abs2.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...). Sorry for the scattered explanation -- it's 4am here, but I wanted to give you some kind of writeup, even if it's rather disjointed. If you have more questions, be sure to ask! I can give better details tomorrow.
- gbh444g 6y ago
  Thanks for the notes! Another idea I've been thinking about is to capture the phase by splitting ACF into multiple complementary parts at the spectral density step, i.e. when we draw the spectrogram, we use |r exp(i phi)|^2 of the FFT output and that |...|^2 drops the phi. However, we could split this |...|^2 into a sum of a few terms that would add up to the same spectral density, but would separately capture the phase. In the simplest case, |x + iy|^2 = x^2 + y^2, can be interpreted as two half-spectrograms: one for x^2 and another for y^2. If colored the same, we'd get the original spectrogram, but if colored differently, we'd know values of x^2 and y^2 and that's enough to know the phase up to a quadrant.

sillysaurusx 6y ago

You're in luck -- I managed to dig up my WIP notes from a year ago. https://imgur.com/xLcvLIm https://imgur.com/xLcvLIm As you can see, the raw phase waveform is very "wavy", as might be expected. It oscillates rapidly, making it hard to see the patterns. But if you go to the tweets I linked above, you'll see the phase is much smoother in those images. How did I do it? The key is to focus on every other line. Notice that if you simply pay attention to every odd row, it will be smooth. I think I simply did "row 0, row 2, row 4, ... row n" followed by "row 1, row 3, row 5, ... row n + 1" As for the fft of the fft trick for phase, I'm rsync'ing all of my old code and demo images to here: https://battle.shawwn.com/sdb/voicecloning/ https://battle.shawwn.com/sdb/voicecloning/ You may be interested in the png images, in particular the ones with "phase" in the names. You can probably ignore all the code except repl2.py. Those images were generated via unknown methods -- sadly my repl sessions weren't saved. But, I happened to write down in repl2.py how the tweet images were generated: cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs.png"), np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png"))))) cv2.imwrite(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase-fft-abs2.png"), -1+2.0*np.abs(np.fft.fft2(-1+2*1/255*cv2.imread(os.path.expanduser("~/Downloads/mel-phase-spectrogram-phase.png"))))) So, input: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro... Then, using the code above, the result: https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase-fft-abs2.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro... I've verified that it still works. I think you can wget those images and copy-paste that code into a python repl. So the only remaining question is, how was mel-phase-spectrogram-phase.png generated? Unfortunately that seems to be lost with the sands of time. But, as a hint, I think it was simply a matter of turning the phase component into x,y using atan2, then turning it into blue and red. Also, completely unrelated, but I once made a super high resolution mel spectrogram that looked way cool and I can't resist showing it off: https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png https://battle.shawwn.com/sdb/voicecloning/ultra-mel.png I did all this when making 'Dr Kleiner sings "I Am the Very Model of a Modern Major General"' around a year ago. https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=ShawnPresser https://www.youtube.com/watch?v=koU3L7WBz_s&ab_channel=Shawn... Kinda funny that all of this visualization work was just to make memes, but the quest to meme turns out to be surprisingly motivating. https://battle.shawwn.com/sdb/voicecloning/demo_output_101.wav https://battle.shawwn.com/sdb/voicecloning/demo_output_101.w... https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wav https://battle.shawwn.com/sdb/voicecloning/demo_output_75.wa... Anyway, I think there's a lot left to discover in terms of audio visualization! I would definitely encourage you to play around with the phase component. The results can be pretty striking, as you can see from the "Result" image above (https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectrogram-phase-fft-abs2.png https://battle.shawwn.com/sdb/voicecloning/mel-phase-spectro...). Sorry for the scattered explanation -- it's 4am here, but I wanted to give you some kind of writeup, even if it's rather disjointed. If you have more questions, be sure to ask! I can give better details tomorrow.

gbh444g 6y ago
Thanks for the notes! Another idea I've been thinking about is to capture the phase by splitting ACF into multiple complementary parts at the spectral density step, i.e. when we draw the spectrogram, we use |r exp(i phi)|^2 of the FFT output and that |...|^2 drops the phi. However, we could split this |...|^2 into a sum of a few terms that would add up to the same spectral density, but would separately capture the phase. In the simplest case, |x + iy|^2 = x^2 + y^2, can be interpreted as two half-spectrograms: one for x^2 and another for y^2. If colored the same, we'd get the original spectrogram, but if colored differently, we'd know values of x^2 and y^2 and that's enough to know the phase up to a quadrant.