17 ms·
Highres Spectrograms with the DFT Shift Theorem
- gbh444g 5y agoHello HN! Author here. I was thinking to call the post "The underappreciated complexity of musical sounds" but decided to stick with the DFT one as it would probably get more attention. This is a small discovery I came across this weekend. FFT-based spectrograms of musical instruments isn't a novel thing do, but I thought what if I do a super highres spectrogram with a continuum of freqencies, instead of the N fixed ones FFT gives. Turns out, FFT "supports" such frequency shifting by multiplying the input by a specially constructed complex exponent. As a result, I've found out that musical instruments produce sophisticated ornaments in between the harmonic levels.
- stainforth 5y agoWhat is an ornament?
- CyberRabbi 5y agoMy mind is kind of blown that birdsong virtually does not include higher harmonics. I didn’t even think that was possible for a physical resonator. Great post
- ttoinou 5y agomaybe they were not captured by the bandiwth-limited microphone ?
- akomtu 5y agoI think the mystery has a simple explanation: when a bird sings at 7 kHz and the mp3 file captures only first 20 kHz, there isn't much room for harmonics. Maybe birds do have interesting harmonics at 56 kHz, we just don't know.
- cviilgan 5y agoDid I understand this correctly, what you are doing is essentially: X[n] = F[x[k]][n/2] if (n even) else F[x'[k]][(n+1)/2] With F[x[k]] the DFT of the time-domain signal x[k], x'[k] = x[k]·exp(2·pi·i·k·alpha) and this alpha some constant which yields a frequency-domain shift by 25Hz. If so: How does this method compare to zero-padding the time-domain signal (i.e. sinc-interpolating the frequency domain)? It is an interesting concept, but alas it's not immediately clear to me how to analyze this...
- gbh444g 5y agoThis sounds about right. I assume your (n+1)/2 is really n+1/2. The idea, like you've said, is to get Y[k+1/2] values where Y = FFT[X]. Whether this is mathematically sound is another question. I presume that it is, for two reasons. First, FFT essentially convolves X with a bunch of sinusoids with frequencies from a fixed set: 0 Hz, 50 Hz, 100 Hz and so on. There's nothing wrong with manually convolving X with a 57.3 Hz sinusoid, it's just FFT isn't designed for this (it's designed for rapid computation). The other reason is that combining such shifted FFTs we get what looks almost exactly like a CWT (i.e. wavelet transform). As for sinc-interpolation, I think it's mathematically equivalent. Say we shift the input X with Z[k] = exp(ik/N...) and get XZ. Then we transform it to FFT[XZ] = FFT[X] conv FFT[Z], so it's convolving FFT[X] with FFT[Z] where FFT[Z] is probably that sinc kernel. I certainly know from experiments that FFT of exp(2·pi·i·k·alpha) where alpha doesn't precisely align with the 1024 grid produces a fuzzy function with a max around alpha and a bell-shaped curved around it, the width of the curve depends on how precisely alpha fits into one of the 1024 grid points.
- ggerganov 5y agoInstead of combining 2 FFTs of 1024 bins (one shifted + one non-shifted), could you not just calculate 1 FFT of 2048 bins? Isn't it the same result?
- gbh444g 5y agoLarger FFT window has undesired side effects because the estimated frequencies are averaged over the entire window. Moreover, the FFT output always spans from 0 Hz to 24 kHz (with 48 kHz sample rate), so to zoom into the 0..6 kHz region we'll need a window with 8192 bins or about 150 ms. With such window it would be impossible to capture rapid volume oscillations.
- Prcmaker 5y agoThanks for writing this up, I'm always on the look out for alternative methods for DFTs and the like, currently concentrating on interpolation of low frequencies (after DC, but still within the first 5% of wave numbers) . I'll see if this fits my use case soon, hopefully today.
- Lichtso 5y agoOn that note, also checkout wavelets to generate spectrograms: https://en.wikipedia.org/wiki/Wavelet https://en.wikipedia.org/wiki/Wavelet I have some implementations here: https://github.com/Lichtso/CCWT https://github.com/Lichtso/CCWT https://github.com/Lichtso/WebSpectrogram https://github.com/Lichtso/WebSpectrogram
- kortex 5y agoThis is fantastic! About 5 years ago (just before this repo was made it seems) I was doing a ton of stuff with EEG analysis with python. Used CWTs a ton but it was slooow, even with lots of numpy tricks. This would have been super handy. I also learned in that time that while you can extract a complex signal from a real one using the Hilbert transform, it's not quite the same, and I've always wondered if we could achieve better fidelity/encoding/compression by starting with quadrature signals. Never figured out quite why, since Shannon-Nyquist says you should be able to encode all information of a bandwidth f signal with 2f sample rate, but I suspect it has to do with the difference between ideal real number math and nonlinear, quantizing, imperfect ADCs. Not sure how you'd actually get quadrature signals from sound waves or any wideband scalar signal (maybe record at far higher sampling rate to get more phase information, then downsample), but it's a fun thought experiment.
- krapht 5y agoYou can convert to quadrature by either sampling a signal at >= 2 * Nyquist and using the Hilbert transform, or using two ADCs 90 degrees out of phase. When you run two ADCs 90 degrees out of phase that introduces another source of error, due to timing jitter. There's no reason to bother doing this for audio signals because modern ADCs are more than capable of accurately sampling at audio rates.
- efnx 5y agoI love this and have been looking for a program that's like Photoshop for sound.
- layoutIfNeeded 5y agoYou can try interpreting images as spectrograms, but the result will be a cacophonic mess. There’s a reason why nobody does this. (Other than avantgarde experimental composers maybe, but they are looking for cacophony)
- deleted 5y ago[deleted]
- danwills 5y agoMost often yields a cacophonous mess, it's true - but it really depends how carefully the image is made. If the image was made using a fullly-detail fft of a sound then you could in theory get the exact same sound back out that you put in I reckon! :) (admittedly negative color values will be required (or clever mapping), and spreading time-in-the-sound over space-in-the-image).
- danwills 5y agoWords escaped me before by "fully detail", as explained on another threads up there :), I meant that the input image must contain (or have extracted from it) phase information in addition to the amplitudes that a spectrograph-like extraction would provide to have full control over the result (alternatively, a complex valued image (usually positive and negative) would do it). It's hard to make phase-info up from nothing I reckon, maybe that's why many image-to-sound things sound harsh/strange?
- dkarras 5y ago>looking for a program that's like Photoshop for sound. Look at Izotope RX. Especially the Spectral Repair module might be what you are imagining, but it has a lot of interesting tools. This is from an older version: https://www.youtube.com/watch?v=vNtxg28wx_M https://www.youtube.com/watch?v=vNtxg28wx_M
- crazygringo 5y agoThis looks cool! But really needs "before" and "after" comparison images -- lo-res vs hi-res. Seeing the hi-res images only gives me no idea what kind of improvement this is showing... @gbh444g Hope you could maybe add some lo-res versions :) (Would also be cool to have audio clips next to each image as well, but that's less important.)
- swiley 5y agoIt had a bit of that for the bird songs.
- crazygringo 5y ago> Smoothness in the time direction is easier to achieve: the 1024 bins window can be advanced by arbitrarily small time steps. It appears you're doing just that, but the time "width" is still readily apparent in many of the spectrograms, most obviously on the birdsong ones -- almost like a horizontal motion blur. Would a deconvolution filter be able to meaningfully horizontally "deblur" the spectrograms? So the birdsongs didn't appear to be drawn with a wide-tip marker, but rather a ballpoint pen? So not just hi-res, but hi-focus.
- malka 5y agoThis might be relevant: https://ccrma.stanford.edu/~juhan/super_spec.html https://ccrma.stanford.edu/~juhan/super_spec.html
- crazygringo 5y agoThank you for that, that is fascinating!
- LeegleechN 5y agoIt's unfortunate that the article doesn't get into the fundamental limits of spectrogram resolution which are based on the famous uncertainty principle(https://en.wikipedia.org/wiki/Fourier_transform#Uncertainty_principle https://en.wikipedia.org/wiki/Fourier_transform#Uncertainty_...). For example there is a fundamental tradeoff between frequency resolution and time resolution similar to the position/momentum tradeoff in quantum mechanics. The Continuous Wavelet Transform which is alluded to in the article is a way to tune that tradeoff by frequency bin to best align with human sound perception.
- andai 5y agoI've been wondering about the apparent contradiction between the limitations of spectrograms and the remarkable fidelity of MP3 files, which I thought operated along similar lines. When you convert a spectrogram back into sound it sounds like crap, but then how does MP3 store the frequency information (and why can't we use that for visualizations)? The math is beyond my understanding, can anyone give some kind of analogy maybe?
- gugagore 5y agoThe general concepts are described here: https://en.m.wikipedia.org/wiki/Psychoacoustics https://en.m.wikipedia.org/wiki/Psychoacoustics I'm not sure what you mean by converting the spectrogram to sound, but my guess is that the windowing done on the short-time Fourier transform is causing artifacts.
- achillesheels 5y agoMy hypothesis: it is stored magnetically (after all magnetic sinusoidals exist) and converted electrically once the mp3 is activated in time.
- achillesheels 5y agoI’m answering further: it is a result of the software programs limitations to reconstruct the mathematically perfect electromagnetic waveform quantized beforehand which is causing noise. No one writes perfect code.
- andai 5y agoJust a heads up, you have to click the images to see the full resolution version! I spent a good while confused about not being able to see the details mentioned in the images.
- kazinator 5y ago> A typical FFT-based spectrogram uses 1024 bins on a 48 kHz audio, with about 50 Hz step per pixel. Most of the interesting audio activity happens below 3 kHz, so 50 Hz per pixel gives only 60 pixels for that area. That seems misleading. First of all, how often do you take a 1024 sample FFT? In theory, you could calculate it every sample, in which case you have 60 pixels, but 48,000 times per second. Secondly, you can make use of frame-over-frame phase information. If you are looking at signals with mostly periodic content in that 3 kHz band, the phase information can indicate how much the signal in a given band deviates from that band's center frequency. If the signal is dead on the frequency, then the phase component is stable frame-over-frame; the value does not move. If the signal is off, the phase angle shifts, kind of like a CRT television that is out of vertical sync. Each frame finds catches the signal in a different phase compared to the previus frame due to the frequency drift. The farther the signal is from the FFT band's frequency, the faster the phase angle rotates. If you analyze the movement of phase of the same bin between successive frames, you can get a higher resolution estimate of the frequency than what you might think is possible from the 50 Hz resolution of that bin. What you can't resolve is the situation when multiple independent signals clash into that same frequency bin. The assumption has to holds that the the bin has caught one periodic signal.
- jmpeax 5y ago> Despite this CWT implementation runs on GPU and this “advanced” FFT runs on JS, CWT is about 50-100x slower. Sounds like a really crappy implementation of CWT. Besides this, the mother wavelet used was not specified, so maybe the author doesn't really know much about CWT.
- bobowzki 5y agoThe spectrograms on this site have a lot of spectral leakage. This can be improved a lot by applying a window function (blackman, hanning etc). It doesn't seem like the author does this.
- gbh444g 5y agoApplying the Hann window function eliminates all the spectral leakage, but it also makes the image rather dull and precise, very similar to CWT. You've made me realise that the intricate patterns seen on violin spectragrams are the result of interference of the spectral leakage from main harmonics. It doesn't mean the patterns are fake. It means the patterns emerge only when the input sound is transformed a certain way (FFT with the rectangular window).
- zihotki 5y agoI wonder how can we make assumptions about the bird songs while not taking into account how birds perceive the sound. For humans it's easier, there were a plenty of studies done in that regards and there is even a separate science field for studying the human sound perception - Psychoacoustics. Humans perceive sound in bands (a band is a range of frequencies), not separate frequencies. And the size of bands vary per frequency so that in the voice range it's more narrow than, for example, high frequencies. The FFT fits very nicely into that picture and codecs were designed considering the human perception. As for animals, I don't know any studies in that regards. I would assume that the way of perception should be very similar to the one human has, at least on the mechanics level. As for the sensitivity and the size of bands as well as dynamic range - it's hard to say. I'd love to see some studies that dig into details there but it seems that it's very hard to do them. Animals don't give you a direct feedback.
- tantalor 5y ago> as if birds “draw” with sound something that’s flying backwards in time huh?
- neogodless 5y agoSome of my family and I have been enjoying playing with the BirdNET[0] app which seems to use the ideas presented here to identify birds from recordings, utilizing machine learning. [0] https://play.google.com/store/apps/details?id=de.tu_chemnitz.mi.kahst.birdnet https://play.google.com/store/apps/details?id=de.tu_chemnitz..., https://apps.apple.com/us/app/birdnet/id1541842885 https://apps.apple.com/us/app/birdnet/id1541842885