Most of these just work on playing loops set at certain BPMs at the lower end of actual responsiveness, to a fast fourier transform which can maybe separate out

Most of these just work on playing loops set at certain BPMs at the lower end of actual responsiveness, to a fast fourier transform which can maybe separate out further based on frequency range, maybe just roughly from bass/mid/treble, further mapped to rendering (Like here in projectm, which as I recall is just a fork of milkdrop). There was some really cool research I found recently which I think I'm linking here if I'm not mistaken: https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-music-visualization-pt-1/ https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-mu... They've gone and tweaked the basic formula from just FFT to something that actually does "peak detection", which is better explained in that article than I can here as it involves a lot of math. There's some ideas out there to be sure. YEARS ago (so it looks kind of fuzzy now) I remember this guy hacked together some synths he programmed with visuals generated for them, it's really psychedelic stuff: https://vimeo.com/3288925 https://vimeo.com/3288925 I actually talked to the person who made this and they said this took like 9 hours to render. So not exactly real-time stuff back then, might be now. The coolest stuff these days is in 3D and projection mapped. https://vimeo.com/485066000 https://vimeo.com/485066000