4 ms·

Music visualization is really underdeveloped in my opinion. I've never seen anything I find impressive. The problem is it never really looks very much like the

by audunw 5y ago

Music visualization is really underdeveloped in my opinion. I've never seen anything I find impressive. The problem is it never really looks very much like the visualization relates to the music I'm listening too. Usually you just kind of see the rythm of the music at most. I want to see something where harmonies look like beautiful patterns, noise looks like noise , and perhaps you could even pick out individual instruments in the visualization. It's really difficult though, which I guess is why it hasn't been done. You need to re-implement much of human audio perception, and then map that to a visual language.

Guillaume86 5y ago
I think it's a problem similar to maps in rythm games (thinking Beat Saber in my case), auto generated ones are never as good as manually crafted ones. If it was possible to extract the dominantly perceived beat/instrument/melody, it would probably lead to better map generation too.
Retr0id 5y ago
I think this is something that neural networks would be great at, I suppose the hard part is figuring out exactly what to train one for/with. As a starting point, you could separate the different instruments, which I'm pretty sure is a solved problem by now.
ushakov 5y ago
maybe deadmau5 would change your opinion on that? https://m.youtube.com/watch?v=Jrb5PqiDMSY https://m.youtube.com/watch?v=Jrb5PqiDMSY
jamal-kumar 5y ago
Most of these just work on playing loops set at certain BPMs at the lower end of actual responsiveness, to a fast fourier transform which can maybe separate out further based on frequency range, maybe just roughly from bass/mid/treble, further mapped to rendering (Like here in projectm, which as I recall is just a fork of milkdrop). There was some really cool research I found recently which I think I'm linking here if I'm not mistaken: https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-music-visualization-pt-1/ https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-mu... They've gone and tweaked the basic formula from just FFT to something that actually does "peak detection", which is better explained in that article than I can here as it involves a lot of math. There's some ideas out there to be sure. YEARS ago (so it looks kind of fuzzy now) I remember this guy hacked together some synths he programmed with visuals generated for them, it's really psychedelic stuff: https://vimeo.com/3288925 https://vimeo.com/3288925 I actually talked to the person who made this and they said this took like 9 hours to render. So not exactly real-time stuff back then, might be now. The coolest stuff these days is in 3D and projection mapped. https://vimeo.com/485066000 https://vimeo.com/485066000
Fellshard 5y ago
The gag to this song[1] is a bit crass, but one of the visualization tools in particular has always caught my eye: the hexagonally-arranged note visualization. It seems to capture a number of chord relationships in a very intriguing and intuitive way. [1] https://www.youtube.com/watch?v=2hmwXThnqi0 https://www.youtube.com/watch?v=2hmwXThnqi0
- motohagiography 5y ago
  Thought I recognized one of the viz methods, and the hexoganal representation of the chords may be based on this idea from Euler: https://en.wikipedia.org/wiki/Tonnetz https://en.wikipedia.org/wiki/Tonnetz
  - codetrotter 5y ago
    Also similar: https://www.semantic-danielou.com/semantic-danielou-53/hex-hal-axis-64-keyboards-and-doepfer-tape-semantic-danielou-53/ https://www.semantic-danielou.com/semantic-danielou-53/hex-h... https://llllllll.co/t/hybrid-hexagonal-keyboard-for-midi-typing/36679 https://llllllll.co/t/hybrid-hexagonal-keyboard-for-midi-typ... http://www.yeco.io/introhexblog.html http://www.yeco.io/introhexblog.html https://www.lumatone.io/ https://www.lumatone.io/ https://hackaday.com/2019/07/13/isomorphic-keyboards-with-cv-out/ https://hackaday.com/2019/07/13/isomorphic-keyboards-with-cv... https://reverb.com/au/item/18946829-c-thru-music-axis-49-isomorphic-usb-midi-controller https://reverb.com/au/item/18946829-c-thru-music-axis-49-iso... https://en.wikipedia.org/wiki/Isomorphic_keyboard https://en.wikipedia.org/wiki/Isomorphic_keyboard https://web.archive.org/web/20120219085411/http://www.theshapeofmusic.com/note-pattern.php https://web.archive.org/web/20120219085411/http://www.thesha... https://www.dynamictonality.com/hex.htm https://www.dynamictonality.com/hex.htm I saw some music software once also a few years ago that mentioned this type of input device. Don’t remember if the software in question was a VST or a DAW or what it was so can’t find it at the moment. The software that mentioned it may have been open source but I don’t remember.
  - Fellshard 5y ago
    I figured there would be mathematical foundations to it, and it would figure Euler is involved. Very neat!
laumars 5y ago
In fairness, projectm is literally built on 90s tech (Winamp’s milkdrop). For some reason around the early 00s peoples love for visualisations fell out of favour. Meanwhile clubs would still hire dedicated VJs and/or lighting guys so didn’t rely on math-based visualisations. Which meant fewer people contributing to developing cool new math. Worse still, the few industries left that did care about beat detection were games like Guitar Hero, Beat Sabre and those dance machines in arcades; all of which were easier to have patterns programmed by a human due to the limited selection of songs on offer.
- dmitriid 5y ago
  > For some reason around the early 00s peoples love for visualisations fell out of favour It didn't. Music companies effectively killed it because music visualization is a combination of derivative work and performance. So, copyright, licensing etc. Can't find the relevant link though.
- sli 5y ago
  I still setup both Winamp AVS and ProjectM inside foobar2000 simultaneously to this day. I don't use foobar too much these days, but when I do, I have those two things running.
  - iszomer 5y ago
    ProjectM integrates well with PowerAmp on Android too. QNMP (Linux) also supports it but is not as intuitive to configure.
eointierney 5y ago
Really really difficult, and maybe not possible for many years to come. Someone recently posted a link to Terence Tao on mathematical notation. Much the same problem. Notation is hard. Indeed I reckon notation is one of the most difficult problems we have. That said, visualising Music is kinda like musicalising Art. Possible but not necessarily useful. Synesthesia is one way to deal with it, but is hugely reliant on very human perception and kinda iffy (not knocking those who can experience synesthetically, more suggesting https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F). A general theory of correspondence of meaning and notation is still unavailable, notwithstanding a vast amount of research in the field. If you can make any useful contribution at all you are worthy of historical recognition. I've been thinking about this stuff for decades, and can only recommend Charles Saunders Pierce as a good beginning for the modern mind.
nabakin 5y ago
I wanted something like this 4 years ago so I started writing a project called Drop [0] with the intention of making it possible to add custom visualizations through a plugin interface, providing plugin developers with tools such as beat detection, and a decent user interface. I added a lot of features and basically finished the interface but never got around to finishing the plugin portion. [0] https://github.com/xNul/drop https://github.com/xNul/drop You might be interested in this btw https://delu.medium.com/a-perceptually-meaningful-audio-visualizer-ee72051781bc https://delu.medium.com/a-perceptually-meaningful-audio-visu...
JKCalhoun 5y ago
Yep. I remember the early visualizers did indeed display something akin to frequency spectral data. I wrote a few visualizers for SoundJam (later iTunes) and the API was exactly that: left and right bucketed spectral data. I know, bar graphs got a little tiresome so people tried writing more interesting stuff but at some point the "visualizer" became so disconnected from the music that even if you paused the audio the fireworks and "flow fazing" just kept right on doing its thing.
john61 5y ago
It is very simple: just go out and look in a face of a playing/singing musician
jgjot-singh 5y ago
The problem there isn't necessarily development, but rather the data accessible to developers. What you need to have that level of visual interaction is the original stems (or something which emulates them relatively closely). Services like https://www.lalal.ai/ https://www.lalal.ai/ are a step in the right direction, and will probably lead to what you--and many of us--crave. But as it is now, they require way too much computing to provide meaningful enough data fast enough to build a real-time visualiser. But even then there is the whole issue of copyright which would bottleneck development even further...
naikrovek 5y ago
try finding something that lets you do an XY plot like an oscilloscope does. left channel moves the "pen" left and right, right channel moves it up and down, and it draws a fading line as it goes. it's amazing to watch, to me.

by audunw 5y ago

Guillaume86 5y ago

I think it's a problem similar to maps in rythm games (thinking Beat Saber in my case), auto generated ones are never as good as manually crafted ones. If it was possible to extract the dominantly perceived beat/instrument/melody, it would probably lead to better map generation too.

Retr0id 5y ago

I think this is something that neural networks would be great at, I suppose the hard part is figuring out exactly what to train one for/with. As a starting point, you could separate the different instruments, which I'm pretty sure is a solved problem by now.

ushakov 5y ago

maybe deadmau5 would change your opinion on that? https://m.youtube.com/watch?v=Jrb5PqiDMSY https://m.youtube.com/watch?v=Jrb5PqiDMSY

jamal-kumar 5y ago

Most of these just work on playing loops set at certain BPMs at the lower end of actual responsiveness, to a fast fourier transform which can maybe separate out further based on frequency range, maybe just roughly from bass/mid/treble, further mapped to rendering (Like here in projectm, which as I recall is just a fork of milkdrop). There was some really cool research I found recently which I think I'm linking here if I'm not mistaken: https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-music-visualization-pt-1/ https://ciphrd.com/2019/09/01/audio-analysis-for-advanced-mu... They've gone and tweaked the basic formula from just FFT to something that actually does "peak detection", which is better explained in that article than I can here as it involves a lot of math. There's some ideas out there to be sure. YEARS ago (so it looks kind of fuzzy now) I remember this guy hacked together some synths he programmed with visuals generated for them, it's really psychedelic stuff: https://vimeo.com/3288925 https://vimeo.com/3288925 I actually talked to the person who made this and they said this took like 9 hours to render. So not exactly real-time stuff back then, might be now. The coolest stuff these days is in 3D and projection mapped. https://vimeo.com/485066000 https://vimeo.com/485066000

Fellshard 5y ago

The gag to this song[1] is a bit crass, but one of the visualization tools in particular has always caught my eye: the hexagonally-arranged note visualization. It seems to capture a number of chord relationships in a very intriguing and intuitive way. [1] https://www.youtube.com/watch?v=2hmwXThnqi0 https://www.youtube.com/watch?v=2hmwXThnqi0

motohagiography 5y ago
Thought I recognized one of the viz methods, and the hexoganal representation of the chords may be based on this idea from Euler: https://en.wikipedia.org/wiki/Tonnetz https://en.wikipedia.org/wiki/Tonnetz
- codetrotter 5y ago
  Also similar: https://www.semantic-danielou.com/semantic-danielou-53/hex-hal-axis-64-keyboards-and-doepfer-tape-semantic-danielou-53/ https://www.semantic-danielou.com/semantic-danielou-53/hex-h... https://llllllll.co/t/hybrid-hexagonal-keyboard-for-midi-typing/36679 https://llllllll.co/t/hybrid-hexagonal-keyboard-for-midi-typ... http://www.yeco.io/introhexblog.html http://www.yeco.io/introhexblog.html https://www.lumatone.io/ https://www.lumatone.io/ https://hackaday.com/2019/07/13/isomorphic-keyboards-with-cv-out/ https://hackaday.com/2019/07/13/isomorphic-keyboards-with-cv... https://reverb.com/au/item/18946829-c-thru-music-axis-49-isomorphic-usb-midi-controller https://reverb.com/au/item/18946829-c-thru-music-axis-49-iso... https://en.wikipedia.org/wiki/Isomorphic_keyboard https://en.wikipedia.org/wiki/Isomorphic_keyboard https://web.archive.org/web/20120219085411/http://www.theshapeofmusic.com/note-pattern.php https://web.archive.org/web/20120219085411/http://www.thesha... https://www.dynamictonality.com/hex.htm https://www.dynamictonality.com/hex.htm I saw some music software once also a few years ago that mentioned this type of input device. Don’t remember if the software in question was a VST or a DAW or what it was so can’t find it at the moment. The software that mentioned it may have been open source but I don’t remember.
- Fellshard 5y ago
  I figured there would be mathematical foundations to it, and it would figure Euler is involved. Very neat!

laumars 5y ago

In fairness, projectm is literally built on 90s tech (Winamp’s milkdrop). For some reason around the early 00s peoples love for visualisations fell out of favour. Meanwhile clubs would still hire dedicated VJs and/or lighting guys so didn’t rely on math-based visualisations. Which meant fewer people contributing to developing cool new math. Worse still, the few industries left that did care about beat detection were games like Guitar Hero, Beat Sabre and those dance machines in arcades; all of which were easier to have patterns programmed by a human due to the limited selection of songs on offer.

dmitriid 5y ago
> For some reason around the early 00s peoples love for visualisations fell out of favour It didn't. Music companies effectively killed it because music visualization is a combination of derivative work and performance. So, copyright, licensing etc. Can't find the relevant link though.
sli 5y ago
I still setup both Winamp AVS and ProjectM inside foobar2000 simultaneously to this day. I don't use foobar too much these days, but when I do, I have those two things running.
- iszomer 5y ago
  ProjectM integrates well with PowerAmp on Android too. QNMP (Linux) also supports it but is not as intuitive to configure.

eointierney 5y ago

Really really difficult, and maybe not possible for many years to come. Someone recently posted a link to Terence Tao on mathematical notation. Much the same problem. Notation is hard. Indeed I reckon notation is one of the most difficult problems we have. That said, visualising Music is kinda like musicalising Art. Possible but not necessarily useful. Synesthesia is one way to deal with it, but is hugely reliant on very human perception and kinda iffy (not knocking those who can experience synesthetically, more suggesting https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F). A general theory of correspondence of meaning and notation is still unavailable, notwithstanding a vast amount of research in the field. If you can make any useful contribution at all you are worthy of historical recognition. I've been thinking about this stuff for decades, and can only recommend Charles Saunders Pierce as a good beginning for the modern mind.

nabakin 5y ago

I wanted something like this 4 years ago so I started writing a project called Drop [0] with the intention of making it possible to add custom visualizations through a plugin interface, providing plugin developers with tools such as beat detection, and a decent user interface. I added a lot of features and basically finished the interface but never got around to finishing the plugin portion. [0] https://github.com/xNul/drop https://github.com/xNul/drop You might be interested in this btw https://delu.medium.com/a-perceptually-meaningful-audio-visualizer-ee72051781bc https://delu.medium.com/a-perceptually-meaningful-audio-visu...

JKCalhoun 5y ago

Yep. I remember the early visualizers did indeed display something akin to frequency spectral data. I wrote a few visualizers for SoundJam (later iTunes) and the API was exactly that: left and right bucketed spectral data. I know, bar graphs got a little tiresome so people tried writing more interesting stuff but at some point the "visualizer" became so disconnected from the music that even if you paused the audio the fireworks and "flow fazing" just kept right on doing its thing.

john61 5y ago

It is very simple: just go out and look in a face of a playing/singing musician

jgjot-singh 5y ago

The problem there isn't necessarily development, but rather the data accessible to developers. What you need to have that level of visual interaction is the original stems (or something which emulates them relatively closely). Services like https://www.lalal.ai/ https://www.lalal.ai/ are a step in the right direction, and will probably lead to what you--and many of us--crave. But as it is now, they require way too much computing to provide meaningful enough data fast enough to build a real-time visualiser. But even then there is the whole issue of copyright which would bottleneck development even further...

naikrovek 5y ago

try finding something that lets you do an XY plot like an oscilloscope does. left channel moves the "pen" left and right, right channel moves it up and down, and it draws a fading line as it goes. it's amazing to watch, to me.