5 ms·

I find this surprising, from the simplistic (and probably naive) view that images are 2D signals while music is 1D.

deleted 9y ago
[deleted]
contrarian_ 9y ago
Much more leeway for "unintended cheating" with 2D signals. Think of all the imperceptible patterns that could be used as distinguishers. ML methods basically search for these shortcuts.
phelm 9y ago
how about 2d in frequency domain?
kastnerkyle 9y ago
"Style transfer" also rarely works for object level transfer - it is more pattern based (high frequency content is often the "style" that is enhanced and transferred). Really nice transfers in practice sometimes require the object level content in the images to be similar, c.f. [0][1]. And all of this is coupled with really heavy human curation (people don't normally show their bad outputs)! In music the "style" is the content in some sense. For example jazz has very different "style" than classical, at many levels (key and tempo choice/mode choice/melodic intervals/motifs/amount of repetition of said motif/how it varies/harmonization and chord choice/global structure (AABA format)) and it isn't easy separate what pieces make it "jazz", and what don't (what factors of variation matter). The equivalent in images would be replacing objects as well as texture, to form a new image that is reminiscent of the original but also novel at multiple scales - think Simpson's "Last Supper" as the goal of a style transfer [2]. It is also hard because as consumers we are used to hearing high quality versions of these types of "style transfer" for some styles all the time - and we even have a name for it ... "muzak". [0] https://raw.githubusercontent.com/awentzonline/image-analogies/master/examples/images/sugarskull-analogy.jpg https://raw.githubusercontent.com/awentzonline/image-analogi... [1] https://github.com/chuanli11/CNNMRF https://github.com/chuanli11/CNNMRF [2] http://s267.photobucket.com/user/wiro_bucket/media/last%20supper/SimpsonsLastSupper.jpg.html http://s267.photobucket.com/user/wiro_bucket/media/last%20su...
- taco_emoji 9y ago
  I think "cover song" is a more generic term than "muzak" for musical style transfer.
  - kastnerkyle 9y ago
    It can be, though some covers are "straight up", while others (generally the memorable ones) are practically a new creation in themselves, with a sliding scale in between. But for "cover song" meaning something like Hendrix's "All Along the Watchtower" or Coltrane's "My Favorite Things", I agree.
- euyyn 9y ago
  Thanks for this detailed explanation. The Simpson's Last Supper analogy really made it click for me.
lioeters 9y ago
Wouldn't music be 4D, ąs sound in space through time?
- ben_w 9y ago
  Only if you have a different track playing at every point in space.
teilo 9y ago
Even images, from an analysis perspective are far more than 2D. Think of the problem space: the types of lines, shading, coloration, blending of colors, overall structure, etc. All of this must come into play with style transfer. With music, it's much more complicated due not only to the time domain, but also to the emotional content, the interrelationships of rhythm, tempo, chordal progressions, expression, timber, transitions from the simple to the complex, the way the music is layered, and how all these things are perceived by the listener. Like human faces, music elicits an emotional response that runs deep, and whereas the emotional content of a picture is relatively static, music's emotional content is dynamic and evolving. Music has its own "uncanny valley" between the machine and the human. Think of a MIDI track that is generated by precisely laying out notes and velocities on a piano roll, vs. a track that is recorded live and captures the dynamics of the player. The difference is immediately obvious, and that with a track that is composed by a human.
TheOtherHobbes 9y ago
Music isn't any more 1D than language is.
- enord 9y ago
  We perceive sound two-dimensionally by spectral decomposition in the cochlea, so theres that...
enord 9y ago
Sound (as we perceive it) is 2d by the time it reaches the brain. The cochlea works as a "discrete" fourier transformer, with little hairs that resonate around certain frequencies. It is the fact that sound could be encoded in 1d that was, at one time, astounding.

deleted 9y ago

[deleted]

Much more leeway for "unintended cheating" with 2D signals. Think of all the imperceptible patterns that could be used as distinguishers. ML methods basically search for these shortcuts.

phelm 9y ago

how about 2d in frequency domain?

kastnerkyle 9y ago

"Style transfer" also rarely works for object level transfer - it is more pattern based (high frequency content is often the "style" that is enhanced and transferred). Really nice transfers in practice sometimes require the object level content in the images to be similar, c.f. [0][1]. And all of this is coupled with really heavy human curation (people don't normally show their bad outputs)! In music the "style" is the content in some sense. For example jazz has very different "style" than classical, at many levels (key and tempo choice/mode choice/melodic intervals/motifs/amount of repetition of said motif/how it varies/harmonization and chord choice/global structure (AABA format)) and it isn't easy separate what pieces make it "jazz", and what don't (what factors of variation matter). The equivalent in images would be replacing objects as well as texture, to form a new image that is reminiscent of the original but also novel at multiple scales - think Simpson's "Last Supper" as the goal of a style transfer [2]. It is also hard because as consumers we are used to hearing high quality versions of these types of "style transfer" for some styles all the time - and we even have a name for it ... "muzak". [0] https://raw.githubusercontent.com/awentzonline/image-analogies/master/examples/images/sugarskull-analogy.jpg https://raw.githubusercontent.com/awentzonline/image-analogi... [1] https://github.com/chuanli11/CNNMRF https://github.com/chuanli11/CNNMRF [2] http://s267.photobucket.com/user/wiro_bucket/media/last%20supper/SimpsonsLastSupper.jpg.html http://s267.photobucket.com/user/wiro_bucket/media/last%20su...

taco_emoji 9y ago
I think "cover song" is a more generic term than "muzak" for musical style transfer.
- kastnerkyle 9y ago
  It can be, though some covers are "straight up", while others (generally the memorable ones) are practically a new creation in themselves, with a sliding scale in between. But for "cover song" meaning something like Hendrix's "All Along the Watchtower" or Coltrane's "My Favorite Things", I agree.
euyyn 9y ago
Thanks for this detailed explanation. The Simpson's Last Supper analogy really made it click for me.

lioeters 9y ago

Wouldn't music be 4D, ąs sound in space through time?

ben_w 9y ago
Only if you have a different track playing at every point in space.

teilo 9y ago

Even images, from an analysis perspective are far more than 2D. Think of the problem space: the types of lines, shading, coloration, blending of colors, overall structure, etc. All of this must come into play with style transfer. With music, it's much more complicated due not only to the time domain, but also to the emotional content, the interrelationships of rhythm, tempo, chordal progressions, expression, timber, transitions from the simple to the complex, the way the music is layered, and how all these things are perceived by the listener. Like human faces, music elicits an emotional response that runs deep, and whereas the emotional content of a picture is relatively static, music's emotional content is dynamic and evolving. Music has its own "uncanny valley" between the machine and the human. Think of a MIDI track that is generated by precisely laying out notes and velocities on a piano roll, vs. a track that is recorded live and captures the dynamics of the player. The difference is immediately obvious, and that with a track that is composed by a human.

TheOtherHobbes 9y ago

Music isn't any more 1D than language is.

enord 9y ago
We perceive sound two-dimensionally by spectral decomposition in the cochlea, so theres that...

enord 9y ago

Sound (as we perceive it) is 2d by the time it reaches the brain. The cochlea works as a "discrete" fourier transformer, with little hairs that resonate around certain frequencies. It is the fact that sound could be encoded in 1d that was, at one time, astounding.