3 ms·

An image has boundaries, shading, light effects like reflection and refraction, shadows, gradients and coherence. The complexity is way, way more than what you

An image has boundaries, shading, light effects like reflection and refraction, shadows, gradients and coherence. The complexity is way, way more than what you would find in an audio file. I don’t think your analysis is valid.

syntheweave 4y ago
When you store it on disk, an image is zeros and ones, the same as audio. The informational complexity is definable in terms of how much it can be compressed, and if lossy, an acceptable range of information loss.
- bergenty 4y ago
  Yes but in terms of degrees of freedom, an image is way more complex. You can literally quantify every aspect of how audio is represented, something you can’t do with an image.
  - syntheweave 4y ago
    "Every aspect" would mean unlimited numbers of sound sources from all distances and directions, which is not how we represent sound, which is usually limited to two stereo channels at 44.1kHz and 16 bits in CD-quality audio, but with higher resolutions for surround or to reduce signal processing artifacts. An image is likewise limited by representation in that we define against a resolution, color depth, gamma, etc. Images define a large number of sample sources, but without variation over time. And in most circumstances, like with audio, we also currently limit the sources to fixed areas, rather than generating surround images. What we're relying on in using these formats successfully is knowing the limits of our sensory perception, and then what our brain does to process it. It says nothing about "A is easier to represent symbolically than B", because of course you can come up with symbolic compression schemes; those are how humans generate interesting images and audio. But the transmission protocol doesn't need to know them.
masswerk 4y ago
I think, these are all criteria of seamlessness, something, which an artificial composition must manage as well. Images, on the other hand lack the temporal dimension, variations in tempo, musical tones and scale, emotional impact, etc, which are crucial to seamlessness and continuity in music. The problem really arises with framing, composition, readable intent and motivation, i.e., general terms of expression – and image generation doesn't exactly shine here, as well. (The answer to any motivational questions will probably always be, "strongest signal". An artistic motivation will probably never be to just express the strongest signal in a common cultural context, rather, it will be contrast and interpretation. Otherwise, we should start worrying about that given society.)
woojoo666 4y ago
That complexity can actually make it more forgiving to mistakes. Visual art can contain lighting inconsistencies and still look fine. For example, a recent HN discussion looked at Dall-E Mini art involving origami [1], and while they look good, there are lots of clear mistakes. Perhaps in music, these mistakes are more obvious. Though at the same time, it seems like Dall-E2 is really good at faces [2], which is an example where there is a lot of structure, and viewers are very good at finding mistakes (the "uncanny valley"). Dall-E2 is of course a much bigger model with more training than Dall-E mini. Perhaps with the same amount of training, music AI can get to the same level. [1]: https://news.ycombinator.com/item?id=31787743 https://news.ycombinator.com/item?id=31787743 [2]: https://www.vice.com/en/article/g5vbx9/dall-e-is-now-generating-realistic-faces-of-fake-people https://www.vice.com/en/article/g5vbx9/dall-e-is-now-generat...
notahacker 4y ago
I don't think your analysis of my comment is valid. :) Regardless of opinion on audio files, it's the case that many AI approaches to music have focused on symbolic representations of music, which works, as do non-NN ways of procedurally generating music using the same symbols. What AI can theoretically add to a procedurally generated midi track is expressive performance, but that's not exactly its strength and probably needs better curation than just giving it lots of audio files. And even the "pure audio" approaches like OpenJukebox have to break out vocals and treat them separately (because it's something of an understatement to say that reflecting a static image is not a more complex problem than generating four minutes of singing with appropriate pitch, intonation and emotion and the impression of a consistent genre-appropriate vocalist), and the background music is... kind of muddy sounding and compositionally uneven. The main thrust of my argument wasn't the complexity of music though, the main thrust is comparative difficulty of compiling the training sets. More work to annotate and more licensing complications than just scooping up public domain images especially if you want a well rounded model. For another example take OpenAI's jukebox. It's clear the "Heavy Metal In The Style of Rage" vocals use a model trained on metal bands, but it's also clear that Rage Against The Machine wasn't prominent amongst them. Also probably the case there's simply not enough Rage Against The Machine albums to get your digital Zach de la Rocha vocals out of the uncanny valley even with the relevant permissions
xodjmk 4y ago
An image has none of these things. That is all put together in a person's mind psychologically, and what neural nets are trained to infer. In a 2D image there is no difference between actual shadows, and for example some flat picture of different colored shapes. Images are very simple. There is very little correlation past a few pixels in any direction, which is why most image filter are only a few pixels wide.

syntheweave 4y ago

When you store it on disk, an image is zeros and ones, the same as audio. The informational complexity is definable in terms of how much it can be compressed, and if lossy, an acceptable range of information loss.

bergenty 4y ago
Yes but in terms of degrees of freedom, an image is way more complex. You can literally quantify every aspect of how audio is represented, something you can’t do with an image.
- syntheweave 4y ago
  "Every aspect" would mean unlimited numbers of sound sources from all distances and directions, which is not how we represent sound, which is usually limited to two stereo channels at 44.1kHz and 16 bits in CD-quality audio, but with higher resolutions for surround or to reduce signal processing artifacts. An image is likewise limited by representation in that we define against a resolution, color depth, gamma, etc. Images define a large number of sample sources, but without variation over time. And in most circumstances, like with audio, we also currently limit the sources to fixed areas, rather than generating surround images. What we're relying on in using these formats successfully is knowing the limits of our sensory perception, and then what our brain does to process it. It says nothing about "A is easier to represent symbolically than B", because of course you can come up with symbolic compression schemes; those are how humans generate interesting images and audio. But the transmission protocol doesn't need to know them.

masswerk 4y ago

I think, these are all criteria of seamlessness, something, which an artificial composition must manage as well. Images, on the other hand lack the temporal dimension, variations in tempo, musical tones and scale, emotional impact, etc, which are crucial to seamlessness and continuity in music. The problem really arises with framing, composition, readable intent and motivation, i.e., general terms of expression – and image generation doesn't exactly shine here, as well. (The answer to any motivational questions will probably always be, "strongest signal". An artistic motivation will probably never be to just express the strongest signal in a common cultural context, rather, it will be contrast and interpretation. Otherwise, we should start worrying about that given society.)

woojoo666 4y ago

That complexity can actually make it more forgiving to mistakes. Visual art can contain lighting inconsistencies and still look fine. For example, a recent HN discussion looked at Dall-E Mini art involving origami [1], and while they look good, there are lots of clear mistakes. Perhaps in music, these mistakes are more obvious. Though at the same time, it seems like Dall-E2 is really good at faces [2], which is an example where there is a lot of structure, and viewers are very good at finding mistakes (the "uncanny valley"). Dall-E2 is of course a much bigger model with more training than Dall-E mini. Perhaps with the same amount of training, music AI can get to the same level. [1]: https://news.ycombinator.com/item?id=31787743 https://news.ycombinator.com/item?id=31787743 [2]: https://www.vice.com/en/article/g5vbx9/dall-e-is-now-generating-realistic-faces-of-fake-people https://www.vice.com/en/article/g5vbx9/dall-e-is-now-generat...

notahacker 4y ago

I don't think your analysis of my comment is valid. :) Regardless of opinion on audio files, it's the case that many AI approaches to music have focused on symbolic representations of music, which works, as do non-NN ways of procedurally generating music using the same symbols. What AI can theoretically add to a procedurally generated midi track is expressive performance, but that's not exactly its strength and probably needs better curation than just giving it lots of audio files. And even the "pure audio" approaches like OpenJukebox have to break out vocals and treat them separately (because it's something of an understatement to say that reflecting a static image is not a more complex problem than generating four minutes of singing with appropriate pitch, intonation and emotion and the impression of a consistent genre-appropriate vocalist), and the background music is... kind of muddy sounding and compositionally uneven. The main thrust of my argument wasn't the complexity of music though, the main thrust is comparative difficulty of compiling the training sets. More work to annotate and more licensing complications than just scooping up public domain images especially if you want a well rounded model. For another example take OpenAI's jukebox. It's clear the "Heavy Metal In The Style of Rage" vocals use a model trained on metal bands, but it's also clear that Rage Against The Machine wasn't prominent amongst them. Also probably the case there's simply not enough Rage Against The Machine albums to get your digital Zach de la Rocha vocals out of the uncanny valley even with the relevant permissions

xodjmk 4y ago

An image has none of these things. That is all put together in a person's mind psychologically, and what neural nets are trained to infer. In a 2D image there is no difference between actual shadows, and for example some flat picture of different colored shapes. Images are very simple. There is very little correlation past a few pixels in any direction, which is why most image filter are only a few pixels wide.