6 ms·

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius. I can't wait to hear some serious AI music-making a few years from now.

josalhor 4y ago
Makes me wonder if we will see a generalization of this idea. Just like in a CPU 90%+ of want you want to do can be modeled with very few instructions (mov, add, jmp..) we could see a set of very refined models (Stable difussion, GPT, etc) and all of their abstractions on top (ChatGPT, Rifussion, etc).
- DarmokJalad1701 4y ago
  Maybe next up is a model that generates Piet code https://www.dangermouse.net/esoteric/piet.html https://www.dangermouse.net/esoteric/piet.html
  - chii 4y ago
    and you ask stable diffusion to generate piet code for a slightly better version of stable diffusion (or chatGPT) ...which then you can further use to generate a better version, and so on. Singularity here we come!
- danuker 4y ago
  Indeed, I think this would be a cost-effective way to go forward.
- amelius 4y ago
  Perhaps GPT could run on top of Stable-diffusion, generating output in the form of written text (glyphs).
Tenoke 4y ago
For what is worth, people were trying the same thing with GANs (I also played with doing it with stylegan a bit) but the results weren't as good. The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.
rco8786 4y ago
> I can't wait to hear some serious AI music-making a few years from now. I think this will be particularly useful for musical compositions in movies and film, where the producer can "instruct" the AI about what to play, when, and how to transition so that the music matches the scene progression.
- adamhp 4y ago
  Not only that but sampling. I'd say there's at least one sample from something in most modern music. This can essentially create "sounds" that you're looking for as an artist. I need a sort of high pitched drone here... Rather than dig through sample libraries you just generate a few dozen results from a diffusion model with some varying inputs and you'd have a small sample set on the exact thing you're looking for. There's already so much processing of samples after the fact, the actual quality or resolution of the sample is inconsequential. In a lot of music, you're just going after the texture and tonality and timbre of something... This can be seen in some Hans Zimmer videos of how he slows down certain sounds massively to arrive at new sounds... or in granular synthesis... This is going to open up a lot of cool new doors.
- tstrimple 4y ago
  I was thinking gaming where music can and should dynamically shift based on different environmental and player conditions.
sebzim4500 4y ago
I suspect that if you had tried this with previous image models the results would have been terrible. This only works since image models are so good now.
superpope99 4y ago
I'm super excited about the Audio AI space, as it seems permanently a few years behind image stuff - so I think we're going to see a lot more of this. If you're interested, the idea of applying Image processing techniques to Spectrograms of audio is explored in brief in the first lesson of one of the most recommended AI courses on HN: Practical Deep Learning for Coders https://youtu.be/8SF_h3xF3cE?t=1632 https://youtu.be/8SF_h3xF3cE?t=1632
hackernewds 4y ago
You already hear a ton of them. Lofi music on these massively popular channels are basically auto-generated "music" + auto generated artwork.
- cjtrowbridge 4y ago
  Do you have any sources for more information about this?
- heavyarms 4y ago
  I dabble in music production and know some of the people in the "Lofi" world, so I know for a fact that this is not true. It's just a formulaic sub-genre where people are trying to make similar instrumentals with the same vibe. It would be jarring to listen to a playlist while studying and each song had wildly different tempos, instruments, etc. Also, the music doesn't sound "Lofi" because it's generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.
dangom 4y ago
This idea is presented by Jeremy Howard on literally their first Deep Learning for Coders class (most recent edition). A student wanted to classify sounds, but only knew how to do vision, so they converted sounds to spectrograms, fine tuned the model on the labelled spectra, and the classification worked pretty well on test data. That of course does not take the merit away from the Riffusion authors though.
- Analog24 4y ago
  The idea of connecting CV to audio via spectrograms pre dates Jeremy Howard's course by quite a bit. That's not really the interesting part here though. The fact that a simple extension of an image generation pipeline produces such impressive results with generative audio is what is interesting. It really emphasizes how useful the idea of stable diffusion is. edit: added a bit more to the thought
- aix1 4y ago
  The idea to apply computer vision algorithms to spectrograms is not new. I don't know who first came up with it, but I first came across it about a decade ago. I just ran a quick Google Scholar search, and the first result is https://ieeexplore.ieee.org/abstract/document/5672395 https://ieeexplore.ieee.org/abstract/document/5672395 This is from 2010. I didn't go looking, but it wouldn't surprise me if the idea is older than that.
  - hecanjog 4y ago
    There were a number of systems designed for composers in the 90s (also continuing through to today) designed for the workflow of converting a sound to a spectrogram, doing visual processing on the image, and then re-synthesizing the sound from the altered spectrogram. Many were inspired by Xenakis' UPIC system which was designed around the second half of this workflow: you'd draw the spectrogram with a pen and then synthesize it. https://en.wikipedia.org/wiki/UPIC https://en.wikipedia.org/wiki/UPIC Edit: my favorite of all these systems was Chris Penrose's HyperUPIC which provided a lot of freedom in configuring how the analysis and synthesis steps worked.
- deleted 4y ago
munificent 4y ago
As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to be poorly suited for?
- bheadmaster 4y ago
  As someone who loves making music and loves listening to music (regardless of its origins, in my case), it doesn't make me that sad. Sure, at first, I had an uncomfortable feeling that AI could make this sacred magic thing that only I and other fellow humans know how to do... But then I realized same thing is happening with visual art, so I applied the same counterarguments that've been cooking in my head. I think that kind of attitude is defeatist - it's implying that humans will be stopped from making music if AI learns how to do it too. I don't think that will happen. Humans will continue making music, as they always have. When Kraftwerk started using computers to make music back in the 70s, people were also scared of what that will do to musicians. To be fair, live music has died out a bit (in a sense that there aren't that many god-on-earth-level rockstars), but it's still out there, people are performing, and others who want to listen can go and listen. Maybe consumers will start consuming more and more AI music, instead of human music [0], but the worst thing that can happen is that music will no longer be a profitable activity. But then again, today's music industry already has some elements of the automation - washed-out rhythms, sexual thematics over and over again, re-hashing same old songs in different packages... So nothing's gonna change in the grand scheme of things. [0] https://www.youtube.com/watch?v=S1jWdeRKvvk https://www.youtube.com/watch?v=S1jWdeRKvvk
  - munificent 4y ago
    > but the worst thing that can happen is that music will no longer be a profitable activity. For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected to machines and not other people.

josalhor 4y ago

Makes me wonder if we will see a generalization of this idea. Just like in a CPU 90%+ of want you want to do can be modeled with very few instructions (mov, add, jmp..) we could see a set of very refined models (Stable difussion, GPT, etc) and all of their abstractions on top (ChatGPT, Rifussion, etc).

DarmokJalad1701 4y ago
Maybe next up is a model that generates Piet code https://www.dangermouse.net/esoteric/piet.html https://www.dangermouse.net/esoteric/piet.html
- chii 4y ago
  and you ask stable diffusion to generate piet code for a slightly better version of stable diffusion (or chatGPT) ...which then you can further use to generate a better version, and so on. Singularity here we come!
danuker 4y ago
Indeed, I think this would be a cost-effective way to go forward.
amelius 4y ago
Perhaps GPT could run on top of Stable-diffusion, generating output in the form of written text (glyphs).

Tenoke 4y ago

For what is worth, people were trying the same thing with GANs (I also played with doing it with stylegan a bit) but the results weren't as good. The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.

rco8786 4y ago

> I can't wait to hear some serious AI music-making a few years from now. I think this will be particularly useful for musical compositions in movies and film, where the producer can "instruct" the AI about what to play, when, and how to transition so that the music matches the scene progression.

adamhp 4y ago
Not only that but sampling. I'd say there's at least one sample from something in most modern music. This can essentially create "sounds" that you're looking for as an artist. I need a sort of high pitched drone here... Rather than dig through sample libraries you just generate a few dozen results from a diffusion model with some varying inputs and you'd have a small sample set on the exact thing you're looking for. There's already so much processing of samples after the fact, the actual quality or resolution of the sample is inconsequential. In a lot of music, you're just going after the texture and tonality and timbre of something... This can be seen in some Hans Zimmer videos of how he slows down certain sounds massively to arrive at new sounds... or in granular synthesis... This is going to open up a lot of cool new doors.
tstrimple 4y ago
I was thinking gaming where music can and should dynamically shift based on different environmental and player conditions.

sebzim4500 4y ago

I suspect that if you had tried this with previous image models the results would have been terrible. This only works since image models are so good now.

superpope99 4y ago

I'm super excited about the Audio AI space, as it seems permanently a few years behind image stuff - so I think we're going to see a lot more of this. If you're interested, the idea of applying Image processing techniques to Spectrograms of audio is explored in brief in the first lesson of one of the most recommended AI courses on HN: Practical Deep Learning for Coders https://youtu.be/8SF_h3xF3cE?t=1632 https://youtu.be/8SF_h3xF3cE?t=1632

hackernewds 4y ago

You already hear a ton of them. Lofi music on these massively popular channels are basically auto-generated "music" + auto generated artwork.

cjtrowbridge 4y ago
Do you have any sources for more information about this?
heavyarms 4y ago
I dabble in music production and know some of the people in the "Lofi" world, so I know for a fact that this is not true. It's just a formulaic sub-genre where people are trying to make similar instrumentals with the same vibe. It would be jarring to listen to a playlist while studying and each song had wildly different tempos, instruments, etc. Also, the music doesn't sound "Lofi" because it's generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.

dangom 4y ago

This idea is presented by Jeremy Howard on literally their first Deep Learning for Coders class (most recent edition). A student wanted to classify sounds, but only knew how to do vision, so they converted sounds to spectrograms, fine tuned the model on the labelled spectra, and the classification worked pretty well on test data. That of course does not take the merit away from the Riffusion authors though.

Analog24 4y ago
The idea of connecting CV to audio via spectrograms pre dates Jeremy Howard's course by quite a bit. That's not really the interesting part here though. The fact that a simple extension of an image generation pipeline produces such impressive results with generative audio is what is interesting. It really emphasizes how useful the idea of stable diffusion is. edit: added a bit more to the thought
aix1 4y ago
The idea to apply computer vision algorithms to spectrograms is not new. I don't know who first came up with it, but I first came across it about a decade ago. I just ran a quick Google Scholar search, and the first result is https://ieeexplore.ieee.org/abstract/document/5672395 https://ieeexplore.ieee.org/abstract/document/5672395 This is from 2010. I didn't go looking, but it wouldn't surprise me if the idea is older than that.
- hecanjog 4y ago
  There were a number of systems designed for composers in the 90s (also continuing through to today) designed for the workflow of converting a sound to a spectrogram, doing visual processing on the image, and then re-synthesizing the sound from the altered spectrogram. Many were inspired by Xenakis' UPIC system which was designed around the second half of this workflow: you'd draw the spectrogram with a pen and then synthesize it. https://en.wikipedia.org/wiki/UPIC https://en.wikipedia.org/wiki/UPIC Edit: my favorite of all these systems was Chris Penrose's HyperUPIC which provided a lot of freedom in configuring how the analysis and synthesis steps worked.
deleted 4y ago

munificent 4y ago

As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad. Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to be poorly suited for?

bheadmaster 4y ago
As someone who loves making music and loves listening to music (regardless of its origins, in my case), it doesn't make me that sad. Sure, at first, I had an uncomfortable feeling that AI could make this sacred magic thing that only I and other fellow humans know how to do... But then I realized same thing is happening with visual art, so I applied the same counterarguments that've been cooking in my head. I think that kind of attitude is defeatist - it's implying that humans will be stopped from making music if AI learns how to do it too. I don't think that will happen. Humans will continue making music, as they always have. When Kraftwerk started using computers to make music back in the 70s, people were also scared of what that will do to musicians. To be fair, live music has died out a bit (in a sense that there aren't that many god-on-earth-level rockstars), but it's still out there, people are performing, and others who want to listen can go and listen. Maybe consumers will start consuming more and more AI music, instead of human music [0], but the worst thing that can happen is that music will no longer be a profitable activity. But then again, today's music industry already has some elements of the automation - washed-out rhythms, sexual thematics over and over again, re-hashing same old songs in different packages... So nothing's gonna change in the grand scheme of things. [0] https://www.youtube.com/watch?v=S1jWdeRKvvk https://www.youtube.com/watch?v=S1jWdeRKvvk
- munificent 4y ago
  > but the worst thing that can happen is that music will no longer be a profitable activity. For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected to machines and not other people.