3 ms·

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduc

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour full of AI mistakes is really tiring. With a symbolic representation you could at least fix the mistakes if there is one good output.

dimatura 4y ago
I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite competent, especially in more constrained domains like NES chiptunes or classical piano, so a full text-to-audio pipeline probably seemed a more interesting research problem. And for a lot of use cases, where people might truly not care too much about tweaking the output to their liking, the generated audio might be good enough; the examples were pretty plausible to my ear, if somewhat lo-fi sounding (probably because it's operating at 24kHz, compared to the more standard 44-48kHz). In the future a more hybrid approach probably makes sense for at least some applications, where MIDI is generated along with some way of specifying the timbre for each instrument (hopefully something better than general MIDI, though even that would be fun; not sure if it's been done). I'm sure that in the near we'll see a lot more work in the DAW and plugin space to have these kind of things built-in, but in a way that they can be edited by the user.
- westurner 4y ago
  awesome-sheet-music lists a number of sheet music archives https://github.com/ad-si/awesome-sheet-music https://github.com/ad-si/awesome-sheet-music Other libraries of (Royalty Free, Public Domain) sheet music: - https://musopen.org/ https://musopen.org/ Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli... FWIU, Current LLMs can't yet do explainable AI well enough to satisfy the optional Attribution clause of e.g. Creative Commons licenses? "Sufficiently Transformative" is the current general copyright burden according to precedent; Transformative use and fair use: https://en.wikipedia.org/wiki/Transformative_use https://en.wikipedia.org/wiki/Transformative_use
- TylerE 4y ago
  If it could output one track per instrument (aka stems) that’s probably be the most useful thing.
- Lramseyer 4y ago
  I would offer a counter point: Chat GPT uses text, which is analogous to the MIDI of spoken language. Now, what's to prevent a speech to text and text to speech AI slapped on both ends to make it audibly conversational? Or train the franken-model model end to end in one monolithic model in hopes that it might make other interesting connections. That being said I have seen very little in the AI transcription space (Listening to music, and outputting legible sheet music while identifying instruments) especially when it comes to multi-instrument music. I've also seen very little in the midi generation space aside from what you mentioned. It would be nice to see midi to midi AI remixes, analogous to how Chat GPT can (sort of) turn wikipedia into rap lyrics.
  - dimatura 4y ago
    Oh, there's already all sorts of work mashing together GPT text with text-to-speech and speech-to-text. There were some viral videos featuring "Steve Jobs" talking about AI and one being interviewed by "Joe Rogan". I also saw somewhere here on HN a website featuring an "infinite" conversation between "Zizek" and "Herzog", in a similar vein. Though something end-to-end would probably make more sense; text is a lossy way to encode speech, as it doesn't capture a lot of nuance that goes into spoken words. Interestingly, the same could be said of sheet music and even MIDI in the music realm. As for transcription for multi-instrument music, there's been a lot of work on that front, and a lot of progress that's already finding its way into commercial products. The latest I've seen in this domain is https://samplab.com/ https://samplab.com/, which has some compelling videos of not only transcribing multi-instrument audio but editing it on a note-by-note basis.
- woolion 4y ago
  The difference is that for images almost everything is done with raster graphics, vector is the niche exception. And for most uses where it makes sense (e.g. a logo) you can easily convert a raster to SVG automatically generally without requiring any manual tweaking. If the argument is that "maximize immediate practical usefulness" is not something interesting, I would say it makes sense to say "I don't understand why this approach is pushed". Anyway.
snowram 4y ago
You can absolutely "overpaint" sound, as sound can indeed be represented as an image and be extended in frequency of in time. In fact this is exactly what Riffusion is doing.
- woolion 4y ago
  You sure _can_, but that's not how a music producer works. I meant it as how you could actually use the output into a professional workflow. For images it's definitely easy since it integrates in a meaningful way into the pipeline. And in many cases it improves over photobashing. In music, 'fixing' problems in sound is way more costly, so a producer would say that it's not something you can work with in any reasonable manner.
  - williamcotton 4y ago
    Oooh, using LLMs for "inpainting" audio to remove pops from a vocal take! Or to fix a late snare hit! Of course these tasks can be done right now they just take minutes instead of what could be seconds.
jameshart 4y ago
“For a song you need midi” Pretty sure the Beatles never handed George Martin any midi files. What’s the symbolic representation that captures the tone of every bend in a Hendrix solo? Did Daft Punk go back and grab the raw master stems of the old vinyl recordings they used to assemble their tracks? Music producers have been astonishingly creative given inputs in a vast range of formats. Sheet music and midi are one tool, but ultimately it’s about combining sounds in the mix isn’t it?
- woolion 4y ago
  Why do you strawman my argument? The issue with combining sounds in the mix is the issues add up in a super-linear way. Because of that you could even say that modern music producers tend to care too much about perfect sounds. The Beatles did not have the same production requirement modern music have since there is high-fidelity equipment. The music industry as a whole changed a lot. Modern music production have multiple passes just to clean up one "perfect vocal take", like de-essing, breath removal, etc. This is insanely more advanced than what was the standard in the 60s. I'm not saying it's better either. >What’s the symbolic representation that captures the tone of every bend in a Hendrix solo? That's not how music works. You refer to the score, and the precise way you bend your tones is up to your interpretation. Even if you wanted to copy one-to-one Hendrix, you'd copy the style, not one performance exactly, so capturing every little bend does not really matter. Which could be an AI project: Hendrixify a guitar track.
  - jameshart 4y ago
    Pretty sure the only score Jimi ever referred to was scoring some weed. My point is that performances are as valid an input to music production as anything else. And that these generative snippets are potentially a source of ‘performances’ that producers will tap in creative and interesting ways.

dimatura 4y ago

I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite competent, especially in more constrained domains like NES chiptunes or classical piano, so a full text-to-audio pipeline probably seemed a more interesting research problem. And for a lot of use cases, where people might truly not care too much about tweaking the output to their liking, the generated audio might be good enough; the examples were pretty plausible to my ear, if somewhat lo-fi sounding (probably because it's operating at 24kHz, compared to the more standard 44-48kHz). In the future a more hybrid approach probably makes sense for at least some applications, where MIDI is generated along with some way of specifying the timbre for each instrument (hopefully something better than general MIDI, though even that would be fun; not sure if it's been done). I'm sure that in the near we'll see a lot more work in the DAW and plugin space to have these kind of things built-in, but in a way that they can be edited by the user.

westurner 4y ago
awesome-sheet-music lists a number of sheet music archives https://github.com/ad-si/awesome-sheet-music https://github.com/ad-si/awesome-sheet-music Other libraries of (Royalty Free, Public Domain) sheet music: - https://musopen.org/ https://musopen.org/ Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli... FWIU, Current LLMs can't yet do explainable AI well enough to satisfy the optional Attribution clause of e.g. Creative Commons licenses? "Sufficiently Transformative" is the current general copyright burden according to precedent; Transformative use and fair use: https://en.wikipedia.org/wiki/Transformative_use https://en.wikipedia.org/wiki/Transformative_use
TylerE 4y ago
If it could output one track per instrument (aka stems) that’s probably be the most useful thing.
Lramseyer 4y ago
I would offer a counter point: Chat GPT uses text, which is analogous to the MIDI of spoken language. Now, what's to prevent a speech to text and text to speech AI slapped on both ends to make it audibly conversational? Or train the franken-model model end to end in one monolithic model in hopes that it might make other interesting connections. That being said I have seen very little in the AI transcription space (Listening to music, and outputting legible sheet music while identifying instruments) especially when it comes to multi-instrument music. I've also seen very little in the midi generation space aside from what you mentioned. It would be nice to see midi to midi AI remixes, analogous to how Chat GPT can (sort of) turn wikipedia into rap lyrics.
- dimatura 4y ago
  Oh, there's already all sorts of work mashing together GPT text with text-to-speech and speech-to-text. There were some viral videos featuring "Steve Jobs" talking about AI and one being interviewed by "Joe Rogan". I also saw somewhere here on HN a website featuring an "infinite" conversation between "Zizek" and "Herzog", in a similar vein. Though something end-to-end would probably make more sense; text is a lossy way to encode speech, as it doesn't capture a lot of nuance that goes into spoken words. Interestingly, the same could be said of sheet music and even MIDI in the music realm. As for transcription for multi-instrument music, there's been a lot of work on that front, and a lot of progress that's already finding its way into commercial products. The latest I've seen in this domain is https://samplab.com/ https://samplab.com/, which has some compelling videos of not only transcribing multi-instrument audio but editing it on a note-by-note basis.
woolion 4y ago
The difference is that for images almost everything is done with raster graphics, vector is the niche exception. And for most uses where it makes sense (e.g. a logo) you can easily convert a raster to SVG automatically generally without requiring any manual tweaking. If the argument is that "maximize immediate practical usefulness" is not something interesting, I would say it makes sense to say "I don't understand why this approach is pushed". Anyway.

snowram 4y ago

You can absolutely "overpaint" sound, as sound can indeed be represented as an image and be extended in frequency of in time. In fact this is exactly what Riffusion is doing.

woolion 4y ago
You sure _can_, but that's not how a music producer works. I meant it as how you could actually use the output into a professional workflow. For images it's definitely easy since it integrates in a meaningful way into the pipeline. And in many cases it improves over photobashing. In music, 'fixing' problems in sound is way more costly, so a producer would say that it's not something you can work with in any reasonable manner.
- williamcotton 4y ago
  Oooh, using LLMs for "inpainting" audio to remove pops from a vocal take! Or to fix a late snare hit! Of course these tasks can be done right now they just take minutes instead of what could be seconds.

jameshart 4y ago

“For a song you need midi” Pretty sure the Beatles never handed George Martin any midi files. What’s the symbolic representation that captures the tone of every bend in a Hendrix solo? Did Daft Punk go back and grab the raw master stems of the old vinyl recordings they used to assemble their tracks? Music producers have been astonishingly creative given inputs in a vast range of formats. Sheet music and midi are one tool, but ultimately it’s about combining sounds in the mix isn’t it?

woolion 4y ago
Why do you strawman my argument? The issue with combining sounds in the mix is the issues add up in a super-linear way. Because of that you could even say that modern music producers tend to care too much about perfect sounds. The Beatles did not have the same production requirement modern music have since there is high-fidelity equipment. The music industry as a whole changed a lot. Modern music production have multiple passes just to clean up one "perfect vocal take", like de-essing, breath removal, etc. This is insanely more advanced than what was the standard in the 60s. I'm not saying it's better either. >What’s the symbolic representation that captures the tone of every bend in a Hendrix solo? That's not how music works. You refer to the score, and the precise way you bend your tones is up to your interpretation. Even if you wanted to copy one-to-one Hendrix, you'd copy the style, not one performance exactly, so capturing every little bend does not really matter. Which could be an AI project: Hendrixify a guitar track.
- jameshart 4y ago
  Pretty sure the only score Jimi ever referred to was scoring some weed. My point is that performances are as valid an input to music production as anything else. And that these generative snippets are potentially a source of ‘performances’ that producers will tap in creative and interesting ways.