6 ms·
Generating music in the waveform domain (2020)
- tgv 3y agoIsn't that usually called the time domain?
- DiogenesKynikos 3y agoThe author makes a distinction between two different modeling approaches: 1. Representing music as a series of notes (with additional information about dynamics, etc.), and then building a model that transforms this musical score into sound. 2. Modeling the waveform itself directly, without reference to a score. This waveform could presumably be in either the frequency or the time domain, but the author chooses to use the time domain. The author's terminology is a bit confusing, but I think that they mean option 2 when they say "the waveform domain."
- earthnail 3y agoThe frequency domain is a misleading name. I assume you’re referring to STFT, or spectrogram, which is a series of windowed time segments transformed into the frequency domain. But both you and OP are right in that waveform domain is usually called time domain.
- munificent 3y agoThe two piano examples where the second one has the phase randomized is also an excellent example of why allpass filters, which change the phase but not amplitude of all frequencies, are a building block for digital reverbs. The second piano example with the randomized phases sounds more blurred out and almost reverb-y.
- hammock 3y ago>change the phase but not amplitude of all frequencies Kind of what walls do as well so it makes sense
- anigbrowl 3y agoWhy do people keep doing this? Musicians who want an accompanist/virtual producer still want control over the orchestration, tonality, and shaping of sounds. Even karaoke machines use a signal pipeline to blend the singer's voice with the backing track. Generating finished waveforms is only good for elevator music.
- kastnerkyle 3y agoResearch into "pure" unconditional generation can often lead to gains in the conditional setting. See literally any GAN research, VQ-VAE, VAE, diffusion, etc - all started from "unconditional/low information" pretty much. Both directly (in terms of modeling) and indirectly (by forcing you to really reason about what conditioning is telling you about the modeling, and what's in the data), these approaches really force you to think about what it means to just "make music". Also, I think artistic uses (such as Dadabots, who heavily used SampleRNN) show clearly that "musicians" like interesting tools, even if uncontrolled in some cases. Tools to exactly execute an idea are important (DAW-like), but so are novelty generating machines like (many) unconditional generators end up being. Jukebox is another nice example of this. On the "good for elevator music" comment - the stuff I've heard from these models is rarely relaxing enough to be in any elevator I would ride. But there are snippets of inspiration in there for sure. Generally, I do favor controllable models with lots of input knobs and conditioning for direct use, but there's space for many different approaches in pushing the research forward. Different creators will work all kind of odd models into their workflows, even things that are objectively less "high quality", and not really controllable. To me, that's a great thing and reason enough to keep pushing unsupervised learning forward.
- alexahn 3y agoSeems like a reasonable way to try to design an AGI. Maybe the real Turing test is whether an intelligent system enjoys and seeks out novel music.
- p1esk 3y agoA lot of humans would fail such a test.
- blt 3y agoPeople working on this problem: have diffusion models taken off in this field too?
- p1esk 3y agoYes: https://github.com/riffusion/riffusion https://github.com/riffusion/riffusion
- brcmthrowaway 3y agoAny dummies explanation of what diffusion is?
- benanne 3y agoI've since moved on to work primarily on diffusion models, so I have a series of blog posts about that topic as well! - https://sander.ai/2022/01/31/diffusion.html https://sander.ai/2022/01/31/diffusion.html is about the link between diffusion models and denoising autoencoders, IMO the easiest to understand out of all interpretations; - https://sander.ai/2023/07/20/perspectives.html https://sander.ai/2023/07/20/perspectives.html covers a slew of different perspectives on diffusion models (including the "autoencoder" one). In a nutshell, diffusion models break up the difficult task of generating natural signals (such as images or sound) into many smaller partial denoising tasks. This is done by defining a corruption process that gradually adds noise to an input until all of the signal is drowned out (this is the "diffusion"), and then learning how to invert that process step-by-step. This is not dissimilar to how modern language models work: they break up the task of generating text into a series of easier next-word-prediction tasks. In both cases, the model only solves a small part of the problem at a time, and you apply it repeatedly to generate a signal.
- zaptrem 3y agoyeah, we're training a diffusion model for this here: https://sonauto.ai/ https://sonauto.ai/ example 70s dolly parton song made by a rando earlier today: https://sonauto.ai/songs/t4XGcHqsYwS6B73MjtV5 https://sonauto.ai/songs/t4XGcHqsYwS6B73MjtV5
- albertzeyer 3y agoSince 2020, there are a number of new models, for example: - Stable Audio: https://stability-ai.github.io/stable-audio-demo/ https://stability-ai.github.io/stable-audio-demo/ https://www.stableaudio.com/ https://www.stableaudio.com/ - MusicGen: https://ai.honu.io/papers/musicgen/ https://ai.honu.io/papers/musicgen/ - MusicLM: https://google-research.github.io/seanet/musiclm/examples/ https://google-research.github.io/seanet/musiclm/examples/
- p1esk 3y agoBut none of these is a significant improvement over Jukebox. I think at this point everyone is waiting for Jukebox2.
- GaggiX 3y agoSunoAI is very good in my opinion.
- p1esk 3y agoI tried a dozen of different prompts in suno ai to generate some music - it completely ignored them. It just generated some simple pop sounding tunes every time. I’m not impressed.
- zaptrem 3y agotry this one my friend and I are training: https://sonauto.ai/ https://sonauto.ai/ audio quality is still WIP and UI is ugly but prompting should allow more detail. e.g., 70s Dolly Parton-style song rando made earlier today: https://sonauto.ai/songs/t4XGcHqsYwS6B73MjtV5 https://sonauto.ai/songs/t4XGcHqsYwS6B73MjtV5
- bradgpt 3y agothis is cool! i've played with stable audio a little but it doesn't do lyrics, how are you getting the vocals so good?
- gorkish 3y agoFor any other DSP people tearing their hair out at the author's liberal terminology, whenever you see "wave" or "waveform" the author is talking about non-quadrature time domain samples. I feel this work would be a lot better if it was built around a more foundational understanding of DSP before unleashing the neural nets on data that is arguably the worst possible representation of signal state. But then again there are people training on the raw bitstream from a CCD camera instead of image sequences so maybe hoping for an intermediate format that can be understood by humans has no place in the future!
- kajecounterhack 3y agoAside from using "waveform" to mean "time domain" the terminology the author used in the blog post is consistent with what I've seen in audio ML research. In what ways would you suggest improving the representation of the signal?
- jabagonuts 3y agoAre you a musician? Have you ever used DAW like Cubase or Pro Tools? If not, have you ever tried the FOSS (GPLv3) Audacity audio editor [1]. Waves and Waveforms are colloquial terminology, so the terms are familiar to anyone in the industry as well as your average hobbyist. Additionally, PCM [2] is at the heart of many of these tools, and is what is converted between digital and analog for real-world use cases. This is literally how the ear works [3], so before arguing that this is the "worst possible representation of signal state," try listening to the sounds around you and think about how it is that you can perceive them. [1] https://manual.audacityteam.org/man/audacity_waveform.html https://manual.audacityteam.org/man/audacity_waveform.html [2] https://en.wikipedia.org/wiki/Pulse-code_modulation https://en.wikipedia.org/wiki/Pulse-code_modulation [3] https://www.nidcd.nih.gov/health/how-do-we-hear https://www.nidcd.nih.gov/health/how-do-we-hear
- gorkish 3y agoYeah, except no; the ear works by having cillia that each resonate at different frequencies, differentiated into a log-periodic type response. It is mostly a "frequency domain" mechanism though in the real world the time component is obviously necessary to manifest frequency. If we want to have a debate about how best to call it, the closest term I might reach for from the quite often mislabeled vernacular of the music/production/audio world would be "grains" / granular synthesis. WRT the waveform tool in DAWs you should be aware that it doesnt normally work like you may assume it does. If you start dragging points around in there you typically are not actually doing raw edits to the time domain samples but having your edits applied through a filter that tries to minimize ringing and noise. That is to say the DAW will typically not just let you move a sample to any value you wish. In this case the tool is bending to its use as an audio editor and not defaulting to behavior that would otherwise just introduce clicks and pops every time it was used. I stand by my argument that the author's terminology appears ignorant in an area where it ought to be very deliberately specific. I question the applicability and relevance of the work beginning at that point, even though the approach may have yielded a useful result.