13 ms·
Show HN: Cloning a musical instrument from 16 seconds of audio
In 2020, Magenta released DDSP [1], a machine learning algorithm / python library which made it possible to generate good sounding instrument synthesizers from about 6-10 minutes of data. While working with DDSP for a project, we realised how
it was actually quite hard to find 6-10 minute of clean recordings of monophonic instruments.
In this project, we have combined the DDSP architecture with a domain adaptation technique from speech synthesis [2]. This domain adaptation technique works by pre-training our model on many different recordings from the Solos dataset [3] first and then fine-tuning parts of the model to the new recording. This allows us to produce decent sounding instrument synthesisers from as little as 16 seconds of target audio instead of 6-10 minutes.
[1] https://arxiv.org/abs/2001.04643 https://arxiv.org/abs/2001.04643
[2] https://arxiv.org/abs/1802.06006 https://arxiv.org/abs/1802.06006
[3] https://arxiv.org/abs/2006.07931 https://arxiv.org/abs/2006.07931
We hope to publish a paper on the topic soon.
- deleted 5y ago[deleted]
- smaddox 5y agoVery cool. I wonder if you could turn something like this into the ultimate keyboard synth. That would be one hell of a product.
- abdljasser2 5y agoCopy pasting my a reply to a similar question Short answer is yes! Previous work has shown that we can obtain very good results from controlling DDSP models from midi input. The solutions I am familiar with employ a two stage approach where the first stage takes midi and turns it into control signals (pitch & loudness contours etc..) and the second stage turns the controls signals into audio (like the particular model I discuss in the blog post)[1][2][3]. I actually think that the first stage could also benefit from the transfer learning techniques we discuss in the blogpost. In terms of actually releasing a MIDI playable VST plugin I believe that Magenta have something like it in the works[4]. I hope that it will come with some ability for users to quickly create their own instruments, presumably using a transfer learning technique similar to the one we have presented. Real-time rendering poses multiple challenges. For one, some instrument sounds occur before a note properly onsets (for example the sound of the fingers pressing the keys of a saxophone occurs before the first note of the piece). Secondly, the research models are quite heavy and considerably more compute intensive than a standard VST instrument which poses a problem if you want to use it inside a DAW. I think this latter problem can be solved with some clever engineering and the general trend of hardware being more and more accommodating to machine learning applications. [1] https://erl-j.github.io/controlsynthesis/#/ https://erl-j.github.io/controlsynthesis/#/ (Our previous work) [2] https://rodrigo-castellon.github.io/midi2params/ https://rodrigo-castellon.github.io/midi2params/ (Focuses on realtime rendering) [3] https://arxiv.org/abs/2112.09312 https://arxiv.org/abs/2112.09312 (Magenta's recent paper on the subject)
- abdljasser2 5y ago[4] https://forum.juce.com/t/ddsp-tone-transfer-vst-possibility/46155/8 https://forum.juce.com/t/ddsp-tone-transfer-vst-possibility/...
- skybrian 5y agoVery interesting! Is there a preprint or demo that you forgot to link to? Will you be releasing source or data? I'm wondering what the hardware requirements would be for real-time synthesis. I work on a musical instrument project as a hobby and would like a good accordion sound.
- abdljasser2 5y agoHello! Thank you for the interest. There are links to a colab etc at the bottom of the blog post!
- michae2 5y agoWow, this seems like it could be bigger than Auto-Tune. I wonder if we'll reach a point where artists license a DDSP model of their instrument or voice, rather than perform directly.
- birdyrooster 5y agoWell, I suppose a form of this technically has already happened. Sakiko Fujita is the voice actor who was employed to create the samples for the Vocaloid Hatsune Miku. The copyright holder currently is SEGA.
- MrTrvp 5y agohttps://www.youtube.com/watch?v=Eiwzx9hxUe0 https://www.youtube.com/watch?v=Eiwzx9hxUe0 Thought this was interesting enough to share
- ajross 5y agoWhile it's doing a great job emulating the timbre of the notes, what's interesting to me is what it's not doing. Play those top two passages to anyone who's spent time around sax players, and they'll instantly tell you which one was the "fake". Real instruments have mechanical behavior this isn't going to see absent some kind of modeling. Different notes on a sax (to take this example) are actuated by different fingers and different valves and have different (often multiple) embouchures, and both interact with the transitions between pairs of notes (and with the dynamics with which they're played). All that complexity is absolutely hearable in the transitions between notes, and the ML layers aren't going to have the ability to pick it up absent a much larger training set. Basically: 16 seconds of audio is enough to get you the frequency spectrum of the notes, which you can do with 3-4 lines of synthesis code in an imperative regime. It's very much the "easy part" of instrument synthesis.
- davio 5y agoI played sax through college and the second passage sounds legit to me. Couple of interesting things: - You can hear the keys and pads hitting and clicking - Vibrato on the last note is very realistic - Can hear the air and "spit" - The timing seems human, especially as it slows down a little on the lower notes which take more air and usually involve more awkward fingering with the pinky fingers. Definitely sounds like a non-professional, probably how an average high school player would play the passage. It sounds like what a player would hear versus a studio recording. Reminds me of being in a really reflective & tiny practice room.
- LegitShady 5y agoir you read down the article the 16 seconds is just the input to a ml model. the actual training set is fifty something minutes.
- ushakov 5y agonice! check out “Steerable discovery of neural audio effects” paper https://csteinmetz1.github.io/steerable-nafx/ https://csteinmetz1.github.io/steerable-nafx/ i’m compiling a list of research papers for our GuitarML project, feel free to open a pull request/issue when your papers are published! https://github.com/GuitarML/mldsp-papers https://github.com/GuitarML/mldsp-papers
- abdljasser2 5y agoThank you! Will do.
- scrozier 5y agoBut if one is trying to create a great sounding instrument synthesizer, it is quite easy to obtain 6-10 minutes of clean recording. Why do you have to find it? Not sure what the use case is here...?
- abdljasser2 5y agoHello! Immediate use case would be sampling. Say you like a certain sound in a song and would like to use it as a starting point for your own sound patch. I also believe that transfer learning has benefits even for making great sounding instruments in cases where you have access to lots of data. That’s my intuition at least. At the very least, it saves you a lot of memory/bandwith. Instead of having one large model per instrument you only need one large models with a few extra instrument specific weights.
- samstave 5y agoRather than doing a melody, what happens when you train it with a bunch of single notes of the instrument for each note. Then give it a melody and have it play the melody based on its library of notes for a given interment?
- kristianbrigman 5y agothis is closer to what Redmatica (now Logic) AutoSampler used to do.... you would just play notes (separately by space), it would split by space, detect the root note for each sample, and then make a multisample out of it. but actually trying to reproduce a real instrument involves legato between notes, handling pitch and timbre changes as you play, etc. So a much harder problem.
- samstave 5y agoThanks for that info. Familiar with Logic, didnt know that history though. Thanks
- willis936 5y ago
- cellover 5y agoWhy in the name of God would a site prevent me from using shortcut nav to go back in nav history?! I can not use Cmd + left on a mac, is this a Notion thing?
- capableweb 5y agoI didn't understand what you meant first, clicking on my back arrow on the physical mouse, or the go back icon in the browser and Firefox would go back like expected. Then I tried Alt+Left arrow on the keyword, and it didn't work! But it wasn't just not going back, like when you have a "too many history.pushState()" bug that fucks it up for you. Instead it went to the bottom of the page. Then I noticed it was selecting some block in the bottom right and that it also happens when you just press left arrow on the keyboard without Alt. So TLDR: bug with some selection/focus thing on the Notion page. Poor execution, I rate their implementation of "static notebook HTML pages" 7/10.
- cellover 5y agoOh right I see, I did not have the time to debug when I noticed this issue, thanks for the heads up! Also sorry for parasiting the comments here, very interesting research! It would have been interesting to record original samples in the same settings; we can hear the flute having much less reverb than the 1st saxophone and it could be helpfup to have a common room size / reverb time to help comparison.
- dang 5y ago"Please don't complain about tangential annoyances—things like article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- nyanpasu64 5y agoDoes this work on synthesized sounds (like analog or physical modeling rather than samplers)? How does it handle the complexity of piano timbres (notoriously difficult to synthesize)?
- abdljasser2 5y agoGreat question! I think this approach (and DDSP more generally) could work well for most instruments if you make some adjustments to the synthesis architecture. The current architecture imposes some strong assumptions on the source (such as harmonicity) which does not necessarily apply to all instruments (bells for instance). The harmonic + noise model is not that good with transients either (which is a key ingredient in piano sounds) but of course you could solve that through different means.
- radarsat1 5y agonice, i worked on something similar some years ago but this seems more sophisticated. i had a lot of difficulty with atonal sounds especially because i was trying to take a kind of "neural wavetable" approach, but overall i found the results intriguing nonetheless. curious to check out more recent efforts. i recall the DDSP paper and assumed those ideas would apply quite nicely to this problem. [1] https://arxiv.org/abs/1806.09617 https://arxiv.org/abs/1806.09617
- abdljasser2 5y agoVery cool! Thank you for this. Speaking of wavetables, have you seen this relatively recent work? https://lamtharnhantrakul.github.io/diffwts.github.io/ https://lamtharnhantrakul.github.io/diffwts.github.io/?
- radarsat1 5y agooh nice, i have not. thanks i've got some catching up to do! i see that they used the nsynth dataset, i didn't realize it was publicly available. i recall that the nsynth paper came out just as i was finishing that work, you can imagine i felt a bit scooped ;). (but nsynth was far more impressive so what could i say..)
- skykooler 5y agoHave to say I'm disappointed with the chosen excerpts in the "More examples" section - they do not show very much of the generated instrument's ability (the last one is just a single note!)
- abdljasser2 5y agoThank you for the feedback! We will synthesize longer excerpts in the future. For the time being there is a colab where you can play with a pretrained model.
- squarefoot 5y agoAny chances this could turn into a MIDI playable instrument with some pre-trained models plus the possibility to submit user generated ones?
- abdljasser2 5y agoShort answer is yes! Previous work has shown that we can obtain very good results from controlling DDSP models from midi input. The solutions I am familiar with employ a two stage approach where the first stage takes midi and turns it into control signals (pitch & loudness contours etc..) and the second stage turns the controls signals into audio (like the particular model I discuss in the blog post)[1][2][3]. I actually think that the first stage could also benefit from the transfer learning techniques we discuss in the blogpost. In terms of actually releasing a MIDI playable VST plugin I believe that Magenta have something like it in the works[4]. I hope that it will come with some ability for users to quickly create their own instruments, presumably using a transfer learning technique similar to the one we have presented. Real-time rendering poses multiple challenges. For one, some instrument sounds occur before a note properly onsets (for example the sound of the fingers pressing the keys of a saxophone occurs before the first note of the piece). Secondly, the research models are quite heavy and considerably more compute intensive than a standard VST instrument which poses a problem if you want to use it inside a DAW. I think this latter problem can be solved with some clever engineering and the general trend of hardware being more and more accommodating to machine learning applications. [1] https://erl-j.github.io/controlsynthesis/#/ https://erl-j.github.io/controlsynthesis/#/ (Our previous work) [2] https://rodrigo-castellon.github.io/midi2params/ https://rodrigo-castellon.github.io/midi2params/ (Focuses on realtime rendering) [3] https://arxiv.org/abs/2112.09312 https://arxiv.org/abs/2112.09312 (Magenta's recent paper on the subject)
- mkr-hn 5y agoWhile interesting, I'm skeptical an algorithmic approach will ever come close to a decent wavetable-based synth (using samples/noises from the real instrument) or scripted sampler instrument (like Kontakt). It might help if you can't otherwise find an existing synth/sample-based instrument but can find it in use, but those are few and far between.
- uoaei 5y ago"ever"? I am not usually on the techno-optimist side of things but I have a hard time believing, after observing the progress of SOTA in the past decade on the big problems explored by ML research, that we are more than 5-10 years out from exceptional generative samplers based on tiny snippets of audio.
- mkr-hn 5y agoI don't know what SOTA is. Are we still on-topic here? The context is realistic instruments. Synths can already produce timbres pre-electronic instrument designers could only dream of.
- graphpercolator 5y agoI don't think this is even the problem though. We have amazing physical modeling synths but the problem is it is not "playable" with a keyboard in the sense of being able to control all the needed variables. You basically need a breathe controller to make a realistic playable sax sound otherwise you can't control the variables in real time that are needed. Personally, I want to hear synth sounds that are not even trying to be reproductions of acoustic instruments. If I want to hear a sax sound I want to hear someone who is really good on a real sax and who knows how to actually play a sax.
- samirsd 5y agocould this model be applied to mimic the characteristics of a guitar amp or pedal for example?
- abdljasser2 5y agoHello! I think this can be done but I would not advise you to do it with this particular algorithm. I think you would be interested in the work of Jatin Chowdhury[1] as well as the work of Christian J. Steinmetz[2]. [1].https://arxiv.org/pdf/2009.02833.pdf https://arxiv.org/pdf/2009.02833.pdf [2] https://arxiv.org/abs/2102.06200 https://arxiv.org/abs/2102.06200
- ratherlongname 5y agoHow did you create the image (in 'How does this work')? Looks super clean.
- abdljasser2 5y agoThank you! It was made with figma.
- srcreigh 5y agoListenable stuff here https://storage.googleapis.com/ddsp/index.html https://storage.googleapis.com/ddsp/index.html
- abdljasser2 5y agoHello! I think you accidentally mistook the first citation link for a link to the work we are presenting. Here is the link I think you are looking for: https://erlj.notion.site/Neural-Instrument-Cloning-from-very-few-samples-2cf41d8b630842ee8c7eb55036a1bfd6 https://erlj.notion.site/Neural-Instrument-Cloning-from-very... Thank you!
- Minor49er 5y agoImpressive. This is really cool. To my ear, some of the synthesized results sound a bit closer to a stringed instrument at times, but it doesn't sound out of place. I just thought that was interesting
- kazinator 5y agoThe use of a technique from speech synthesis must be what helps the transposed passage not sound like a dumb sampling rate change. If you sing a higher notes, "formants" (frequency response profile of the vocal tract, roughly) stay the same.
- 8bitsrule 5y agoInteresting study. But conventional vis-a-vis all the unexplored timbre-space out there. I'm wondering if it'll be modifiable to 'morph' between two timbres in finite steps (by mod'ing partials). So you could play on any of those steps, or float around, or gliss along them. Flute+tuba => fluba. Fluba+violin => fluvioluba. And so on. (Harry Partch also wonders.)
- abdljasser2 5y agoHello! I think this will be possible!
- shannifin 5y agoThis would be another exciting area to explore... https://experiments.withgoogle.com/ai/sound-maker/view/ https://experiments.withgoogle.com/ai/sound-maker/view/ I especially love trombone+thunder
- ipnon 5y agoThe crux of the DDPS architecture seems to be allowing a model to train the many parameters of the sound synthesizer. You would never be able to create a truly realistic piano from a software synth by merely tuning the knobs because there are too many near-imperceptible minor fluctuations in the wave form, like the way the vibrations in aggregate vibrate the wooden piano housing. Training on real life examples, in this case an impressively short clip, gives the model enough data to train these near-imperceptible variations into the synth parameters. https://magenta.tensorflow.org/ddsp https://magenta.tensorflow.org/ddsp
- tgv 5y agoBit late to the action, but do I understand correctly that this trains a harmonic synthesizer? That would explain why the reverb in "How does the amount of training data affect synthesis fidelity?" sounds a bit off, as it would get synthesized on unrelated input. But what I wanted to ask: is the final model actually understandable or visualizable, e.g. as an envelope per harmonic that depends on F0, loudness and F0 confidence? Anyway, I'm impressed.
- abdljasser2 5y agoI can't say for sure but I think the issue with why the reverb sounds off in that particular example is that the reverb present in the recording has a longer decay than the maximum reverb duration I set for the experiments (1s). I will set it longer in the future. Yes, that kind of visualisation could be performed. The authors of the prior work we are building on (DDSP) have made some great visualisations which I think you will find useful! Here: https://storage.googleapis.com/ddsp/index.html https://storage.googleapis.com/ddsp/index.html
- shannifin 5y agoExciting stuff! I'm especially excited to see where this area will lead in regards to strings; I have yet to hear a sample string library produce a decent sounding classical symphony, at least without a lot of dynamics finessing (which, as a composer, I don't want to do; that's what human players would do naturally).