6 ms·
Audio Decomposition – open-source seperation of music to constituent instruments
- DidYaWipe 2y agoSome of those videos don't have audio, as far as I can tell...
- bottom999mottob 2y agoThis is really cool, but there's real-world instrument physics that might not be captured by simple Fourier transform templates, like a trumpet playing softly can have a significantly different harmonic spectrum than the same trumpet playing loudly, even at the same pitch Trumpets produce a rich harmonic series with strong overtones, meaning their Fourier transform would show prominent peaks at integer multiples of the fundamental frequency. Instruments like flutes have more pure tones, but brass instruments typically have stronger higher harmonics, which would lead to more complex partial derivatives in the matrix equation shown in the article So this script uses bandpass filtering and cross-correlation of attack/release envelopes to identify note timing. Given that brass instruments can exhibit non-linear behavior where the harmonic content changes significantly with playing intensity (think of the brightness difference between pp and ff passages), not sure how would this algorithm could handle intensity-dependent timbral variations. I'd consider adding intensity-dependent Fourier templates for each instrument to improve accuracy
- atoav 2y agoAs someone who uses source separation twice a week for mixing purposes the number of other instruments that can produce sounds of "vocal" quality is high. These models all stop functiining well when you have bands where the instruments don't sound typical and aren't played and/or mixed in a way that achieves maximum separation between them — e.g. an electrical guitar with a distorted harmonic hitting the same note as your singer while the drummer plays only shrieking noises on their cymbals and the bass player simulates a punching kick drum on their instrument. In these situations (experimental music) source separation will produce completely unpredictable results, thst may or may not be useful for musical rebalancing.
- fnordlord 2y agoWhat tool do you use for the source separation? Everything I've used so far is great for learning or transcribing to MIDI but the separated tracks always have a strange phasing sound to them. Are you doing something to clean that up before mixing back in or are the results already good enough?
- atoav 2y agoiZotope RX with musical rebalance, great to reduce drum spill from vocal mics
- ipsum2 2y agoI must be dumb, but none of the YouTube video demos are demonstrating source separation? Edit: to clarify, source separation in audio research means separating out the audio into separate clips.
- timlod 2y agoThe title is a bit confusing as open-source separation of ... reads like source separation, which this is not. Rather, it is a pitch detection algorithm which also classifies the instrument the pitch originated with. I think it's really neat, but the results look like it could take more time to fix the output than using a manual approach (if really accurate results are required).
- emptiestplace 2y agoNo, it doesn't read like that. The hyphen completely eliminates any possible ambiguity.
- croes 2y agoMaybe added later by OP? Because there is no hyphen in the article’s subtitle. >Open source seperation of music into constituent instruments.
- emptiestplace 2y agoThe complaint: > The title is a bit confusing as open-source separation of ... reads like source separation, which this is not.
- ipsum2 2y agoThe title of the submission was modified. It you read the article it says: Audio Decomposition [Blind Source Seperation]
- earthnail 2y agoThanks for clarifying. In fairness to the author, he is still at high school: https://matthew-bird.com/about.html https://matthew-bird.com/about.html Amazing work for that age.
- timlod 2y agoWow, I didn't see that. Great to see this level of interest early on!
- baq 2y agoGot a flashback of playing audiosurf 15 or so years ago. Time flies. https://en.wikipedia.org/wiki/Audiosurf https://en.wikipedia.org/wiki/Audiosurf
- fxj 2y agoIf you are interested in audio (or stem) separation have a look at RipX https://hitnmix.com/ripx-daw-pro/ https://hitnmix.com/ripx-daw-pro/ It can even export the separated tracks as midi files. It still has some problems but works very well. Stem separation is now standard in the musical software and almost every DAW provides it.
- makz 2y agoThanks for the information. I’m a long time Logic Pro user and I wasn’t aware of this feature.
- Sporktacular 2y agoOn an M1/2/3/4 processor. Not Intel.
- sbarre 2y agoStemroller[0] has been around for a while too, it's free and based on Meta's models: 0: https://www.stemroller.com/ https://www.stemroller.com/
- oidar 2y ago> almost every DAW provides it. It's an up and coming feature that nearly every DAW should have, but most don't yet. Ableton Live - No Bigwig - No Cubase - No FL - Yes Logic - Yes Pro Tools - No Reason - No Reaper - No Studio One - Yes
- ekianjo 2y agoLooks like this may be the work of Joshua Bird's little brother (?). Joshua bird did some impressive projects already, that were featured on HN before: https://www.youtube.com/@joshuabird333 https://www.youtube.com/@joshuabird333
- njb99 2y agoYes, Matt is Josh's little brother. I'm impressed - and very pleased - you noticed this.
- generalizations 2y agoNo one else is going to mention that "separation" was misspelled four times?
- orbitingpluto 2y agoIf we can all hear the tiny violin, who cares?
- generalizations 2y agoDegradation of the environment. https://en.wikipedia.org/wiki/Broken_windows_theory#Theoretical_explanation https://en.wikipedia.org/wiki/Broken_windows_theory#Theoreti...
- orbitingpluto 2y agoSomeone created something. Its value greatly exceeds the perceived "degradation of the environment" of a spelling mistake. Not acknowledging that says more about the pedant than the creator.
- bastloing 2y agoI can't find the source code, but the project looks interesting.
- loubbrad 2y agoI didn't see it referenced directly anywhere in this post. However, for those interested, automatic music transcription (i.e., audio->MIDI) is actually a decently sized subfield of deep learning and music information retrieval. There have been several successful models for multi-track music transcription - see Google's MT3 project (https://research.google/pubs/mt3-multi-task-multitrack-music-transcription https://research.google/pubs/mt3-multi-task-multitrack-music...). In the case of piano transcription, accuracy is nearly flawless at this point, even for very low-quality audio: https://github.com/EleutherAI/aria-amt https://github.com/EleutherAI/aria-amt Full disclaimer: I am the author of the above repo.
- WiSaGaN 2y agoHow does the problem simplify when it's restricted to piano?
- loubbrad 2y agoEssentially, the leading way to do automatic music transcription is to train a neural network on supervised data, i.e., paired audio-MIDI data. In the case of piano recordings, there is a very good dataset for this task which was released by Google in 2018: https://magenta.tensorflow.org/datasets/maestro https://magenta.tensorflow.org/datasets/maestro Most current research involves refining deep learning based approaches to this task. When I worked on this problem earlier this year, I was interested in adding robustness to these models by training a sort of musical awareness into them. You can see a good example of it in this tweet: https://x.com/loubbrad/status/1794747652191777049 https://x.com/loubbrad/status/1794747652191777049
- bravura 2y agoI know the reported scores of MT3 are very good, but have you had success with using it yourself? https://replicate.com/turian/multi-task-music-transcription https://replicate.com/turian/multi-task-music-transcription I ported their colab to runtime so I could use it more easily. The MIDI output is... puzzling? I've tried feeding it even simple stems and found the output unusable for some tracks, i.e. the MIDI output and audio were not well aligned and there were timing issues. On other audio it seemed to work fine.
- testoveride 2y agoFf
- kasajian 2y agodude can't spell
- berbec 2y agoHe's in high school and pulls of a project like this. I thought I was slick convincing the 7-11 guy to give me my Twist-a-Pepper soda without charging me bottle deposit or tax.
- fonema 2y agoI'm a long-time fan of Ultrastar Deluxe, which is an open-source clone of Singstar. This is a karaoke game where people compete by singing along to the tune. It recognizes the notes you are singing and compares them to a vocals-timings mapping file for that particular song. The better you sing to the tune (getting the words correct doesn't matter), the higher your score. While there are extensive libraries of fan-made song mappings, it's never enough, and there are very few mapped songs in languages other than English or Spanish (if you or your friends prefer your native language). Doing the entire mapping manually is time-consuming, not to mention that I am almost tone-deaf myself, which would make it even more difficult. I have been wondering for a long time what software I could use to make this process easier to automate. This seems like a great tool for capturing vocal timings and notes from original songs. I have it on my bucket list to create a Singstar playlist in my native language and host a singing party with friends. Does anyone have suggestions for other similar tools?
- alok-g 2y agoLovely. I did not know of this. Sounds like the text file needs vocals and pitches along with time stamps. AI is getting there to allow automating it's creation. For myself: Adding a link I just found for reading further. https://www.reddit.com/r/karaoke/comments/x61kzy/modern_equivalent_of_singstarultrastar/ https://www.reddit.com/r/karaoke/comments/x61kzy/modern_equi...