8 ms·
Pop2Piano: Pop audio-based piano cover generation
- Rochus 4y agoThat's amazing, congrats to the authors. Wonder how it would perform on harmonically more complex pop songs (e.g. https://www.youtube.com/watch?v=ZnRxTW8GxT8 https://www.youtube.com/watch?v=ZnRxTW8GxT8).
- aebtebeten 4y agoHaving been strongly influenced by Marx & Cannel How to Play the Piano Despite Years of Lessons, I suspect there's a much simpler harmony that would also back the melody. (that said, IIUC what pop2piano does, it's style transfer for arrangements — so my prediction is it would track the complex harmony, with appropriate realisations of each chord?) [Edit: > Pop2Piano uses only four-beat length audio for the context of input. Therefore, features such as melody contour or tex- ture of accompaniment have less consistency when generating longer than four-beat. Also, time quantization based on eighth note beats prevents the model from generating piano covers with other rhythms such as triplets, 16th notes, and trills. So, yeah, with that window it ought to track... https://arxiv.org/pdf/2211.00895.pdf https://arxiv.org/pdf/2211.00895.pdf ]
- sampo 4y agoThe song by Younha in the video ( https://www.youtube.com/watch?v=-SuI6snsbc4#t=1m26s https://www.youtube.com/watch?v=-SuI6snsbc4#t=1m26s ), doesn't it look like you'd need more than 2 hands to play the piano arrangement?
- fumblebee 4y agoUnbelievably cool. Great job. Tangentially — apologies if this is the wrong thread for this —, I know this is distinct to AI generated music (the melody here is a one-to-one mapping between the input song and output piano), but I’m curious when folks here think when the first totally generated artist will go mainstream with chart hits and millions of “followers”. With both an AI generated avatar and AI generated music. And no, not like Gorillaz who simply had fake cartoon personas. If you believe the argument floating around that content we consume in the future will be hyper individualised to the point music/tv will be generated just for us, then maybe never. IMO that runs counter to shared idolatry that people seem to crave, seems to me just a matter of time.
- probably_wrong 4y agoI think the answer is "never", but mostly because the target is both vague and impossibly strict. Let's take Hatsune Miku. You could argue that "she" fits the bill (3D character, algorithmic voice, lots of followers), but of course there are humans writing those songs and music. If you automated that away (AI lyrics, AI music) you would still have humans checking that the music fits "her" style, that the lyrics make sense, and that the result is not just a racist tirade due to 4chan training data. And as long as those humans are there you can't really say it's 100% AI, can you? One could argue (wrongly, IMHO) that doing the selection and filtering is not the same as making music, but that criteria would then classify DJs as "not musicians" and I know they hate that.
- ordu 4y ago> And as long as those humans are there you can't really say it's 100% AI, can you? I don't know how real artists do things, but I suspect that at least some of them rely on other people opinions about their new piece of work before going public with it. Basically for the same reasons. It is hard to objectively judge your own creative work. Especially because your fans do not judge your works objectively. You need to listen some voice of sanity to not loose your connection with reality. If it doesn't work as an argument, I can propose a thought experiment. Let's imagine a human artist with a mental disability who sometimes allow himself to do some really strange things. To protect him from big mistakes there is a small group of mentally healthy and competent people to filter his works. The question is: can we say that our disabled artist's works are not his works? Or not 100% his works? > And as long as those humans are there you can't really say it's 100% AI, can you? I believe it depends. How much of filtering those humans do?
- alar44 4y agoThat's just not how it works. If anything the opposite is true. Music is full of big egos, ie, this is my band, we do it my way or the highway. What you're saying might be true for boy bands and pop trash, but actual artists are extremely headstrong.
- 4y ago
- amelius 4y agoWould this work on e.g. heavy metal? Or would that be too far outside the network's training set?
- amelius 4y agoReminds me of: https://www.microsoft.com/en-us/research/project/songsmith-2/ https://www.microsoft.com/en-us/research/project/songsmith-2...
- bumby 4y agoIs it possible to have the output in traditional sight-reading notation?
- ls15 4y agoThere is notation software that can import MIDI, such as Musescore or Lilypond (both free)
- DoingSomeThings 4y ago1. Very cool tool. Awesome work! 2. How would one go about practicing irl to be able to play in the style of the piano covers? This is the style of piano I would love to be able to perform. I can read lead sheets and know music theory, but I just don't have the hand chops to perform. Every piano instruction book or tutorial I run across is based on developing progressive skills for classical performance. Pop style playing is distinct and I've never known how to progress my skills.
- minxomat 4y agoPiano With Johnny has good contemporary classes, building up improv skills gradually
- atonalfreerider 4y agoPlease look at PianoVision.app
- sigmar 4y agoVery cool. Note that it outputs midi and the demo chose to use a terrible piano sound, there's better free samples they could have used. I threw in a flac of the weirdest (read: most nonsensical melody, nowhere close to pop) song I could think of (Mupp - vendetta) into the collab and it seemed to do pretty well capturing the melody. Composer 1 lost some of the nuance of the melody at 0:50, but composer 14 got much closer.
- fassssst 4y agoOooo it outputs midi. Time to fire up Ableton and make use of Keyscape and all my VST’s. Thanks!
- DLeychIC 4y ago[dead]
- pimlottc 4y agoThis is pretty neat, but are they actually playable by a single human pianist? There's parts of the demo video where there are 10 simultaneous notes being played over 4 octaves, which doesn't seem humanly possible. Identifying the notes and chords being played is a big step but you've also got to adapt to the limitations of the instrument and figure out how to simplify it to be playable while retaining the same essence. That's a big part of what makes arrangement difficult.
- cjhanks 4y agoI didn't see catch this part, but having 10 notes over 4 octaves is not out of the question. If all of those notes are rhythmic, that could be a challenge. All of the fairly wide chords I saw could be played either with sustain or could be easily substituted with spread chords. Pop pianists don't usually play what's exactly on the page.
- alar44 4y agoAs a piano player, yes. However, the rhythms and chord selections are extremely weird and no human would ever play this way. It's a cute toy, but is pretty far from actual piano comping. It doesn't sound human or natural at all.
- vikbez 4y agoWell I recently home-made a player piano [1] this will be perfect for it :D Thanks ! [1] : https://www.youtube.com/watch?v=atJ_YsPFDjQ https://www.youtube.com/watch?v=atJ_YsPFDjQ
- tboerstad 4y agoMagic!
- yetanotherloser 4y agoVery nice work! How does it (does it at all?) control how hard it hits from one note to the next? What does the striking - solenoids?
- squarefoot 4y agoYes, it uses solenoid based actuators. Not sure if it controls the dynamics too, but one way to do that would be by driving the actuators using a curve modulated by high frequency PWM; probably quite hard to calibrate to MIDI key velocity since those actuators are essentially on/off devices, but doable. A very nice project, I'd love to see some more information about it.
- vikbez 4y agoWIP code but I added infos here: https://github.com/vikbez/PlayerPianoController/ https://github.com/vikbez/PlayerPianoController/
- yetanotherloser 4y agoThanks! So - midi velocity translates to PWM and hence solenoid force? Cool. You can always add some kind of mapping function if lower velocities are too low force. Nice work. (old pre-electronics player pianos could do dynamics but most relied heavily on input from the operator to do so - with various limitations - it's probably a good plan to avoid directly imitating them :-)
- deleted 4y ago
- ThrowawayTestr 4y agoThe covers sound really good. I wanna try running it against similar genres of music, see if anything is transferable.
- agolio 4y agoCool - but there is no proof that these demo songs are out-of-sample, and the trained model does not seem to be available in the github. If I am understanding correctly, the github code doesn't contain the code to train the model? [1] The paper also doesn't seem to be peer-reviewed? (I guess it is pre-review, since it is on arxiv only 2m ago) No disrespect to the authors - looks like a really cool project - I am just a bit suspicious without seeing any performance on user-submitted data. [1] https://github.com/sweetcocoa/pop2piano/issues/3 https://github.com/sweetcocoa/pop2piano/issues/3
- TaupeRanger 4y agoNearly 100% certainty these are cherry picked. Almost all "music generation" approaches are, which is a canary in the coal mine. Good music gen is much harder than most non-musical AI researchers assume.
- deleted 4y ago[deleted]
- oars 4y agoI just tried it with my own song submissions from a range of different genres, it works quite well.
- switchstance 4y agoقفل
- deleted 4y ago[deleted]
- dehrmann 4y agoSk8er Boi is an impressive song to demo. The chorus is in a different key than the verse, and one of the verse chords is out-of-key, borrowed from the chorus.
- dehrmann 4y agoI went to watch the video, saw "Seoul National University," figured that makes sense with how big K-Pop is. Then "Dynamite" came on.
- dehrmann 4y agoIs there a way to make sure the output is playable? Playing this looks like a bit of a stretch: https://i.imgur.com/pFC7tAr.png https://i.imgur.com/pFC7tAr.png
- smrtinsert 4y agoDoes it generate human playable arrangements? I don't even know what that means, but I assume there's some max width fingers can travel or have pressed down at the same time. This is an awesome thing and I can't wait to play with its output in my daws. This is continued proof to me that the future is art on demand that is instantly created and shared and does not need to be captured in a static form like a youtube video.
- abidlabs 4y agoPop2piano Gradio demo: https://huggingface.co/spaces/sweetcocoa/pop2piano https://huggingface.co/spaces/sweetcocoa/pop2piano
- Namidairo 4y agoI wonder if this would yield even better results if it were to be split into multiple tracks in the pre-processing stage with something like Demucs.[1] https://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs