5 ms·
The Sound of Pixels
- keyle 8y agoCool stuff. Any real world usage/benefits for this? I can't think of any.
- drhodes 8y agomatching faces with voices in a surveillance situation, which person is talking?
- vernie 8y agoThat's actually this paper: https://ai.googleblog.com/2018/04/looking-to-listen-audio-visual-speech.html https://ai.googleblog.com/2018/04/looking-to-listen-audio-vi...
- black_puppydog 8y agothat's actually a whole subfield, not "this paper".
- deleted 8y ago[deleted]
- utopcell 8y agoIt would be great for practicing songs. In this scenario you would be clicking on the instrument to turn off.
- rolfvandekrol 8y agoFor the sound splitting, it's probably not extremely useful. You can however use this frequency analysis from video to synchronise audio with a video (for a music video where you want to use a studio audio recording with a video clip)
- hrayr 8y agoInteresting, I was wondering what to do with my audiblepixel.com domain.
- funkdified 8y agoThis is going to be huge for the hard of hearing.
- jovial_cavalier 8y agohow
- tmalsburg2 8y agoNot op but I imagine the idea must be that they can point on the thing they want to year in a noisy environment and software can amplify the sound generated at that location.
- pjgrad 8y agoUnlike reading, I don't think audio can convey the same meaning in a different sensory format. At best, they perceive it but in an alien way to most people. It's like describing a painting in musical notes.
- xtagon 8y agoI wonder if this can segregate vocals from instrumentals in a mix? That would be great for mashups.
- bscphil 8y agoIncidentally, due to the way a lot of stereo tracks are mixed, it's often possible to mostly remove the vocal track from a song. I'm more curious if this algorithm could perform the reverse task - playing the vocals only. My intuition is that the results would be poor because of the wide human vocal range and the fact that words need to be discernible, not just notes. But I would love to be proven wrong here.
- StavrosK 8y agoIf you can remove the vocals from a piece, you can then subtract that from the original to get just the vocals.
- nawgszy 8y agoWell that's not quite true. The point is, I believe, that vocals are generally put right in the center of the sound-stage, so they play equally in the left and right channels. Thus right - left is most of the rest of the song, but the vocals cancelled each other out. However, the right - left mix isn't exactly the song minus the vocals, it's an odd off-version, so subtracting that from the original song will leave mostly the vocals but with artifacts from the difference between the song truly without vocals and the right - left mix's interpretation thereof
- app4soft 8y agoIt remind me such apps like PhonoPaper[0], Nature - Oscillator[1] and PixiVisor[2] [0] http://warmplace.ru/soft/phonopaper http://warmplace.ru/soft/phonopaper [1] http://warmplace.ru/soft/nosc http://warmplace.ru/soft/nosc [2] http://warmplace.ru/soft/pixivisor/ http://warmplace.ru/soft/pixivisor/
- rflrob 8y agoI haven’t tried my hand at any machine learning, but I’m impressed that it could work with only 60 hours of training data. Perhaps the input clips were fairly short, which would increase the total number of videos.
- ggm 8y agoDoes anyone remember Gerry Anderson? he designed relays attached to puppets, which made the jaws clack in time to the sound-track being played, while they filmed the puppets for Thunderbirds are go in the 1960s. Look at me ma! my puppet is speaking! Thus, it only took us 50-odd years to write the reverse-compiler..
- a-dub 8y agoInteresting. I wonder how well a logistic regression that spits out masks would perform in the source separation task. Also a bit surprising to see that they had to STFT the audio before feeding it into a convnet. I thought half the point of convnets was that they figure out how to do spectral domain representations on their own...
- black_puppydog 8y agoin theory yes, but in practice, giving the network the full information in the right format is crucial to have it train well and quickly.
- a-dub 8y agoisn't that supposed to be the magic of convnets though? they _figure out_ the right format. instead of doing feature engineering, like stfts and mel warping, you do stuff like build convolution layers into an ann and let it sort it out?
- WhiteNoiz3 8y agoI've been messing around with audio nets and that was the first thing that surprised me too - convnets don't work nearly as well (out of the box) for audio as they do for images. This article has some good reasons why audio data is different from image data: https://towardsdatascience.com/whats-wrong-with-spectrograms-and-cnns-for-audio-processing-311377d7ccd https://towardsdatascience.com/whats-wrong-with-spectrograms...