4 ms·

Visually Indicated Sounds

50 points by markhkim 10y ago

peterburkimsher 10y ago
The neural network could learn to make a "boom" sound for explosions, e.g. Michael Bay films. It also reminded me of this GIF: How to get a song stuck in someones head within 3 frames: http://imgur.com/gallery/c18TRq0 http://imgur.com/gallery/c18TRq0
Zolomon 10y ago
I would love to experience an action movie that uses this to generate the sound effects automatically. Would also be a nice tool for professional foley specialists to automate the process.
- leni536 10y ago
  Preferably with a network trained on Terence Hill and Bud Spencer movies.
- virgil_disgr4ce 10y ago
  And a sound generation API!
unwind 10y ago
Previous submission, roughly three weeks ago, plenty of comments there: https://news.ycombinator.com/item?id=11893868 https://news.ycombinator.com/item?id=11893868.
dahart 10y ago
I'm struck by how very dissimilar each successive "match" in the video is. If this is a neural network that makes matches based on video, how on earth is it picking matches from completely different training videos for every subsequent impact? It might happen if the training videos only contained a single impact sample, but that's not the case here, the paper says each training video has 48 actions. I can totally understand if the authors wanted to make sure it doesn't pick the exact same sample multiple times in a row and penalized duplicates, but I don't see any mention of that in the paper, and even if they did, I'd expect to see subsequent matches from the same training video, rather than picks from completely different videos.
rangibaby 10y ago
I can hear the sound of people's voices when I am watching a video with the sound off, even if I have never heard them speak before. I had glue ear that went undiagnosed for a long time as a kid, so that may have something to do with it.
blevin 10y ago
Anyone needing an idea for a hack project: make a realtime version of this for phone or webcam. Bonus points for swapping in ridiculous sound sets, or keying augmented-reality visual fx overlays.

50 points by markhkim 10y ago

The neural network could learn to make a "boom" sound for explosions, e.g. Michael Bay films. It also reminded me of this GIF: How to get a song stuck in someones head within 3 frames: http://imgur.com/gallery/c18TRq0 http://imgur.com/gallery/c18TRq0

Zolomon 10y ago

I would love to experience an action movie that uses this to generate the sound effects automatically. Would also be a nice tool for professional foley specialists to automate the process.

leni536 10y ago
Preferably with a network trained on Terence Hill and Bud Spencer movies.
virgil_disgr4ce 10y ago
And a sound generation API!

unwind 10y ago

Previous submission, roughly three weeks ago, plenty of comments there: https://news.ycombinator.com/item?id=11893868 https://news.ycombinator.com/item?id=11893868.

dahart 10y ago

I'm struck by how very dissimilar each successive "match" in the video is. If this is a neural network that makes matches based on video, how on earth is it picking matches from completely different training videos for every subsequent impact? It might happen if the training videos only contained a single impact sample, but that's not the case here, the paper says each training video has 48 actions. I can totally understand if the authors wanted to make sure it doesn't pick the exact same sample multiple times in a row and penalized duplicates, but I don't see any mention of that in the paper, and even if they did, I'd expect to see subsequent matches from the same training video, rather than picks from completely different videos.

rangibaby 10y ago

I can hear the sound of people's voices when I am watching a video with the sound off, even if I have never heard them speak before. I had glue ear that went undiagnosed for a long time as a kid, so that may have something to do with it.

blevin 10y ago

Anyone needing an idea for a hack project: make a realtime version of this for phone or webcam. Bonus points for swapping in ridiculous sound sets, or keying augmented-reality visual fx overlays.