4 ms·

I wanted to do exactly this, but I wanted to do it at a word level granularity. So that you could input any text and get a video of random clips from many sourc

by nsxwolf 12y ago

I wanted to do exactly this, but I wanted to do it at a word level granularity. So that you could input any text and get a video of random clips from many sources each saying exactly the words in the input string. I don't think the metadata is quite there yet.

gerbal 12y ago
You could get a gigantic bunch of film & television subtitles. Almost every movie ever made and every TV show in the past 30 years has pretty good subtitles available from a variety of online sources. Using common Python NLP techniques you could very easily search for every instance of a phrase across a massive corpora of subtitles. If you got a large enough collection of subtitles and videos in a single directory this tool would do what you are asking.
vitovito 12y ago
Word (and phoneme) level granularity is usually used for lip-synching (CG, video games) and karaoke-type applications. If you have an accurate text transcript, but not detailed enough timings, you can use speech recognition on the audio and it will be very accurate, since you know exactly what is being said (unlike speech recognition on arbitrary speech, this is more like command-and-control). You can do word-level or phoneme-level timing granularity pairing an accurate transcript with the original audio. The metadata isn't there in regular subtitles, but you can certainly get it there with some post-processing.
- arafalov 12y ago
  Is there a good easy-to-use software/service that does that kind of alignment of accurate text transcript to audio/video? I used to have some links but the companies went out of business.
  - vitovito 12y ago
    Probably not for free. I also haven't done this kind of work in over a decade. The current links I have are: Annosoft's SDK: http://www.annosoft.com/prices http://www.annosoft.com/prices Annosoft made a command-line front-end to the Microsoft Speech API, which is what many of these other Windows-based systems may also use, and I used in a project in 1999-2001: http://www.annosoft.com/sapi_lipsync/docs/ http://www.annosoft.com/sapi_lipsync/docs/ (There are other SAPI front-ends if you dig around online, too.) Others, including open-source ones: http://en.wikipedia.org/wiki/List_of_speech_recognition_software http://en.wikipedia.org/wiki/List_of_speech_recognition_soft... Magpie, used in animation and gaming: http://www.thirdwishsoftware.com/magpiepro.html http://www.thirdwishsoftware.com/magpiepro.html Crazytalk, used in animation, uses SAPI: http://www.reallusion.com/crazytalk/crazytalk.aspx http://www.reallusion.com/crazytalk/crazytalk.aspx FaceFX, used in gaming: http://www.facefx.com/documentation/2013.2/W194 http://www.facefx.com/documentation/2013.2/W194 Source Filmmaker includes it, although I'd be surprised if it wasn't Sphinx or SAPI or some other existing library: https://developer.valvesoftware.com/wiki/SFM/Lip-sync_animation_and_Extract_Phonemes https://developer.valvesoftware.com/wiki/SFM/Lip-sync_animat...
jevinskie 12y ago
Something similar to this (but automated, of course)? http://video.bobdylan.com/desktop.html http://video.bobdylan.com/desktop.html
wtracy 12y ago
How about a program that automatically finds, say, the smallest number of segments that covers all of the input words? At least then the program would do a lot of the heavy lifting for you, and you could do the last bit of cleanup manually.
swatkat 12y ago
Here's an example of "true" video search: http://www.baarzo.com/ http://www.baarzo.com/
blago 12y ago
I've had the same idea on the back burner for some time. My primary concern was - and still is - copyright.
- crisnoble 12y ago
  Surely a one word clip would fall under fair use.
sunsu 12y ago
You should check out the Op3nVoice API: http://www.op3nvoice.com/ http://www.op3nvoice.com/ Lets you search video/audio for spoken words.

by nsxwolf 12y ago

gerbal 12y ago

You could get a gigantic bunch of film & television subtitles. Almost every movie ever made and every TV show in the past 30 years has pretty good subtitles available from a variety of online sources. Using common Python NLP techniques you could very easily search for every instance of a phrase across a massive corpora of subtitles. If you got a large enough collection of subtitles and videos in a single directory this tool would do what you are asking.

vitovito 12y ago

Word (and phoneme) level granularity is usually used for lip-synching (CG, video games) and karaoke-type applications. If you have an accurate text transcript, but not detailed enough timings, you can use speech recognition on the audio and it will be very accurate, since you know exactly what is being said (unlike speech recognition on arbitrary speech, this is more like command-and-control). You can do word-level or phoneme-level timing granularity pairing an accurate transcript with the original audio. The metadata isn't there in regular subtitles, but you can certainly get it there with some post-processing.

arafalov 12y ago
Is there a good easy-to-use software/service that does that kind of alignment of accurate text transcript to audio/video? I used to have some links but the companies went out of business.
- vitovito 12y ago
  Probably not for free. I also haven't done this kind of work in over a decade. The current links I have are: Annosoft's SDK: http://www.annosoft.com/prices http://www.annosoft.com/prices Annosoft made a command-line front-end to the Microsoft Speech API, which is what many of these other Windows-based systems may also use, and I used in a project in 1999-2001: http://www.annosoft.com/sapi_lipsync/docs/ http://www.annosoft.com/sapi_lipsync/docs/ (There are other SAPI front-ends if you dig around online, too.) Others, including open-source ones: http://en.wikipedia.org/wiki/List_of_speech_recognition_software http://en.wikipedia.org/wiki/List_of_speech_recognition_soft... Magpie, used in animation and gaming: http://www.thirdwishsoftware.com/magpiepro.html http://www.thirdwishsoftware.com/magpiepro.html Crazytalk, used in animation, uses SAPI: http://www.reallusion.com/crazytalk/crazytalk.aspx http://www.reallusion.com/crazytalk/crazytalk.aspx FaceFX, used in gaming: http://www.facefx.com/documentation/2013.2/W194 http://www.facefx.com/documentation/2013.2/W194 Source Filmmaker includes it, although I'd be surprised if it wasn't Sphinx or SAPI or some other existing library: https://developer.valvesoftware.com/wiki/SFM/Lip-sync_animation_and_Extract_Phonemes https://developer.valvesoftware.com/wiki/SFM/Lip-sync_animat...

jevinskie 12y ago

Something similar to this (but automated, of course)? http://video.bobdylan.com/desktop.html http://video.bobdylan.com/desktop.html

wtracy 12y ago

How about a program that automatically finds, say, the smallest number of segments that covers all of the input words? At least then the program would do a lot of the heavy lifting for you, and you could do the last bit of cleanup manually.

swatkat 12y ago

Here's an example of "true" video search: http://www.baarzo.com/ http://www.baarzo.com/

blago 12y ago

I've had the same idea on the back burner for some time. My primary concern was - and still is - copyright.

crisnoble 12y ago
Surely a one word clip would fall under fair use.

sunsu 12y ago

You should check out the Op3nVoice API: http://www.op3nvoice.com/ http://www.op3nvoice.com/ Lets you search video/audio for spoken words.