4 ms·

Little off topic.. I'm trying to help a friend in media industry. His requirement is ability to identify different voice in a movie and have the output time st

by achow 7y ago

Little off topic.. I'm trying to help a friend in media industry. His requirement is ability to identify different voice in a movie and have the output time stamped - Ex. Voice A: 00.00.00sec - 00.05.30sec, Voice B: 00.05.31 - 00.06.30, etc. It would be very helpful if anyone can point to any tools that exists that can do that (open source or otherwise).

lozenge 7y ago
AWS offer this for $1.44ph
- achow 7y ago
  Thanks. Are you referring to Transcribe? https://aws.amazon.com/transcribe/ https://aws.amazon.com/transcribe/ Transcribe seems to be more for speech-to-text irrespective to who is making the speech. Here the requirement is to identify the unique voices. Ex. if "Mary had a little lamb" is voiced by two different voices then the engine should identify Voice A said "Mary" at 00.00.00sec-00.00.01sec and then Voice B said "had a little" at 0.00.02sec-00.00.03sec, then Voice A again said "lamb".. etc.
  - gregsadetsky 7y ago
    Transcribe does “Recognize Multiple Speakers” — it’s on the page that you linked to, in the list of features. I would also recommend comparing it to Google’s and maybe MS/Azure’s services. Aside from the fidelity of the transcription itself, and the accuracy of disambiguating the different speakers, I’m also not convinced that all of these services will give you an end timestamp (start timestamps are sometimes there, but not necessarily for every word or sentence). You could try a multi prong approach by doing the transcript to get text and speakers only (using the services mentioned above), and then using an aligner such as “gentle” [0] to find the start / end times. You will have some gaps and wrongly transcribed words, but it may be a start..! [0] https://github.com/lowerquality/gentle https://github.com/lowerquality/gentle
  - yorwba 7y ago
    From your link: Recognize Multiple Speakers Amazon Transcribe is able to recognize when the speaker changes and attribute the transcribed text appropriately. This can significantly reduce the amount of work needed to transcribe audio with multiple speakers like telephone calls, meetings, and television shows. So they claim to be able to do it. However, having to do speaker diarization doesn't exactly make speech recognition easier, so you should adjust your expectations regarding the error rate accordingly.
    - achow 7y ago
      Duh. It was right there. But thanks; on doing little bit of search on 'speech diarization' saw that Google Cloud has that service. https://cloud.google.com/speech-to-text/docs/multiple-voices https://cloud.google.com/speech-to-text/docs/multiple-voices And then came across this comparison between various cloud transcription service, which should be helpful for evaluating on diarization aspect as well. AI-POWERED TRANSCRIPTION SERVICES SHOWDOWN: AWS VS. GOOGLE VS. IBM WATSON VS. NUANCE https://www.armedia.com/blog/transcription-services-aws-google-ibm-nuance/ https://www.armedia.com/blog/transcription-services-aws-goog...
thinkersilver 7y ago
If AWS gets you going definitely go for it but if you're looking for existing tools type 'python speaker diarization' in github.' Most of what you'll find would require downsampling your audio to 16khz and you'll find a combination of NN based diarizer and hmm based models pre-ML. One thing to note a lot of the systems will work well for interviews, broadcast media and footage taken from a camera because the audio will tend to be clean. Film and movies will be a challenge because of the background music being identified as a separate voice. It tends to to confuse it. Haven't used the cloud based systems with audio the background sounds you tend to find in film movies.

by achow 7y ago

lozenge 7y ago

AWS offer this for $1.44ph

achow 7y ago
Thanks. Are you referring to Transcribe? https://aws.amazon.com/transcribe/ https://aws.amazon.com/transcribe/ Transcribe seems to be more for speech-to-text irrespective to who is making the speech. Here the requirement is to identify the unique voices. Ex. if "Mary had a little lamb" is voiced by two different voices then the engine should identify Voice A said "Mary" at 00.00.00sec-00.00.01sec and then Voice B said "had a little" at 0.00.02sec-00.00.03sec, then Voice A again said "lamb".. etc.
- gregsadetsky 7y ago
  Transcribe does “Recognize Multiple Speakers” — it’s on the page that you linked to, in the list of features. I would also recommend comparing it to Google’s and maybe MS/Azure’s services. Aside from the fidelity of the transcription itself, and the accuracy of disambiguating the different speakers, I’m also not convinced that all of these services will give you an end timestamp (start timestamps are sometimes there, but not necessarily for every word or sentence). You could try a multi prong approach by doing the transcript to get text and speakers only (using the services mentioned above), and then using an aligner such as “gentle” [0] to find the start / end times. You will have some gaps and wrongly transcribed words, but it may be a start..! [0] https://github.com/lowerquality/gentle https://github.com/lowerquality/gentle
- yorwba 7y ago
  From your link: Recognize Multiple Speakers Amazon Transcribe is able to recognize when the speaker changes and attribute the transcribed text appropriately. This can significantly reduce the amount of work needed to transcribe audio with multiple speakers like telephone calls, meetings, and television shows. So they claim to be able to do it. However, having to do speaker diarization doesn't exactly make speech recognition easier, so you should adjust your expectations regarding the error rate accordingly.
  - achow 7y ago
    Duh. It was right there. But thanks; on doing little bit of search on 'speech diarization' saw that Google Cloud has that service. https://cloud.google.com/speech-to-text/docs/multiple-voices https://cloud.google.com/speech-to-text/docs/multiple-voices And then came across this comparison between various cloud transcription service, which should be helpful for evaluating on diarization aspect as well. AI-POWERED TRANSCRIPTION SERVICES SHOWDOWN: AWS VS. GOOGLE VS. IBM WATSON VS. NUANCE https://www.armedia.com/blog/transcription-services-aws-google-ibm-nuance/ https://www.armedia.com/blog/transcription-services-aws-goog...

thinkersilver 7y ago

If AWS gets you going definitely go for it but if you're looking for existing tools type 'python speaker diarization' in github.' Most of what you'll find would require downsampling your audio to 16khz and you'll find a combination of NN based diarizer and hmm based models pre-ML. One thing to note a lot of the systems will work well for interviews, broadcast media and footage taken from a camera because the audio will tend to be clean. Film and movies will be a challenge because of the background music being identified as a separate voice. It tends to to confuse it. Haven't used the cloud based systems with audio the background sounds you tend to find in film movies.