3 ms·
I have not yet done this, but my thinking is that the diarization process would be useful data, but not so much that I would be willing to accept the quality di
by 0cf8612b2e1e 3y ago
I have not yet done this, but my thinking is that the diarization process would be useful data, but not so much that I would be willing to accept the quality dip of using the tiny trained model. I already run the large model at the word level, so I have high quality transcription + timestamps. Then I would perform the second whisper-pass with the diarization and discard all of the output except when a speaker change is identified. Merge the two results based on timestamps and there you go.
- garblegarble 3y agoInteresting - I'd been considering basically that two-pass approach in reverse (diarize, isolate each utterance in a per-speaker file with a vocal separator between each clip so that I don't have to rely on timestamps being perfectly accurate, run through whisper - could also do a file per utterance, but setup time seems like it would dominate in that case). Concern I had about my approach was that it'd be too different from whisper's training set of conversational data, and that might cause accuracy to take a dive.