13 ms·

There's also https://github.com/akashmjn/tinydiarize https://github.com/akashmjn/tinydiarize which received support recently in core whisper.cpp - it doesn't su

There's also https://github.com/akashmjn/tinydiarize https://github.com/akashmjn/tinydiarize which received support recently in core whisper.cpp - it doesn't support the large model yet, but the results are promising

0cf8612b2e1e 3y ago
That’s a pretty exciting development. Is there any putative timelines on when it might get expanded to the large models? At this point, I would have to do a two pass workflow: first with large for accuracy and a second with the tube to assign speakers.
- garblegarble 3y ago
  Not that I can see, the developer's roadmap[1] currently is at writing a blogpost about it & trying different sampling methods, expanding to the large model looks to be a long way off (which is a real pity, even if there was just a finetune of large for english that would be a big help over the existing small english finetune). Could you go into more detail about your workflow? I'd been considering a two-pass approach myself until I discovered tinydiarize mentioned in whisper.cpp's --help text 1: https://github.com/akashmjn/tinydiarize#roadmap https://github.com/akashmjn/tinydiarize#roadmap
  - 0cf8612b2e1e 3y ago
    I have not yet done this, but my thinking is that the diarization process would be useful data, but not so much that I would be willing to accept the quality dip of using the tiny trained model. I already run the large model at the word level, so I have high quality transcription + timestamps. Then I would perform the second whisper-pass with the diarization and discard all of the output except when a speaker change is identified. Merge the two results based on timestamps and there you go.
    - garblegarble 3y ago
      Interesting - I'd been considering basically that two-pass approach in reverse (diarize, isolate each utterance in a per-speaker file with a vocal separator between each clip so that I don't have to rely on timestamps being perfectly accurate, run through whisper - could also do a file per utterance, but setup time seems like it would dominate in that case). Concern I had about my approach was that it'd be too different from whisper's training set of conversational data, and that might cause accuracy to take a dive.

0cf8612b2e1e 3y ago

That’s a pretty exciting development. Is there any putative timelines on when it might get expanded to the large models? At this point, I would have to do a two pass workflow: first with large for accuracy and a second with the tube to assign speakers.

garblegarble 3y ago
Not that I can see, the developer's roadmap[1] currently is at writing a blogpost about it & trying different sampling methods, expanding to the large model looks to be a long way off (which is a real pity, even if there was just a finetune of large for english that would be a big help over the existing small english finetune). Could you go into more detail about your workflow? I'd been considering a two-pass approach myself until I discovered tinydiarize mentioned in whisper.cpp's --help text 1: https://github.com/akashmjn/tinydiarize#roadmap https://github.com/akashmjn/tinydiarize#roadmap
- 0cf8612b2e1e 3y ago
  I have not yet done this, but my thinking is that the diarization process would be useful data, but not so much that I would be willing to accept the quality dip of using the tiny trained model. I already run the large model at the word level, so I have high quality transcription + timestamps. Then I would perform the second whisper-pass with the diarization and discard all of the output except when a speaker change is identified. Merge the two results based on timestamps and there you go.
  - garblegarble 3y ago
    Interesting - I'd been considering basically that two-pass approach in reverse (diarize, isolate each utterance in a per-speaker file with a vocal separator between each clip so that I don't have to rely on timestamps being perfectly accurate, run through whisper - could also do a file per utterance, but setup time seems like it would dominate in that case). Concern I had about my approach was that it'd be too different from whisper's training set of conversational data, and that might cause accuracy to take a dive.