3 ms·

I've been trying (and mostly failing) at settings up a pipeline to get system audio into whisper and feed that transcription into a seamless m4t text-to-text tr

I've been trying (and mostly failing) at settings up a pipeline to get system audio into whisper and feed that transcription into a seamless m4t text-to-text translation model. It seems like seamless streaming is going to solve most of my issues, and should significantly reduce latency! My ultimate goal is to have realtime translations of video conferences. I've moved to a new country, and while I'm super privileged that most of my colleagues speak English, we still have a number of "all hands" meetings that I get lost in pretty easily.

jwineinger 3y ago
What challenges have you faced? I’ve been planning do the same thing (whisper for ASR, and something else for text to text translation).
- gorbypark 3y ago
  Mostly issues with plumbing it all together. So far what I've accomplished is using blackhole to create a virtual speaker / microphone, so system audio is available as a microphone input device. Then, take that input and feed it into Whisper. Out of the box, Whisper seems designed to work on wav files, not a stream of audio. There are some "hacks" that chunk 2 or 3 seconds of audio and feed that into Whisper. Of course, it needs to be more advanced than just chunking audio, because what if a word is split in half? There are some projects that are attempting to do this intelligently (split chunks between words). So far this is as far as I've got, besides feeding that into seamless m4t, which also doesn't seem to support "streaming" in text and instead wants a txt file or a command line argument. Of course this is all before the new seamless v2 drop, so I'm hoping it will become easier! There's a lot of interesting things in this new seamless release, among them ggml inference and the streaming model. Poking around the code, there is even some whisper binding in there too, so it seems like a possible integration is already in use (I haven't had time to really dive in).

jwineinger 3y ago

What challenges have you faced? I’ve been planning do the same thing (whisper for ASR, and something else for text to text translation).

gorbypark 3y ago
Mostly issues with plumbing it all together. So far what I've accomplished is using blackhole to create a virtual speaker / microphone, so system audio is available as a microphone input device. Then, take that input and feed it into Whisper. Out of the box, Whisper seems designed to work on wav files, not a stream of audio. There are some "hacks" that chunk 2 or 3 seconds of audio and feed that into Whisper. Of course, it needs to be more advanced than just chunking audio, because what if a word is split in half? There are some projects that are attempting to do this intelligently (split chunks between words). So far this is as far as I've got, besides feeding that into seamless m4t, which also doesn't seem to support "streaming" in text and instead wants a txt file or a command line argument. Of course this is all before the new seamless v2 drop, so I'm hoping it will become easier! There's a lot of interesting things in this new seamless release, among them ggml inference and the streaming model. Poking around the code, there is even some whisper binding in there too, so it seems like a possible integration is already in use (I haven't had time to really dive in).