4 ms·

For anyone looking to build on top of this. I have tried a few different STT systems, and they accurately capture what I am saying. Unfortunately, they don't su

For anyone looking to build on top of this. I have tried a few different STT systems, and they accurately capture what I am saying. Unfortunately, they don't support the reasonable workflow I want to open an office document, for example, and start talking. And I want the software to continuously type what I am saying at the cursor with minimal latency. The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.

mijoharas 2mo ago
Agreed. It's something I've found annoying about a few systems. It looks like the rust bindings have streaming examples so hopefully there is a nice solution here.
sipjca 2mo ago
You can fairly easily modify [Handy](https://handy.computer https://handy.computer) to do this if you want I’m planning on having it as a first class feature of the app too just too many other issues to work on first
- mft_ 2mo ago
  I’m really glad to hear this! A while ago, I auditioned about 10 different STT apps on my Mac, with this realtime/streaming transcription as a goal. I failed to find that feature in an app I was happy with, but settled on Handy as the best option otherwise. So if Handy adds this, it will be perfect!
- rolisz 2mo ago
  Can you give some pointers around this? I'd gladly help with a PR for this, but if you have anything docs/ideas around this it would be helpful.
  - sipjca 2mo ago
    I’m on a train right now but off the top of my head the audio pipeline may have to be modified slightly to emit partial text segments as they come in from the transcription engine. And then calling the appropriate paste method the user has in their settings. It may be easier than expected in some way since we already emit events for the live overlay, so it could be as small as a function call, but I don’t know the code path well enough from memory and what complexities it has. Probably with the Tauri context and a bit of other mess we have as this bit of code has gone through a lot of pain
mmmmbbbhb 2mo ago
You know English doesn't work like that. The word you're saying only becomes clear with the surrounding context. Eg, 'there' vs 'their'.
- nilslindemann 2mo ago
  It may be interesting to have it immediately insert the words, even if they are wrong, and when a sentence is finished, replace what has been written with the final corrected sentence.
  - atonse 2mo ago
    Google released this awkwardly named app called edge eloquent recently that does exactly that. In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too. Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.
    - remuskaos 2mo ago
      This sounds fantastic, but I'm utterly surprised that Google, of all companies, only releases this for macOS and iOS, but not Android.
      - atonse 2mo ago
        It’s an awkward app and the whole interaction just feels weird and kind of slapdash. I think it is meant to be a prototype. But in this day and age it’s easy enough to at least write the iOS and Android versions. But maybe not dealing with the play store.
    - mft_ 2mo ago
      I tried this on my Mac soon after launch and it was consuming a significant amount of processor cycles even just sitting idle in the menu bar. (From memory, ~20% of an M1 Max.) It may have been an early issue but with no obvious way to interact and report the issue and, eh, Google’s general attitude around customer satisfaction, I just gave up and deleted it again.
primaprashant 2mo ago
Totally understandable, but I’ve found that software that transcribes everything after I finish recording actually works better for me. I’ve tried both kinds, and systems that continuously type what I’m saying distract me from completing my thought. I end up reading what’s being typed and noticing transcription mistakes instead of focusing on what I’m trying to say. I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.
- solarkraft 2mo ago
  I can understand both modes. I mostly use transcription as input for my AI assistant and there I find it very useful to be able to check my input and just repeat myself in case something wasn’t fully captured. When using Apple’s transcription feature built into iOS and macOS, I also really like being able to edit everything right while the dictation is still active.
- abdullahkhalids 2mo ago
  Dream would be a combination of AI models that are smart like a human transcriber. One can just tell the model what mistakes it has made (e.g. "No, you wrote XYZ but I meant WXY"), and it is intelligent enough to realize when it is being instructed and when it needs to transcribe exactly what I say.
electronstudio 2mo ago
This is what I attempted with https://github.com/electronstudio/low_latency_dictation https://github.com/electronstudio/low_latency_dictation However the accuracy of the real time models is poor, so I did a second pass with a higher accuracy model before committing the text.
catmanjan 2mo ago
You used to be able to do this with dragon naturally speaking (don’t remember if that was it’s exact name) 10 ish years ago
jiehong 2mo ago
> The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful. It really depends on how one uses transcription. For example, I really value being able to open different windows, and look at graphs, or scroll some data while I'm dictating, because it can help me with providing some support information for what I'm saying. Some apps can even take into account things you copy or look at as part of the transcription's context to improve the results [0]. [0]: https://superwhisper.com/docs/common-issues/context#types-of-context https://superwhisper.com/docs/common-issues/context#types-of...
- abdullahkhalids 2mo ago
  This is pretty cool. The dream would be that all editors (including brower tabs) have an api interface which allows arbitrary LLMs to modify the editor content. So you can indeed look at different windows as you talk, but the editor keeps getting updated live, so you can go back and see where you already at.
LoganDark 2mo ago
Apple Dictation does this, or something similar, in my experience. Some apps (e.g. terminals in my experience) buffer the entire transcript but in most apps it's identical to typing as you speak. Have you tried it?
99catmaster 2mo ago
whisper.cpp has realtime capability. Been using it for 2 years at this point
Leftium 2mo ago
That was one of the main motivations for: https://rift-transcription.vercel.app https://rift-transcription.vercel.app I think I prototyped the most fluid real-time streaming transcription experience. For example, you can interrupt/intersperse the realtime transcription with keyboard input. If you click "Replay..." in the demo above, it shows an actual transcription use case (Black represents typed text, the colored text was dictated): - Say "Left" - Type `ium:` (because no model has been able to transcribe my username; special characters are hard) - Continue dictating...

mijoharas 2mo ago

Agreed. It's something I've found annoying about a few systems. It looks like the rust bindings have streaming examples so hopefully there is a nice solution here.

sipjca 2mo ago

You can fairly easily modify [Handy](https://handy.computer https://handy.computer) to do this if you want I’m planning on having it as a first class feature of the app too just too many other issues to work on first

mft_ 2mo ago
I’m really glad to hear this! A while ago, I auditioned about 10 different STT apps on my Mac, with this realtime/streaming transcription as a goal. I failed to find that feature in an app I was happy with, but settled on Handy as the best option otherwise. So if Handy adds this, it will be perfect!
rolisz 2mo ago
Can you give some pointers around this? I'd gladly help with a PR for this, but if you have anything docs/ideas around this it would be helpful.
- sipjca 2mo ago
  I’m on a train right now but off the top of my head the audio pipeline may have to be modified slightly to emit partial text segments as they come in from the transcription engine. And then calling the appropriate paste method the user has in their settings. It may be easier than expected in some way since we already emit events for the live overlay, so it could be as small as a function call, but I don’t know the code path well enough from memory and what complexities it has. Probably with the Tauri context and a bit of other mess we have as this bit of code has gone through a lot of pain

mmmmbbbhb 2mo ago

You know English doesn't work like that. The word you're saying only becomes clear with the surrounding context. Eg, 'there' vs 'their'.

nilslindemann 2mo ago
It may be interesting to have it immediately insert the words, even if they are wrong, and when a sentence is finished, replace what has been written with the final corrected sentence.
- atonse 2mo ago
  Google released this awkwardly named app called edge eloquent recently that does exactly that. In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too. Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.
  - remuskaos 2mo ago
    This sounds fantastic, but I'm utterly surprised that Google, of all companies, only releases this for macOS and iOS, but not Android.
    - atonse 2mo ago
      It’s an awkward app and the whole interaction just feels weird and kind of slapdash. I think it is meant to be a prototype. But in this day and age it’s easy enough to at least write the iOS and Android versions. But maybe not dealing with the play store.
  - mft_ 2mo ago
    I tried this on my Mac soon after launch and it was consuming a significant amount of processor cycles even just sitting idle in the menu bar. (From memory, ~20% of an M1 Max.) It may have been an early issue but with no obvious way to interact and report the issue and, eh, Google’s general attitude around customer satisfaction, I just gave up and deleted it again.

primaprashant 2mo ago

Totally understandable, but I’ve found that software that transcribes everything after I finish recording actually works better for me. I’ve tried both kinds, and systems that continuously type what I’m saying distract me from completing my thought. I end up reading what’s being typed and noticing transcription mistakes instead of focusing on what I’m trying to say. I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.

solarkraft 2mo ago
I can understand both modes. I mostly use transcription as input for my AI assistant and there I find it very useful to be able to check my input and just repeat myself in case something wasn’t fully captured. When using Apple’s transcription feature built into iOS and macOS, I also really like being able to edit everything right while the dictation is still active.
abdullahkhalids 2mo ago
Dream would be a combination of AI models that are smart like a human transcriber. One can just tell the model what mistakes it has made (e.g. "No, you wrote XYZ but I meant WXY"), and it is intelligent enough to realize when it is being instructed and when it needs to transcribe exactly what I say.

electronstudio 2mo ago

This is what I attempted with https://github.com/electronstudio/low_latency_dictation https://github.com/electronstudio/low_latency_dictation However the accuracy of the real time models is poor, so I did a second pass with a higher accuracy model before committing the text.

catmanjan 2mo ago

You used to be able to do this with dragon naturally speaking (don’t remember if that was it’s exact name) 10 ish years ago

jiehong 2mo ago

> The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful. It really depends on how one uses transcription. For example, I really value being able to open different windows, and look at graphs, or scroll some data while I'm dictating, because it can help me with providing some support information for what I'm saying. Some apps can even take into account things you copy or look at as part of the transcription's context to improve the results [0]. [0]: https://superwhisper.com/docs/common-issues/context#types-of-context https://superwhisper.com/docs/common-issues/context#types-of...

abdullahkhalids 2mo ago
This is pretty cool. The dream would be that all editors (including brower tabs) have an api interface which allows arbitrary LLMs to modify the editor content. So you can indeed look at different windows as you talk, but the editor keeps getting updated live, so you can go back and see where you already at.

LoganDark 2mo ago

Apple Dictation does this, or something similar, in my experience. Some apps (e.g. terminals in my experience) buffer the entire transcript but in most apps it's identical to typing as you speak. Have you tried it?

99catmaster 2mo ago

whisper.cpp has realtime capability. Been using it for 2 years at this point

Leftium 2mo ago

That was one of the main motivations for: https://rift-transcription.vercel.app https://rift-transcription.vercel.app I think I prototyped the most fluid real-time streaming transcription experience. For example, you can interrupt/intersperse the realtime transcription with keyboard input. If you click "Replay..." in the demo above, it shows an actual transcription use case (Black represents typed text, the colored text was dictated): - Say "Left" - Type `ium:` (because no model has been able to transcribe my username; special characters are hard) - Continue dictating...