4 ms·

How good is the speech-to-text in practice? I have found that opensource s2t models are extremely far from Google/Apple ones unfortunately, and make a complete

by remram 5y ago

How good is the speech-to-text in practice? I have found that opensource s2t models are extremely far from Google/Apple ones unfortunately, and make a complete assistant really frustrating to use.

nshm 5y ago
Did you try Vosk https://github.com/alphacep/vosk-api https://github.com/alphacep/vosk-api or Wenet https://github.com/wenet-e2e/wenet https://github.com/wenet-e2e/wenet
- remram 5y ago
  They both seem to use Kaldi which I have tried before. edit: Nope Vosk is doing much better than I expected. Thanks a lot for the recommendation!
  - mkmk3 5y ago
    Glad you came back to add the edit, I tried Vosk recently and was also very surprised. Restored some hope for these systems that I wasn't anticipating, with some work I felt a nice hands-free dev environment is in reach.
- Psychokiller 5y ago
  Yes, but at the time I did it wasn't really usable. I shall have a look at it again, thank you
philipp2310 5y ago
Alice offers a few different STT options. While you can stay completely offline, depending on the language and your background noise, indeed the results are not as perfect as the cloud ones. But especially coqui STT (former deepspeech) does a really good job! Another option, if you are a bit more open with "sharing" data and you don't want to miss out the best TTS, is enabling google or azure cloud services - with the big difference, Alice will only send the sound right after detecting your hotword (only while flashing her LED and asking you "yes?" before the recording starts). Nothing else would be shared.
- moffkalast 5y ago
  Does it have an option to cache the whole "Hey Alice, what's gonna be the weather at 11 tomorrow?" utterance then if the wake word is detected send what's been cached after it? That's the main gripe I had with Mycroft, as there was no going around this: "Hey Mycroft." long pause "What's gonna be the weather at 11 tomorrow?" even longer pause <answer> Which is frankly so unnatural and just too annoying to be practical. And it feels like something that should be very straightforward to implement in terms of logic.
  - philipp2310 5y ago
    For the moment, no, nothing is cached until the hotword is recognized. We thought about it though, but it would mean we have to store passed sound input for a few seconds. While this won't be a problem for the main device, satellites aren't power full enough to run ASR them selfes (Raspi Zero), so the sound is streamed to the main device after the hotword detection. This process wouldn't match perfectly with the storing of the data. Another thing to keep in mind is, we use intermediate results for the ASR. Means already while you are speaking, the input is parsed. Only a few ms after you go silent, the parsing is finalized and NLU/TTS will start right away. Of course with a bit bias, I'd say it is more like: "Hey Alice" "Yes?.." "What's gonna be the weather at 11 tomorrow?" short pause (.2 seconds?) <answer>
  - Psychokiller 5y ago
    This is entirely true and I have a few solution I could deploy with some work, the problem is a caching like this consumes power, as you literally listen all the time, as for a wakeword, to cache the audio data in memory and use it ONLY if a wakeword is detected. Now big companies do it in the cloud, we could do it locally, as an option. The path I chose to mitigate that unatural feeling is to use a human answer, bit like at home, you in the kitchen, wife or kids further away, not communicating. At some point you'd call your wife "Alice?" and you'd wait for her to reply for a "yes?" before talking as you are unaware if she's focused on you at the moment or playing with the kids whatever
    - 5y ago
moffkalast 5y ago
> This means a Raspberry Pi and some other platforms such as the Respeaker core or the Matrix Creator is the best choices for a hardware platform. As for which Raspberry Pi, a raspberry 3, 3b, 3b+, 3A+ or 4 for the main unit are good choices. Extrapolating on the expected hardware, it's likely going to be either inaccurate, have very high latency, or both. Even Mycroft which open sourced everything else still runs the voice recognition part in the cloud because it's not possible to host it locally without rather beefy hardware that the average user won't have around (and even that has high latency). I did try some offline speech to text alternatives for it but they all more or less sucked to be frank.
- woojoo666 5y ago
  Google Assistant and Siri have offline functionality so it must at least be possible, but perhaps needs a company's worth of machine learning thrown at it
  - moffkalast 5y ago
    Ok that is genuinely surprising, perhaps the open source STTs are genuinely all like a decade behind the cutting edge.

by remram 5y ago

How good is the speech-to-text in practice? I have found that opensource s2t models are extremely far from Google/Apple ones unfortunately, and make a complete assistant really frustrating to use.

nshm 5y ago

Did you try Vosk https://github.com/alphacep/vosk-api https://github.com/alphacep/vosk-api or Wenet https://github.com/wenet-e2e/wenet https://github.com/wenet-e2e/wenet

remram 5y ago
They both seem to use Kaldi which I have tried before. edit: Nope Vosk is doing much better than I expected. Thanks a lot for the recommendation!
- mkmk3 5y ago
  Glad you came back to add the edit, I tried Vosk recently and was also very surprised. Restored some hope for these systems that I wasn't anticipating, with some work I felt a nice hands-free dev environment is in reach.
Psychokiller 5y ago
Yes, but at the time I did it wasn't really usable. I shall have a look at it again, thank you

philipp2310 5y ago

Alice offers a few different STT options. While you can stay completely offline, depending on the language and your background noise, indeed the results are not as perfect as the cloud ones. But especially coqui STT (former deepspeech) does a really good job! Another option, if you are a bit more open with "sharing" data and you don't want to miss out the best TTS, is enabling google or azure cloud services - with the big difference, Alice will only send the sound right after detecting your hotword (only while flashing her LED and asking you "yes?" before the recording starts). Nothing else would be shared.

moffkalast 5y ago
Does it have an option to cache the whole "Hey Alice, what's gonna be the weather at 11 tomorrow?" utterance then if the wake word is detected send what's been cached after it? That's the main gripe I had with Mycroft, as there was no going around this: "Hey Mycroft." long pause "What's gonna be the weather at 11 tomorrow?" even longer pause <answer> Which is frankly so unnatural and just too annoying to be practical. And it feels like something that should be very straightforward to implement in terms of logic.
- philipp2310 5y ago
  For the moment, no, nothing is cached until the hotword is recognized. We thought about it though, but it would mean we have to store passed sound input for a few seconds. While this won't be a problem for the main device, satellites aren't power full enough to run ASR them selfes (Raspi Zero), so the sound is streamed to the main device after the hotword detection. This process wouldn't match perfectly with the storing of the data. Another thing to keep in mind is, we use intermediate results for the ASR. Means already while you are speaking, the input is parsed. Only a few ms after you go silent, the parsing is finalized and NLU/TTS will start right away. Of course with a bit bias, I'd say it is more like: "Hey Alice" "Yes?.." "What's gonna be the weather at 11 tomorrow?" short pause (.2 seconds?) <answer>
- Psychokiller 5y ago
  This is entirely true and I have a few solution I could deploy with some work, the problem is a caching like this consumes power, as you literally listen all the time, as for a wakeword, to cache the audio data in memory and use it ONLY if a wakeword is detected. Now big companies do it in the cloud, we could do it locally, as an option. The path I chose to mitigate that unatural feeling is to use a human answer, bit like at home, you in the kitchen, wife or kids further away, not communicating. At some point you'd call your wife "Alice?" and you'd wait for her to reply for a "yes?" before talking as you are unaware if she's focused on you at the moment or playing with the kids whatever
  - 5y ago

moffkalast 5y ago

> This means a Raspberry Pi and some other platforms such as the Respeaker core or the Matrix Creator is the best choices for a hardware platform. As for which Raspberry Pi, a raspberry 3, 3b, 3b+, 3A+ or 4 for the main unit are good choices. Extrapolating on the expected hardware, it's likely going to be either inaccurate, have very high latency, or both. Even Mycroft which open sourced everything else still runs the voice recognition part in the cloud because it's not possible to host it locally without rather beefy hardware that the average user won't have around (and even that has high latency). I did try some offline speech to text alternatives for it but they all more or less sucked to be frank.

woojoo666 5y ago
Google Assistant and Siri have offline functionality so it must at least be possible, but perhaps needs a company's worth of machine learning thrown at it
- moffkalast 5y ago
  Ok that is genuinely surprising, perhaps the open source STTs are genuinely all like a decade behind the cutting edge.