4 ms·

Ask HN: Looking for a Firefox compatible extension for voice controlled actions

Basically I would like to have an extension where I could say "click publish" and it will find the button with the text publish and click it. Is there anything like this for FF, I ask because the FF voice repo got shut down a couple years ago and haven't found a competitor.

BtM909 3y ago
As an FYI: most Firefox also supports most Chrome Extensions, so it might help you to go through its store as well :)
osrec 3y ago
Building this in JS would not be too difficult using the speech to text API, if that's available to you. I'm not aware of an extension that does this out of the box though.
- tim1994 3y ago
  I assume you are referring to the SpeechRecognition interface of the Web Speech API. Unfortunately this is not supported by Firefox and even for Chrome it doesn't work offline. Source: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecog...
SCdF 3y ago
I control my browser (actually my whole computer) with my voice. I'm not aware of anything specific where you could say "click publish" and it would intelligently find the published button, though I haven't looked. What I use is a combination of vimium (a vim browser plugin, there are a few) and talon (https://talonvoice.com/ https://talonvoice.com/), either just relying on the vim shortcuts but with my voice, of for specific domains making talon shortcuts. I have also heard good things about Rango, which is supposed to be a vimium like thing but much more voice focused: https://github.com/david-tejada/rango https://github.com/david-tejada/rango. I haven't tried to use it though.
- Karunamon 3y ago
  Dragon (dictate/NaturallySpeaking/professional/whatever they are calling it this year) has the ability to do this in conjunction with their official browser add-on, but it is an absolutely miserable product otherwise and I would not recommend it. I can say the words "click reply" and this message will be sent.
  - rickydroll 3y ago
    as a multi-decade user of the various forms of Dragon, it's not a miserable product. The company is a bunch of ass holes now owned by another bunch of ass holes. But for day-to-day use, it's not bad assuming a Windows environment doesn't mess with it. Most people go south with Dragon when they expect to be of the speak the keyboard or expected to be a perfect transcription. The reality is, it's like dictating to a good typist with a little bit of hearing problems. And like this hypothetical typist, you get worse results if you yell at it. From what I can tell, Dragon, and I suspect most other speech recognition environments work best if you use them for dictating text and have enough commands supporting text editing. Dragon did a good job with what used to be called "Select-and-Say". It was a form of editing that worked with speech model and not some bastard grammar hack Forcing the editors keystroke commands to do a given job.
    - Karunamon 3y ago
      On the contrary, my issues with Dragon are not surrounding its recognition accuracy (though I would kill to be able to hook it up to whisper), but literally everything else. - Running the accuracy tuning tool more than once actually degrades accuracy - User profiles will degrade over time, necessitating a export/new profile/import process every few months or else the accuracy becomes unacceptable - Does not work with electron applications that apparently fucking everything has to be nowadays - The browser plug-in… Dear God. It outright does not function properly on many top websites (Text input on twitter and twitch come to mind immediately), and it occasionally causes Dragon to slow down to an infuriating degree. Sometimes it actually causes certain sites to break! - Unstable. Crashes to desktop a couple of times per week, and leaves some appendage running in such a way that it can't be directly restarted. I had to write a PowerShell script (https://gist.github.com/Karunamon/f2b684d083fc341ee79aab1fb29cf3c5 https://gist.github.com/Karunamon/f2b684d083fc341ee79aab1fb2...) and put it on my start menu to deal with this.
jvanveen 3y ago
Perhaps try something like this? https://github.com/ggerganov/whisper.cpp/tree/master/examples/command.wasm https://github.com/ggerganov/whisper.cpp/tree/master/example...
DeTheBug 3y ago
You may want to give https://serenade.ai https://serenade.ai a try, they have browser support (might be chrome only? not sure), it's good enough, It has the " click publish" feature, and if it doesn't work then you can say like "show inputs" and it will show numbers next to each input so instead of saying "click publish" you say "click 14" etc...
- deleted 3y ago
  [deleted]
- gardnr 3y ago
  Where does their revenue come from?
  - KomoD 3y ago
    There is none, they used to have a pro version
gebbie 3y ago
[dead]
tim1994 3y ago
Assuming this does not yet exist: If anyone is interested in building something like this (that works offline) please let me know, I'd like to join. I have some experience building web extensions but unfortunately not the time to develop & maintain this on my own right now.
- Jemm 3y ago
  Same.
  - loa_in_ 3y ago
    Neither of you have a way (in your profile) to contact you outside of the website
    - tim1994 3y ago
      Perhaps a caching or timing issue? I updated my profile shortly after posting this (and noticing it myself).
- user3939382 3y ago
  All you need is a CSS selector for the aria label and a click action. Once the voice transcription is done it’s 1 line of JavaScript. You’d have to rely on the aria labels though.
  - tim1994 3y ago
    It's not quite that simple. For example the button doesn't necessarily have to have an aria label. Could be the (nested!) content. There could be multiple candidates (buttons). The speech recognition thing is also not trivial. Yes there are WebAssembly based solutions but those have to be properly integrated and tested (not sure how good they actually are)
    - bryanrasmussen 3y ago
      for a Cucumber based testing solution back in 2013 the company I was working for had all element interaction commands go to a function that tried to identify what element it could be so you could write "I click 'home'" and that function would do 1. look in a object we had in which a bunch of DOM objects have been identified via CSS selectors of XPaths to see if there was an object named 'home' if that not exist 2. look for an element with the id home if that not exist 3. look for an element with the class home - take the first one. If that not exist 4. look for an element in which the text node was 'home' There were a few other heuristics to determine what to do also dependent on if you said I select home, I click home, I move the mouse over home etc.
invalid-access 3y ago
Same here. Sadly, my dad is beginning to lose sight in both eyes now and has trouble operating essential websites (e.g. his bank's webpages). Does anyone know of browser extensions that enable a high-level voice command language? I'm thinking commands like: * "Open Bank of America" * "Login with saved password" * "What's my savings bank balance?" Yeah, I know it's fraught with security challenges and is likely acceptable only if using local models. Still, anything out there?
- KomoD 3y ago
  I highly doubt there is something out there already that does this.
devilsAdv0cate 3y ago
You can still use Firefox Voice if you don't mind setting up your own STT server. https://juliacambre.com/firefox-voice/ https://juliacambre.com/firefox-voice/
lasselassesen 3y ago
[flagged]
lasselassesen 3y ago
[flagged]

BtM909 3y ago

As an FYI: most Firefox also supports most Chrome Extensions, so it might help you to go through its store as well :)

osrec 3y ago

Building this in JS would not be too difficult using the speech to text API, if that's available to you. I'm not aware of an extension that does this out of the box though.

tim1994 3y ago
I assume you are referring to the SpeechRecognition interface of the Web Speech API. Unfortunately this is not supported by Firefox and even for Chrome it doesn't work offline. Source: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecog...

SCdF 3y ago

I control my browser (actually my whole computer) with my voice. I'm not aware of anything specific where you could say "click publish" and it would intelligently find the published button, though I haven't looked. What I use is a combination of vimium (a vim browser plugin, there are a few) and talon (https://talonvoice.com/ https://talonvoice.com/), either just relying on the vim shortcuts but with my voice, of for specific domains making talon shortcuts. I have also heard good things about Rango, which is supposed to be a vimium like thing but much more voice focused: https://github.com/david-tejada/rango https://github.com/david-tejada/rango. I haven't tried to use it though.

Karunamon 3y ago
Dragon (dictate/NaturallySpeaking/professional/whatever they are calling it this year) has the ability to do this in conjunction with their official browser add-on, but it is an absolutely miserable product otherwise and I would not recommend it. I can say the words "click reply" and this message will be sent.
- rickydroll 3y ago
  as a multi-decade user of the various forms of Dragon, it's not a miserable product. The company is a bunch of ass holes now owned by another bunch of ass holes. But for day-to-day use, it's not bad assuming a Windows environment doesn't mess with it. Most people go south with Dragon when they expect to be of the speak the keyboard or expected to be a perfect transcription. The reality is, it's like dictating to a good typist with a little bit of hearing problems. And like this hypothetical typist, you get worse results if you yell at it. From what I can tell, Dragon, and I suspect most other speech recognition environments work best if you use them for dictating text and have enough commands supporting text editing. Dragon did a good job with what used to be called "Select-and-Say". It was a form of editing that worked with speech model and not some bastard grammar hack Forcing the editors keystroke commands to do a given job.
  - Karunamon 3y ago
    On the contrary, my issues with Dragon are not surrounding its recognition accuracy (though I would kill to be able to hook it up to whisper), but literally everything else. - Running the accuracy tuning tool more than once actually degrades accuracy - User profiles will degrade over time, necessitating a export/new profile/import process every few months or else the accuracy becomes unacceptable - Does not work with electron applications that apparently fucking everything has to be nowadays - The browser plug-in… Dear God. It outright does not function properly on many top websites (Text input on twitter and twitch come to mind immediately), and it occasionally causes Dragon to slow down to an infuriating degree. Sometimes it actually causes certain sites to break! - Unstable. Crashes to desktop a couple of times per week, and leaves some appendage running in such a way that it can't be directly restarted. I had to write a PowerShell script (https://gist.github.com/Karunamon/f2b684d083fc341ee79aab1fb29cf3c5 https://gist.github.com/Karunamon/f2b684d083fc341ee79aab1fb2...) and put it on my start menu to deal with this.

jvanveen 3y ago

Perhaps try something like this? https://github.com/ggerganov/whisper.cpp/tree/master/examples/command.wasm https://github.com/ggerganov/whisper.cpp/tree/master/example...

DeTheBug 3y ago

You may want to give https://serenade.ai https://serenade.ai a try, they have browser support (might be chrome only? not sure), it's good enough, It has the " click publish" feature, and if it doesn't work then you can say like "show inputs" and it will show numbers next to each input so instead of saying "click publish" you say "click 14" etc...

deleted 3y ago
[deleted]
gardnr 3y ago
Where does their revenue come from?
- KomoD 3y ago
  There is none, they used to have a pro version

gebbie 3y ago

[dead]

tim1994 3y ago

Assuming this does not yet exist: If anyone is interested in building something like this (that works offline) please let me know, I'd like to join. I have some experience building web extensions but unfortunately not the time to develop & maintain this on my own right now.

Jemm 3y ago
Same.
- loa_in_ 3y ago
  Neither of you have a way (in your profile) to contact you outside of the website
  - tim1994 3y ago
    Perhaps a caching or timing issue? I updated my profile shortly after posting this (and noticing it myself).
user3939382 3y ago
All you need is a CSS selector for the aria label and a click action. Once the voice transcription is done it’s 1 line of JavaScript. You’d have to rely on the aria labels though.
- tim1994 3y ago
  It's not quite that simple. For example the button doesn't necessarily have to have an aria label. Could be the (nested!) content. There could be multiple candidates (buttons). The speech recognition thing is also not trivial. Yes there are WebAssembly based solutions but those have to be properly integrated and tested (not sure how good they actually are)
  - bryanrasmussen 3y ago
    for a Cucumber based testing solution back in 2013 the company I was working for had all element interaction commands go to a function that tried to identify what element it could be so you could write "I click 'home'" and that function would do 1. look in a object we had in which a bunch of DOM objects have been identified via CSS selectors of XPaths to see if there was an object named 'home' if that not exist 2. look for an element with the id home if that not exist 3. look for an element with the class home - take the first one. If that not exist 4. look for an element in which the text node was 'home' There were a few other heuristics to determine what to do also dependent on if you said I select home, I click home, I move the mouse over home etc.

invalid-access 3y ago

Same here. Sadly, my dad is beginning to lose sight in both eyes now and has trouble operating essential websites (e.g. his bank's webpages). Does anyone know of browser extensions that enable a high-level voice command language? I'm thinking commands like: * "Open Bank of America" * "Login with saved password" * "What's my savings bank balance?" Yeah, I know it's fraught with security challenges and is likely acceptable only if using local models. Still, anything out there?

KomoD 3y ago
I highly doubt there is something out there already that does this.

devilsAdv0cate 3y ago

You can still use Firefox Voice if you don't mind setting up your own STT server. https://juliacambre.com/firefox-voice/ https://juliacambre.com/firefox-voice/

lasselassesen 3y ago

[flagged]

lasselassesen 3y ago

[flagged]