8 ms·

Cursorless – A spoken language for structural code editing

175 points by yewenjie 4y ago

akihitosan 4y ago
Imho text editing speed is not an issue for programmers.
- eimrine 4y ago
  And TDD is bad because it requires too much of typing /sarcasm
  - marginalia_nu 4y ago
    TDD is bad because it introduces distracting busywork. The actual typing isn't the actual problem. In terms of just typing, I could probably produce something like 50,000 lines of code in a 40 hour work week in a moderately verbose programming language. In practice I'll probably crank out somewhere between 1000-5000. It's the thinking that is slow. It's made even slower by adding additional tasks and context switching.
    - demopathos 4y ago
      The average professional software engineer produces around 10 LOC per day.
- comfypotato 4y ago
  It’s more complicated than that I think. You don’t spend much time typing, but how much does the time you spend (and the cognitive load of) typing deconstruct the mental architecture you’ve created to solve a problem? This is at the core of why vim/others are still so popular. The micro-gains in typing efficiency translate to much more efficient problem solving because you don’t “lose your place” within your mental process. I think the same would extend to cursorless.
  - magicalhippo 4y ago
    I find typing can be helpful. I think more about certain aspects of the problem as I type, so I often discover edge cases I hadn't thought about or potential performance pitfalls. I guess it's a bit like navigating using only the map in the war room vs navigating on the ground without a map. The ideal route comes through a combination of both, and typing allows for the latter point of view. Though after typing for a while, I'll take a step back and think about the overall solution again, going over it in my mind. Then back to typing. That said, I gotta have a responsive editor.
- kwijybo 4y ago
  I admit this a weird edge case, but a few years ago I fractured my left elbow and could only use one hand. When typing one handed, typing speed was a bottleneck.
Jemm 4y ago
I find it annoying that they are not using the standard phoenetic alphabet
- ParetoOptimal 4y ago
  As someone who has voice coded full-time for a long period of time, that would get quite annoying. Notice that some users even start using Talon's noise support. I even changed "scroll up" to "suh" because I said it 300 times per day.
- yewenjie 4y ago
  I think the main reason is too easily distinguish (to the program) between letters - and I guess to also have shorter words.
  - OJFord 4y ago
    That's also the NATO phonetic alphabet's raison d'être, where a lot more's on the line; a lot more research time and money has been spent on it over the years. Air, Jury, and Quench in particular strike me as ambiguous examples NATO would never go for. Krunch is just baffling. Plex breaks the scheme followed by every other (that it's the first letter in the word).
    - lunixbochs 4y ago
      Talon's alphabet is extremely carefully designed. There's de bruijn sequence style redundant phonetic information in each letter allowing the single syllable words to be mashed together at high speed without pauses (even allows for a bit of slurring to fit more than one letter into a syllable). I can type 60wpm with it, see me spelling a last name around 0:40 in this video for typical usage [1]. It's also carefully tested (with an American accent) to not jam up your mouth / tongue when you say the letters in any order. You can remap it to whatever you want, and a few users choose either NATO or a custom alphabet. But most stick with Talon's with minor changes (e.g. they might change a specific word if it's harder to say clearly in their accent, or if their mic or environment makes it more difficult to pick up specific sounds). You can also use the English alphabet for spelling anywhere you can say a word, e.g. "say hello" and "say H E L L O" both work the same. This isn't a voice assistant where you belt out a command once in a while then move on with your day, and this isn't a low fidelity radio link. These words may be said tens of thousands of times a day. Using three-syllable NATO words is slow and exhausting. It's common for voice programming systems to define their own alphabet based on a truncated NATO alphabet. I went further because I wanted to be able to use vim and play DCSS at very high speeds with my key commands instead of binding a bunch of app specific custom commands. [1] https://twitter.com/lunixbochs/status/1378159234861264896 https://twitter.com/lunixbochs/status/1378159234861264896
dag11 4y ago
This kind of work is also fantastic for those who have limited mobility or use of hands. On top of that, it's also a comforting peace of mind for the rest of us that should we ever e.g. lose our hands in a car collision, we might still have a successful path forward in our programming careers and passions. Reminds me of a talk I once saw where a developer developed arthritis and couldn't program effectively anymore, so he developed an almost "verbal vim" plugin for vim (or was it emacs?) that allowed him to write/edit code quickly using verbal utterances. I forget what that project was, but I wonder how these two compare.
- kingcoleman 4y ago
  I was very inspired by a blind programmer who uses text to speech at inhuman speeds. https://www.parhamdoustdar.com/2016/03/27/autobiography-blind-programmer/ https://www.parhamdoustdar.com/2016/03/27/autobiography-blin... That's perhaps an extreme example of what is possible but it's incredible what people can adapt to.
- jffry 4y ago
  Are you thinking perhaps of this [1] PyCon talk by Tavis Rudd, who had RSI and a rotator cuff injury? He demos a voice command system that's all short nonsense syllables, including with a few humorous video clips interspersed, and then explains it after the demo. [1] https://www.youtube.com/watch?v=8SkdfdXWYaI https://www.youtube.com/watch?v=8SkdfdXWYaI (demo starts 9 minutes in)
- sterlind 4y ago
  Cursorless is indeed fantastic for this! I used it extensively last winter/spring when I couldn't type.
LionTamer 4y ago
I wonder if there’s room for integration with Vim/Neovim or the plugins in that ecosystem - I feel like there’s a strong overlap between the “vim language” of text objects, motions, etc and the structure editing this project is working with.
- deleted 4y ago
  [deleted]
- auscompgeek 4y ago
  We have plans for adding a keyboard interface [0] at least, though it's still an open discussion. If you're after straight Neovim support though, perhaps nvim-treesitter-textobjects [1] might tickle your fancy? [0]: https://github.com/cursorless-dev/cursorless/issues/710 https://github.com/cursorless-dev/cursorless/issues/710 [1]: https://github.com/nvim-treesitter/nvim-treesitter-textobjects https://github.com/nvim-treesitter/nvim-treesitter-textobjec...
t_mann 4y ago
Looks great! I've been dreaming about something that lets me use a headband and reads brainwaves to interact with a computer for a long time. Every 4-5 years I look into what's been happening on that front, but so far it seems like we're not even close.
- NJRBailey 4y ago
  I am hoping for an eye-tracking mouse solution in the meantime, where looking at a portion of the screen and pressing a key on the keyboard will click rather than having to move a hand away from the keyboard to use a mouse, or wait for the analog stick mouse to reach the right point of the screen. Seems to be getting more plausible but still fairly expensive for a few years yet.
  - lunixbochs 4y ago
    Talon has an eye tracking mouse, the necessary hardware runs around $250.
    - yep31 4y ago
      I would say that an eye tracking mouse is what OptiKey has, and for Talon it's more of an eye tracking+head pointing+voice clicking mouse. If you have tmj/vocal cords/neck problems you won't be able to use eye tracker as a mouse replacement.
      - lunixbochs 4y ago
        > for Talon it's more of an eye tracking+head pointing+voice clicking mouse > If you have tmj/vocal cords/neck problems you won't be able to use eye tracker as a mouse replacement. I disagree with these assertions. There are several options in Talon for eye tracking. Optikey is a nice suite of functionality, but it's Windows-only, while Talon is cross platform. I also have a few users who use the Optikey UI but prefer to use Talon's eye tracking to control it. I've certainly recommended Optikey to users. The updated direct control mouse mode in Talon 0.3 can be used without any head tracking at all as long as your targets aren't too small. The zoom mode explicitly uses no head tracking as well. To click or trigger the zoom mouse, you can make a pop noise, use voice commands, or a physical mouse button, or keyboard, or any keyboard emulating input device (such as a foot pedal), or use one of the dwell options.
downboots 4y ago
How many wpm?
vanjajaja1 4y ago
Very interesting. Editing code at the level of text is too low, ideally it would be raised to the level of thought. We don't think in terms of characters changing, we think in terms of higher level block constructs. It's not about the increased typing speed, it's about reducing cognitive load and making problem solving more enjoyable. Pairing with the Unison-lang style system of direct AST manipulation could be revolutionary, if the difficulty doesn't prove too high.
orzig 4y ago
I may not ever use it, but I love that such innovative interfaces exist. Huge congratulations to the team
soulofmischief 4y ago
Seems cool. However, I went down a several-hour rabbit hole trying to build non-documented dependencies and eventually gave up. The reliance on mono/skiasharp makes this a far less portable application, and to say there is Linux support seems disingenuous. Binaries only exist for Ubuntu, the binaries are said to work for non-ubuntu systems but nuget fails to install them. Building from scratch leads to several undocumented dependencies and I'm still receiving increasingly inscrutable build failures out of the box. Feels like 2005... My patience ran out. If anyone on an RPM-based system gets everything up and running, drop me a line.
- lunixbochs 4y ago
  You talking about Talon? It comes with all of the dependencies. Use ./run.sh to set the library path and ask on the Slack if there are any further issues. (fwiw it's not skiasharp itself, it's just the mono/skia repo, which is C++)
  - soulofmischief 4y ago
    Thanks for the tip. Any reason run.sh isn't mentioned anywhere in the documentation? https://talonvoice.com/docs/ https://talonvoice.com/docs/
    - auscompgeek 4y ago
      Isn't it mentioned in the README in the tarball?
      - soulofmischief 4y ago
        The tarball, talon-linux.tar.xz, didn't come with a README, neither under root or /resources. I could certainly have broken run.sh open in vim (and should have), I could have looked into /lib and found the missing dependencies, but given that there was a binary literally called talon, the general expectation is this is the binary to be linked to my PATH and run directly. Using a run.sh is not necessary when the same could be achieved by making the talon file a shell script and renaming the binary to something else.
sillylaird 4y ago
The issue i have with this and will always with speech to text software. my voice will never pick it up unless im very clear and i take the time to prounouce a word or a chord?. I will have more luck writing in a stenograph keyboard or a chording keyboard then this. I like the fact that this technology assits for those who can have their voice be read by a computer. It will never work for me... Despite how modern a lot of this software has became.
- pomatic 4y ago
  You may want to look at your hardware setup. Modern (1-3 year old) laptops now generally have incredibly good echo cancellation microphone setups which massively improve recognition accuracy. If you are stuck with an older machine, try a USB based (integral ADC, the logical inverse of a DAC used for playing audio) lapel mic. These can be had off amazon for a tenner, but will infuriate you everytime you forget to take it off and get up from your chair... One last tip: My experience suggests the bluetooth microphone quality is not good enough for voice recognition tasks like this.
  - sillylaird 4y ago
    thank you for the suggestion. i will try to look into it. it just at the same i was in highschool the technology wasn't as good for my voice. so i just learned how to type faster. but im willing to give it a shot and i can be a like "beta tester" or contributer if needed. i tend to use wired stuff since i do music production and am very picky on how stuff comes out when im finally mixing them.
viktour19 4y ago
Combine this with something like Copilot and you get the 10x programmer :)
cryptonector 4y ago
This is impressive. A bit like vi, but voiced.
sterlind 4y ago
Cursorless is fantastic, this saved my bacon last year when I couldn't use my hands for months. I donate to Pokey because I'll need this again in the future, though right now I'm typing okay. I think able-bodied programmers will probably get annoyed setting it up, setting talon up, learning the phonetic alphabet and basic commands, but like.. if your hands don't work, this can literally save your career. this is a disability aid for creating things, it's vital and empowering! the hard part IMO is getting back into the coding headspace you had when you could type. it's different speaking code, formatting it in your head before you say command phrases. everything's a little awkward, you can't get into the zone the same way.
- foerbert 4y ago
  I have some very mixed feelings about talon. I looked into it a decent bit when I started having hand issues as well, and never got anywhere useful with it. For one, it needs to be programmed to do anything. This gets you all the options, which is wonderful, but also is a bit of a problem when you're looking into because you can't, ya know, program. And it's not a simple thing to interact with. Lots of moving parts and various behind-the-scenes interactions. For another, there's virtually zero documentation. Like, officially none, and unofficially not much more than a few paragraphs. And to put the cherry on top, it's a closed source mess of a program. It's not easy to get much in the way of useful information out of it. You can poke around some in a Python REPL, but that'll basically just get you the interface. And somehow, it's one of the most useful sources of information, as at least you've got something to go on. And as mentioned, Python, so you don't have much in the way of language-level guarantees to try to rely on to figure stuff out either. Yes, there is the popular user config. To be clear, I'm not trying to be unkind to it. It managed to overcome a lot, and is very useful to lots of people for many of the reasons I stated above. But. It's also clearly got it's roots as one person's personal config that got popular. There is so much in there. It's basically it's whole own thing to learn on top of talon. In terms of learning from it, well, it doesn't seem trivial. It's basically it's whole own thing, but also how it all interacts is all talon. You're basically learning two separate yet intertwined systems at once and have to figure out what bits belong to what. It's also got various bits from multiple iterations of talon's interface, so the same thing may be accomplished multiple ways. It's actually quite well commented in some places about this fact, which is really nice. But it does make you wonder what isn't well commented... Ultimately I just gave up. It seems like a potentially hugely useful thing, but actually trying to use it required learning a bunch of stuff I wasn't otherwise interested in, figuring out the minutia of a complex unstable interface the hardest ways possible, and then spending a ton of time writing my own program/config. Aside from the time involved, the amount of typing required was prohibitive. So yeah. My feelings on talon... mixed, to say the least. Or most, judging by the above.
chaosprint 4y ago
Very cool. Love to see how the support for different groups of people is considered in the language level! It would also be interesting to test brain interfaces on this language some day. I am thinking about alternative input method for my project Glicol (https://glicol.org/ https://glicol.org/) quite some time. This work is definitely inspiring. I am also thinking about how can we better support the blind with some feedback. If you have some suggestions, please let me know here: https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol
cube2222 4y ago
The whole ecosystem around Talon[0] is fantastic. You'll probably want to start with [1] if it's your first time, it's a community-maintained talon config which is a good starting point for customization. The speech recognition engine built for it is great and the whole thing is free of charge, though I do recommend donating on Patreon [2]. Cursorless is a nice rethinking-from-the-ground kind of project. However, you can just as well keep using your current tools. I've used Talon on and off with Jetbrains IDEs when I had some wrist pain and the overall experience was great, with my coding speed being fast enough without much practice. When voice typing, autocomplete is also very useful, so the introduction of Copilot was an additional huge improvemet. But overall, the design of Talon, its config files, and the way this leads to very easy customizability, is really great. A bit like making your whole OS easily automatable and programmable (even if your OS isn't emacs /s). [0]: https://talonvoice.com/ https://talonvoice.com/ [1]: https://github.com/knausj85/knausj_talon https://github.com/knausj85/knausj_talon [2]: https://www.patreon.com/lunixbochs https://www.patreon.com/lunixbochs
- phillco 4y ago
  I agree. Talon is incredible. Before I discovered it, I was convinced that my career would be ended early by RSI, as it was getting slightly worse every year. Meanwhile voice coding looked like impenetrable wizardry from the few videos that I had seen. But after discovering it and getting through the learning curve (and there is one, that's probably the biggest reason why more people don't use it), it's something I look forward to using for the rest of my life. Even as my wrists start to gradually recover, being able to use voice/noise/eye tracking in tandem to the keyboard, allowing you to use the best for each purpose, is just so much more productive (not to mention, cool! fun!) that I doubt I'll ever go back.
deleted 4y ago
[deleted]
jstanley 4y ago
To anyone tempted to get into Talon: note that it is very much not open source - https://talonvoice.com/EULA.txt https://talonvoice.com/EULA.txt (except when you contibute code to them; in that case you implicitly license your code as open source, but theirs is still their property).
- deleted 4y ago
  [deleted]
- lunixbochs 4y ago
  Are you on Linux?
  - jstanley 4y ago
    Yes. Re your deleted comment about "hostile forks": allowing users to run whatever software they want is precisely the point of open source. It's not "hostile" to have different opinions to you. The project looks cool nonetheless, but it is still true that if the users don't control the software, the software controls the users.
    - lunixbochs 4y ago
      I'll repost that in sibling. I just realized that I'd rather have people saying "ah, typical of proprietary software to not support Linux" than "your work has no value because I can't compete with you using your own code". I'm not going to stop supporting Linux because of this, but it still seems like a backwards incentive. This is a business, not a pet project. I work on it full time and it is my meager source of income. I'm not convinced it is possible to make much of a living on open source. I do mean hostile when I say hostile. I have a particular project of mine in mind, where the fork renamed, split the user base, and lagged behind my changes. They later just reset to my HEAD (as I was making way more progress than the fork), then removed attribution in the docs and solicited donations for the work I had done. I burned out on that project shortly after. > if the users don't control the software, the software controls the users Talon is free, runs offline, cares about user privacy and security, and gives an incredible amount of control to the user. The entire voice interface and behavior is defined in the user's code. There are several open source projects in this space and none of them are anywhere close to Talon on making it easy to build and customize your own voice / hands free interface.
ehnto 4y ago
I do a lot to reduce context switching even on the small scale, and reducing how many times I switch to the mouse is one of those things. I also reduce screen switching, "alt-tabs" or workspace switching, for that I use a tiling window manager (i3). Since it's impossible to avoid web browsing for work, I also use Vimium to bring more keyboard into that process. I would consider my approach "low-mouse" rather than no mouse. Certainly not no-keyboard. Although not quite voice activation, if you can type then typing is concise and accurate, which makes it quick. Does anyone have any experience comparing this to Vim? Especially for Vimium (chrome extension, not Vim), I consider it a superpower, because it feels a bit like telekinesis. I can just look at things and make them happen with my keys, no need to navigate toward them.

175 points by yewenjie 4y ago

akihitosan 4y ago

Imho text editing speed is not an issue for programmers.

eimrine 4y ago
And TDD is bad because it requires too much of typing /sarcasm
- marginalia_nu 4y ago
  TDD is bad because it introduces distracting busywork. The actual typing isn't the actual problem. In terms of just typing, I could probably produce something like 50,000 lines of code in a 40 hour work week in a moderately verbose programming language. In practice I'll probably crank out somewhere between 1000-5000. It's the thinking that is slow. It's made even slower by adding additional tasks and context switching.
  - demopathos 4y ago
    The average professional software engineer produces around 10 LOC per day.
comfypotato 4y ago
It’s more complicated than that I think. You don’t spend much time typing, but how much does the time you spend (and the cognitive load of) typing deconstruct the mental architecture you’ve created to solve a problem? This is at the core of why vim/others are still so popular. The micro-gains in typing efficiency translate to much more efficient problem solving because you don’t “lose your place” within your mental process. I think the same would extend to cursorless.
- magicalhippo 4y ago
  I find typing can be helpful. I think more about certain aspects of the problem as I type, so I often discover edge cases I hadn't thought about or potential performance pitfalls. I guess it's a bit like navigating using only the map in the war room vs navigating on the ground without a map. The ideal route comes through a combination of both, and typing allows for the latter point of view. Though after typing for a while, I'll take a step back and think about the overall solution again, going over it in my mind. Then back to typing. That said, I gotta have a responsive editor.
kwijybo 4y ago
I admit this a weird edge case, but a few years ago I fractured my left elbow and could only use one hand. When typing one handed, typing speed was a bottleneck.

Jemm 4y ago

I find it annoying that they are not using the standard phoenetic alphabet

ParetoOptimal 4y ago
As someone who has voice coded full-time for a long period of time, that would get quite annoying. Notice that some users even start using Talon's noise support. I even changed "scroll up" to "suh" because I said it 300 times per day.
yewenjie 4y ago
I think the main reason is too easily distinguish (to the program) between letters - and I guess to also have shorter words.
- OJFord 4y ago
  That's also the NATO phonetic alphabet's raison d'être, where a lot more's on the line; a lot more research time and money has been spent on it over the years. Air, Jury, and Quench in particular strike me as ambiguous examples NATO would never go for. Krunch is just baffling. Plex breaks the scheme followed by every other (that it's the first letter in the word).
  - lunixbochs 4y ago
    Talon's alphabet is extremely carefully designed. There's de bruijn sequence style redundant phonetic information in each letter allowing the single syllable words to be mashed together at high speed without pauses (even allows for a bit of slurring to fit more than one letter into a syllable). I can type 60wpm with it, see me spelling a last name around 0:40 in this video for typical usage [1]. It's also carefully tested (with an American accent) to not jam up your mouth / tongue when you say the letters in any order. You can remap it to whatever you want, and a few users choose either NATO or a custom alphabet. But most stick with Talon's with minor changes (e.g. they might change a specific word if it's harder to say clearly in their accent, or if their mic or environment makes it more difficult to pick up specific sounds). You can also use the English alphabet for spelling anywhere you can say a word, e.g. "say hello" and "say H E L L O" both work the same. This isn't a voice assistant where you belt out a command once in a while then move on with your day, and this isn't a low fidelity radio link. These words may be said tens of thousands of times a day. Using three-syllable NATO words is slow and exhausting. It's common for voice programming systems to define their own alphabet based on a truncated NATO alphabet. I went further because I wanted to be able to use vim and play DCSS at very high speeds with my key commands instead of binding a bunch of app specific custom commands. [1] https://twitter.com/lunixbochs/status/1378159234861264896 https://twitter.com/lunixbochs/status/1378159234861264896

dag11 4y ago

This kind of work is also fantastic for those who have limited mobility or use of hands. On top of that, it's also a comforting peace of mind for the rest of us that should we ever e.g. lose our hands in a car collision, we might still have a successful path forward in our programming careers and passions. Reminds me of a talk I once saw where a developer developed arthritis and couldn't program effectively anymore, so he developed an almost "verbal vim" plugin for vim (or was it emacs?) that allowed him to write/edit code quickly using verbal utterances. I forget what that project was, but I wonder how these two compare.

kingcoleman 4y ago
I was very inspired by a blind programmer who uses text to speech at inhuman speeds. https://www.parhamdoustdar.com/2016/03/27/autobiography-blind-programmer/ https://www.parhamdoustdar.com/2016/03/27/autobiography-blin... That's perhaps an extreme example of what is possible but it's incredible what people can adapt to.
jffry 4y ago
Are you thinking perhaps of this [1] PyCon talk by Tavis Rudd, who had RSI and a rotator cuff injury? He demos a voice command system that's all short nonsense syllables, including with a few humorous video clips interspersed, and then explains it after the demo. [1] https://www.youtube.com/watch?v=8SkdfdXWYaI https://www.youtube.com/watch?v=8SkdfdXWYaI (demo starts 9 minutes in)
sterlind 4y ago
Cursorless is indeed fantastic for this! I used it extensively last winter/spring when I couldn't type.

LionTamer 4y ago

I wonder if there’s room for integration with Vim/Neovim or the plugins in that ecosystem - I feel like there’s a strong overlap between the “vim language” of text objects, motions, etc and the structure editing this project is working with.

deleted 4y ago
[deleted]
auscompgeek 4y ago
We have plans for adding a keyboard interface [0] at least, though it's still an open discussion. If you're after straight Neovim support though, perhaps nvim-treesitter-textobjects [1] might tickle your fancy? [0]: https://github.com/cursorless-dev/cursorless/issues/710 https://github.com/cursorless-dev/cursorless/issues/710 [1]: https://github.com/nvim-treesitter/nvim-treesitter-textobjects https://github.com/nvim-treesitter/nvim-treesitter-textobjec...

t_mann 4y ago

Looks great! I've been dreaming about something that lets me use a headband and reads brainwaves to interact with a computer for a long time. Every 4-5 years I look into what's been happening on that front, but so far it seems like we're not even close.

NJRBailey 4y ago
I am hoping for an eye-tracking mouse solution in the meantime, where looking at a portion of the screen and pressing a key on the keyboard will click rather than having to move a hand away from the keyboard to use a mouse, or wait for the analog stick mouse to reach the right point of the screen. Seems to be getting more plausible but still fairly expensive for a few years yet.
- lunixbochs 4y ago
  Talon has an eye tracking mouse, the necessary hardware runs around $250.
  - yep31 4y ago
    I would say that an eye tracking mouse is what OptiKey has, and for Talon it's more of an eye tracking+head pointing+voice clicking mouse. If you have tmj/vocal cords/neck problems you won't be able to use eye tracker as a mouse replacement.
    - lunixbochs 4y ago
      > for Talon it's more of an eye tracking+head pointing+voice clicking mouse > If you have tmj/vocal cords/neck problems you won't be able to use eye tracker as a mouse replacement. I disagree with these assertions. There are several options in Talon for eye tracking. Optikey is a nice suite of functionality, but it's Windows-only, while Talon is cross platform. I also have a few users who use the Optikey UI but prefer to use Talon's eye tracking to control it. I've certainly recommended Optikey to users. The updated direct control mouse mode in Talon 0.3 can be used without any head tracking at all as long as your targets aren't too small. The zoom mode explicitly uses no head tracking as well. To click or trigger the zoom mouse, you can make a pop noise, use voice commands, or a physical mouse button, or keyboard, or any keyboard emulating input device (such as a foot pedal), or use one of the dwell options.

downboots 4y ago

How many wpm?

vanjajaja1 4y ago

Very interesting. Editing code at the level of text is too low, ideally it would be raised to the level of thought. We don't think in terms of characters changing, we think in terms of higher level block constructs. It's not about the increased typing speed, it's about reducing cognitive load and making problem solving more enjoyable. Pairing with the Unison-lang style system of direct AST manipulation could be revolutionary, if the difficulty doesn't prove too high.

orzig 4y ago

I may not ever use it, but I love that such innovative interfaces exist. Huge congratulations to the team

soulofmischief 4y ago

Seems cool. However, I went down a several-hour rabbit hole trying to build non-documented dependencies and eventually gave up. The reliance on mono/skiasharp makes this a far less portable application, and to say there is Linux support seems disingenuous. Binaries only exist for Ubuntu, the binaries are said to work for non-ubuntu systems but nuget fails to install them. Building from scratch leads to several undocumented dependencies and I'm still receiving increasingly inscrutable build failures out of the box. Feels like 2005... My patience ran out. If anyone on an RPM-based system gets everything up and running, drop me a line.

lunixbochs 4y ago
You talking about Talon? It comes with all of the dependencies. Use ./run.sh to set the library path and ask on the Slack if there are any further issues. (fwiw it's not skiasharp itself, it's just the mono/skia repo, which is C++)
- soulofmischief 4y ago
  Thanks for the tip. Any reason run.sh isn't mentioned anywhere in the documentation? https://talonvoice.com/docs/ https://talonvoice.com/docs/
  - auscompgeek 4y ago
    Isn't it mentioned in the README in the tarball?
    - soulofmischief 4y ago
      The tarball, talon-linux.tar.xz, didn't come with a README, neither under root or /resources. I could certainly have broken run.sh open in vim (and should have), I could have looked into /lib and found the missing dependencies, but given that there was a binary literally called talon, the general expectation is this is the binary to be linked to my PATH and run directly. Using a run.sh is not necessary when the same could be achieved by making the talon file a shell script and renaming the binary to something else.

sillylaird 4y ago

The issue i have with this and will always with speech to text software. my voice will never pick it up unless im very clear and i take the time to prounouce a word or a chord?. I will have more luck writing in a stenograph keyboard or a chording keyboard then this. I like the fact that this technology assits for those who can have their voice be read by a computer. It will never work for me... Despite how modern a lot of this software has became.

pomatic 4y ago
You may want to look at your hardware setup. Modern (1-3 year old) laptops now generally have incredibly good echo cancellation microphone setups which massively improve recognition accuracy. If you are stuck with an older machine, try a USB based (integral ADC, the logical inverse of a DAC used for playing audio) lapel mic. These can be had off amazon for a tenner, but will infuriate you everytime you forget to take it off and get up from your chair... One last tip: My experience suggests the bluetooth microphone quality is not good enough for voice recognition tasks like this.
- sillylaird 4y ago
  thank you for the suggestion. i will try to look into it. it just at the same i was in highschool the technology wasn't as good for my voice. so i just learned how to type faster. but im willing to give it a shot and i can be a like "beta tester" or contributer if needed. i tend to use wired stuff since i do music production and am very picky on how stuff comes out when im finally mixing them.

viktour19 4y ago

Combine this with something like Copilot and you get the 10x programmer :)

cryptonector 4y ago

This is impressive. A bit like vi, but voiced.

sterlind 4y ago

Cursorless is fantastic, this saved my bacon last year when I couldn't use my hands for months. I donate to Pokey because I'll need this again in the future, though right now I'm typing okay. I think able-bodied programmers will probably get annoyed setting it up, setting talon up, learning the phonetic alphabet and basic commands, but like.. if your hands don't work, this can literally save your career. this is a disability aid for creating things, it's vital and empowering! the hard part IMO is getting back into the coding headspace you had when you could type. it's different speaking code, formatting it in your head before you say command phrases. everything's a little awkward, you can't get into the zone the same way.

foerbert 4y ago
I have some very mixed feelings about talon. I looked into it a decent bit when I started having hand issues as well, and never got anywhere useful with it. For one, it needs to be programmed to do anything. This gets you all the options, which is wonderful, but also is a bit of a problem when you're looking into because you can't, ya know, program. And it's not a simple thing to interact with. Lots of moving parts and various behind-the-scenes interactions. For another, there's virtually zero documentation. Like, officially none, and unofficially not much more than a few paragraphs. And to put the cherry on top, it's a closed source mess of a program. It's not easy to get much in the way of useful information out of it. You can poke around some in a Python REPL, but that'll basically just get you the interface. And somehow, it's one of the most useful sources of information, as at least you've got something to go on. And as mentioned, Python, so you don't have much in the way of language-level guarantees to try to rely on to figure stuff out either. Yes, there is the popular user config. To be clear, I'm not trying to be unkind to it. It managed to overcome a lot, and is very useful to lots of people for many of the reasons I stated above. But. It's also clearly got it's roots as one person's personal config that got popular. There is so much in there. It's basically it's whole own thing to learn on top of talon. In terms of learning from it, well, it doesn't seem trivial. It's basically it's whole own thing, but also how it all interacts is all talon. You're basically learning two separate yet intertwined systems at once and have to figure out what bits belong to what. It's also got various bits from multiple iterations of talon's interface, so the same thing may be accomplished multiple ways. It's actually quite well commented in some places about this fact, which is really nice. But it does make you wonder what isn't well commented... Ultimately I just gave up. It seems like a potentially hugely useful thing, but actually trying to use it required learning a bunch of stuff I wasn't otherwise interested in, figuring out the minutia of a complex unstable interface the hardest ways possible, and then spending a ton of time writing my own program/config. Aside from the time involved, the amount of typing required was prohibitive. So yeah. My feelings on talon... mixed, to say the least. Or most, judging by the above.

chaosprint 4y ago

Very cool. Love to see how the support for different groups of people is considered in the language level! It would also be interesting to test brain interfaces on this language some day. I am thinking about alternative input method for my project Glicol (https://glicol.org/ https://glicol.org/) quite some time. This work is definitely inspiring. I am also thinking about how can we better support the blind with some feedback. If you have some suggestions, please let me know here: https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol

cube2222 4y ago

The whole ecosystem around Talon[0] is fantastic. You'll probably want to start with [1] if it's your first time, it's a community-maintained talon config which is a good starting point for customization. The speech recognition engine built for it is great and the whole thing is free of charge, though I do recommend donating on Patreon [2]. Cursorless is a nice rethinking-from-the-ground kind of project. However, you can just as well keep using your current tools. I've used Talon on and off with Jetbrains IDEs when I had some wrist pain and the overall experience was great, with my coding speed being fast enough without much practice. When voice typing, autocomplete is also very useful, so the introduction of Copilot was an additional huge improvemet. But overall, the design of Talon, its config files, and the way this leads to very easy customizability, is really great. A bit like making your whole OS easily automatable and programmable (even if your OS isn't emacs /s). [0]: https://talonvoice.com/ https://talonvoice.com/ [1]: https://github.com/knausj85/knausj_talon https://github.com/knausj85/knausj_talon [2]: https://www.patreon.com/lunixbochs https://www.patreon.com/lunixbochs

phillco 4y ago
I agree. Talon is incredible. Before I discovered it, I was convinced that my career would be ended early by RSI, as it was getting slightly worse every year. Meanwhile voice coding looked like impenetrable wizardry from the few videos that I had seen. But after discovering it and getting through the learning curve (and there is one, that's probably the biggest reason why more people don't use it), it's something I look forward to using for the rest of my life. Even as my wrists start to gradually recover, being able to use voice/noise/eye tracking in tandem to the keyboard, allowing you to use the best for each purpose, is just so much more productive (not to mention, cool! fun!) that I doubt I'll ever go back.

deleted 4y ago

[deleted]

jstanley 4y ago

To anyone tempted to get into Talon: note that it is very much not open source - https://talonvoice.com/EULA.txt https://talonvoice.com/EULA.txt (except when you contibute code to them; in that case you implicitly license your code as open source, but theirs is still their property).

deleted 4y ago
[deleted]
lunixbochs 4y ago
Are you on Linux?
- jstanley 4y ago
  Yes. Re your deleted comment about "hostile forks": allowing users to run whatever software they want is precisely the point of open source. It's not "hostile" to have different opinions to you. The project looks cool nonetheless, but it is still true that if the users don't control the software, the software controls the users.
  - lunixbochs 4y ago
    I'll repost that in sibling. I just realized that I'd rather have people saying "ah, typical of proprietary software to not support Linux" than "your work has no value because I can't compete with you using your own code". I'm not going to stop supporting Linux because of this, but it still seems like a backwards incentive. This is a business, not a pet project. I work on it full time and it is my meager source of income. I'm not convinced it is possible to make much of a living on open source. I do mean hostile when I say hostile. I have a particular project of mine in mind, where the fork renamed, split the user base, and lagged behind my changes. They later just reset to my HEAD (as I was making way more progress than the fork), then removed attribution in the docs and solicited donations for the work I had done. I burned out on that project shortly after. > if the users don't control the software, the software controls the users Talon is free, runs offline, cares about user privacy and security, and gives an incredible amount of control to the user. The entire voice interface and behavior is defined in the user's code. There are several open source projects in this space and none of them are anywhere close to Talon on making it easy to build and customize your own voice / hands free interface.

ehnto 4y ago

I do a lot to reduce context switching even on the small scale, and reducing how many times I switch to the mouse is one of those things. I also reduce screen switching, "alt-tabs" or workspace switching, for that I use a tiling window manager (i3). Since it's impossible to avoid web browsing for work, I also use Vimium to bring more keyboard into that process. I would consider my approach "low-mouse" rather than no mouse. Certainly not no-keyboard. Although not quite voice activation, if you can type then typing is concise and accurate, which makes it quick. Does anyone have any experience comparing this to Vim? Especially for Vimium (chrome extension, not Vim), I consider it a superpower, because it feels a bit like telekinesis. I can just look at things and make them happen with my keys, no need to navigate toward them.