5 ms·

Neverclick: Desktop application for performing mouse actions with your keyboard

54 points by thunderbong 3mo ago

squigz 3mo ago
QMK-powered keyboards can do this out of the box
- granra 3mo ago
  I mean, it can make mouse movements and clicks but you can't be suggesting it can do on-screen hints, which makes it a way better experience.
  - squigz 3mo ago
    I'll admit I didn't look too closely at TFA :) But yeah, that part looks neat
- pelagicAustral 3mo ago
  thanks for the heads up, I have a keychron gathering dust at home, so I'll test this later on. favved the thread anyway, since I'm interested, but will wait for the macOS version.
  - okeefe77 3mo ago
    There is a MacOS app called ShortCat that gives similar functionality.
granra 3mo ago
This looks cool! How is the planned Linux support (the website says "Coming soon for Linux") suppose to work? X11 only? Wayland? Gnome only (or any other DE)?
- gen2brain 3mo ago
  I am guessing wp_pointer_warp_v1 in Wayland, but not sure if all compositors support that. And that is just for moving the cursor. Can we "send" a wl_pointer click like that?
  - uukelele 3mo ago
    ydotool
DarkIye 3mo ago
Nice, this is like Vimium link hints but for everything. So long carpal tunnel!
deviation 3mo ago
My problem with these types of apps will always be that most software it interacts with will unfortunately be designed without accessibility in mind. I've trialed a handful of these mouse-less programs before and I always run into the same hurdles. E.g. where one or more of my apps have sliders which I can't interact with using clicks instead of drags (so I can't interact with it at all), or needing to click things like my MacOS dock, or even resize a window.
- silon42 3mo ago
  What is needed is "Mouseless Friday" for UI designers/developers.
- lazov 3mo ago
  Hello, I'm the developer behind Neverclick. Neverclick doesn't use accessibility apis. It uses local computer vision so it works in every app. I've had a poor experience with accessibility apis, they're clunky, slow, and unpredictable, and as you said many apps aren't built with accessibility in mind. With computer vision you don't have to worry about that. This might sound strange, but the cv system in Neverclick actually runs significantly faster than UI Automation (which is the accessibility api on windows) in nearly all of my personal tests. I can't believe that accessibility apis are so poorly optimized that raw pixel analysis is faster lol. Also, I have users that use Neverclick on 10 year old hardware and they tell me that the cv runs instantly whereas UI Automation is super laggy for them. Also, accessibility apis typically don't give you useful bounding boxes for text in text editors, this is another advantage of the cv approach. I really wish the title of the post was "Desktop application for performing mouse actions with your keyboard using computer vision" so that it's more obvious that it uses cv, although it's my fault since the current title is how I have it in the readme and they probably just copied that.
andy55a 3mo ago
This is really nice, great job! I built something similar for myself on MacOS recently. Started off as a window manager then began adding keyboard navigation features as I was unhappy with other tools (vimium, shortcat etc). Hint mode as a feature being able to click all active links on the screen is great and very useful. Two features I've added recently that have made never using a mouse much more feasible: 1. Search mode - Search for any text on the page, jump to it and then use vim keybindings to select text from there and copy => I use this 20+ times a day 2. Scroll mode - Any scroll area on the page is highlighted with a hint, but when selected it clicks the scroll area and allows you to use keybindings like j/k/gg/G to scroll up and down. I loved this from vimium and wanted to everywhere Feel free to steal those :) https://github.com/andyhmltn/nflow https://github.com/andyhmltn/nflow
AyanamiKaine 3mo ago
Slightly related. On wayland based linux desktops this would never work because an app is not allowed to hijack the mouse pointer to dispatch events. I believe only the compositor is allowed to do that. I know, because I wanted to automate a GUI task using PyAutoGUI and it wouldnt work on wayland no matter what.
- AlienRobot 3mo ago
  I wonder if blender works on wayland, because it has a feature that makes the mouse cursor "loop" across the window if you are dragging something.
  - jcelerier 3mo ago
    The protocol for this has recently been accepted, now it has to be implemented by every compositor
    - AlienRobot 3mo ago
      That sounds like hell. All this just to set the X, Y of the mouse pointer?
      - toast0 3mo ago
        > That sounds like hell. Wayland isn't an eternal punishment, it's just a protocol. :p
      - jcelerier 3mo ago
        Not even, this was deemed too insecure. So now we have to have one protocol per-use case instead of something that allows to handle easily all the use cases we already know exist - the most stupid thing possible
  - AyanamiKaine 3mo ago
    That features works completly fine when I use Blender(5.2) and wayland cant say if older versions might not work.
ur-whale 3mo ago
Windows only. Github has no source code. Pass for me.
- bobajeff 3mo ago
  Yeah after I read this I looked at the "code". It's a lot of placeholder.cpp with int main(){return}; This is so very sketchy.
krembo 3mo ago
Nice work, but how is this different from Shortcat?
- lazov 3mo ago
  Most software in this space relies on accessibility apis, which means it only works in apps that have accessibility support. Neverclick uses computer vision. There are several advantages to using computer vision, the biggest one is that it works in every app. It also produces really good bounding boxes around text so it's great for editing text. And counterintuitively, it runs significantly faster than the traditional accessibility api approach. The hints show up instantaneously, even on old hardware.
- drdexebtjl 3mo ago
  The Mario Kart youtuber?
gorgoiler 3mo ago
I bought an iBook G3 specifically for running Debian back in the early 2000s. It worked absolutely perfectly and one of the huge features I miss was that the small option key to the right of the space bar was converted to a middle mouse button :)
wingi 3mo ago
I tested also https://omarchy.org/ https://omarchy.org/ ...
okeefe77 3mo ago
I understand the app is only installable on Windows, but what is the thinking behind not allowing downloads on non-Windows devices? I was going to save a copy of the installer to my Google Drive from my phone and the download button doesn't work. Seems like an odd choice that could hinder adoption. (This is partially an assumption on my part as I currently don't have an unrestricted Windows machine to verify, but considering the non-functional button and "download on Windows" message it displays it seemed safe to assume)
- elzbardico 3mo ago
  I think you can get the source directly from github, probably even a pre-compilled installer from the releases page in the github project.
  - okeefe77 3mo ago
    You cannot. No releases and all the code files are "placeholder.cpp"
    - FinnKuhn 3mo ago
      Why even have a github if you don't actually publish anything there. To suggest that you are trustworthy and associate yourself with open source or source available without having to actually do it? Planning to do it, but being ashamed of your code? I don't know, but I definitely find it strange.
      - throw1234567891 3mo ago
        Who is “you” in this case?
        FinnKuhn 3mo ago
        The person publishing the software. Who that is I don't know as it is not displayed anywhere on the website and the the github repo is published by "LazoVelko". Who this is I don't know either. For all I know they are a russian hacker and a future version of this software is going to add an exploit. That is why you should probably think twice about downloading and installing software published by an anonymous person without the source code being available.
deleted 3mo ago
[deleted]
zerr 3mo ago
Developing a muscle memory is the main aspect of keyboard usage. Does this app account for this? i.e. how repeatable are those key-shortcuts? Seems like it's random.
- throw10920 3mo ago
  It seems to implement the Avy (https://github.com/abo-abo/avy https://github.com/abo-abo/avy) model of input. It can't be improved very much, because when dealing with dynamic data, you can't make things repeatable. The idea of "muscle memory" for things that change is kind of fallacious. The best you can do is ensure some predictability in the subset of keys you're using for element selection.
- lazov 3mo ago
  Neverclick dev here. It's not random, a lot of thought went into how the key combinations are generated and assigned. When Neverclick generates the key combinations, it prioritizes keys in the center of the keyboard. It then sorts the key combinations from most to least convenient (so for example JJ and FF would be at the top of the list). It then assigns the most convenient key combinations to UI elements closest to the center of the screen because that's where people are most likely to want to perform a mouse action. As the other commenter already said, with dynamic data, you can't really make things repeatable. However, it's still really cozy to use and way better than if it were randomly generated and assigned (which is how it used to work before it was updated to the new system).
  - FinnKuhn 3mo ago
    One interesting thing would be to have a certain section of your screen always start with the same first letter so you can already intuitively start typing it while checking the 2nd letter.
amelius 3mo ago
> Neverclick uses computer vision so it works in all applications. Ok, how does it use CV? This is not explained. Does it track eye movements? Do I need a webcam, or two?
- benji-york 3mo ago
  My reading is that it uses CV to identify potential click targets on the screen.
- lazov 3mo ago
  It uses CV to analyze the pixels on your screen to detect clickable UI elements.
- Tcepsa 3mo ago
  Presumably it captures a screenshot and then uses CV on the resulting image to "see" the application and identify the various things that you might want to click.
amelius 3mo ago
Anyone with experience with this keyboard which has a built-in trackpoint: https://tex.com.tw/products/shinobi https://tex.com.tw/products/shinobi
axm__ 3mo ago
Why are there 4000~ empty commits with the message "development work"?
- lazov 3mo ago
  The commits are a mirror of the real codebase. My intention was to show people that I've been working on this for a long time and that it's not ai slop abandonware. But after reading some of the comments here I can see how it has raised more questions than it has answered.
unregistereddev 3mo ago
This reeks of likely malware. - Fake commit history - No source code, despite being hosted on github - Windows-only binaries available for download If I weren't on my work computer I would pull down those binaries to see what is in them. It sure smells fishy.
- lazov 3mo ago
  The commit history is a mirror of the real codebase, I wanted to show people that I've been working on this for a long time and that it's not vibe coded ai slop. In retrospect it seems that it was a mistake to do this. Neverclick is digitally signed with Microsoft Trusted Signing (now known as Azure Artifact Signing). I had to send them my address and ID to get my identity verified so that I'd get approved for it. Is there anything else I can do to put your mind at ease?
  - unregistereddev 3mo ago
    Microsoft Trusted Signing only tells me you were able to get a key. If this is intended to be an open source project, transparency would go a long way. The source code and actual history will make a big difference. If this is not intended to be an open source project, then hosting it on a source control system is misleading. Not releasing source code is fine, but then you need to build trust. Is it paid software? Is there a company or a known name behind it (in which case, can you host it on that entity's website)? At minimum, toss up a website that describes what the software does, what the usage terms are, and whether it collects information about user systems and/or uploads data anywhere. Bonus, scan it with Windows Defender and a couple other malware scanners. Advertise that it scans clean.
deleted 3mo ago
[deleted]
y3owk1n 3mo ago
[dead]

54 points by thunderbong 3mo ago

squigz 3mo ago

QMK-powered keyboards can do this out of the box

granra 3mo ago
I mean, it can make mouse movements and clicks but you can't be suggesting it can do on-screen hints, which makes it a way better experience.
- squigz 3mo ago
  I'll admit I didn't look too closely at TFA :) But yeah, that part looks neat
pelagicAustral 3mo ago
thanks for the heads up, I have a keychron gathering dust at home, so I'll test this later on. favved the thread anyway, since I'm interested, but will wait for the macOS version.
- okeefe77 3mo ago
  There is a MacOS app called ShortCat that gives similar functionality.

granra 3mo ago

This looks cool! How is the planned Linux support (the website says "Coming soon for Linux") suppose to work? X11 only? Wayland? Gnome only (or any other DE)?

gen2brain 3mo ago
I am guessing wp_pointer_warp_v1 in Wayland, but not sure if all compositors support that. And that is just for moving the cursor. Can we "send" a wl_pointer click like that?
- uukelele 3mo ago
  ydotool

DarkIye 3mo ago

Nice, this is like Vimium link hints but for everything. So long carpal tunnel!

deviation 3mo ago

My problem with these types of apps will always be that most software it interacts with will unfortunately be designed without accessibility in mind. I've trialed a handful of these mouse-less programs before and I always run into the same hurdles. E.g. where one or more of my apps have sliders which I can't interact with using clicks instead of drags (so I can't interact with it at all), or needing to click things like my MacOS dock, or even resize a window.

silon42 3mo ago
What is needed is "Mouseless Friday" for UI designers/developers.
lazov 3mo ago
Hello, I'm the developer behind Neverclick. Neverclick doesn't use accessibility apis. It uses local computer vision so it works in every app. I've had a poor experience with accessibility apis, they're clunky, slow, and unpredictable, and as you said many apps aren't built with accessibility in mind. With computer vision you don't have to worry about that. This might sound strange, but the cv system in Neverclick actually runs significantly faster than UI Automation (which is the accessibility api on windows) in nearly all of my personal tests. I can't believe that accessibility apis are so poorly optimized that raw pixel analysis is faster lol. Also, I have users that use Neverclick on 10 year old hardware and they tell me that the cv runs instantly whereas UI Automation is super laggy for them. Also, accessibility apis typically don't give you useful bounding boxes for text in text editors, this is another advantage of the cv approach. I really wish the title of the post was "Desktop application for performing mouse actions with your keyboard using computer vision" so that it's more obvious that it uses cv, although it's my fault since the current title is how I have it in the readme and they probably just copied that.

andy55a 3mo ago

This is really nice, great job! I built something similar for myself on MacOS recently. Started off as a window manager then began adding keyboard navigation features as I was unhappy with other tools (vimium, shortcat etc). Hint mode as a feature being able to click all active links on the screen is great and very useful. Two features I've added recently that have made never using a mouse much more feasible: 1. Search mode - Search for any text on the page, jump to it and then use vim keybindings to select text from there and copy => I use this 20+ times a day 2. Scroll mode - Any scroll area on the page is highlighted with a hint, but when selected it clicks the scroll area and allows you to use keybindings like j/k/gg/G to scroll up and down. I loved this from vimium and wanted to everywhere Feel free to steal those :) https://github.com/andyhmltn/nflow https://github.com/andyhmltn/nflow

AyanamiKaine 3mo ago

Slightly related. On wayland based linux desktops this would never work because an app is not allowed to hijack the mouse pointer to dispatch events. I believe only the compositor is allowed to do that. I know, because I wanted to automate a GUI task using PyAutoGUI and it wouldnt work on wayland no matter what.

AlienRobot 3mo ago
I wonder if blender works on wayland, because it has a feature that makes the mouse cursor "loop" across the window if you are dragging something.
- jcelerier 3mo ago
  The protocol for this has recently been accepted, now it has to be implemented by every compositor
  - AlienRobot 3mo ago
    That sounds like hell. All this just to set the X, Y of the mouse pointer?
    - toast0 3mo ago
      > That sounds like hell. Wayland isn't an eternal punishment, it's just a protocol. :p
    - jcelerier 3mo ago
      Not even, this was deemed too insecure. So now we have to have one protocol per-use case instead of something that allows to handle easily all the use cases we already know exist - the most stupid thing possible
- AyanamiKaine 3mo ago
  That features works completly fine when I use Blender(5.2) and wayland cant say if older versions might not work.

ur-whale 3mo ago

Windows only. Github has no source code. Pass for me.

bobajeff 3mo ago
Yeah after I read this I looked at the "code". It's a lot of placeholder.cpp with int main(){return}; This is so very sketchy.

krembo 3mo ago

Nice work, but how is this different from Shortcat?

lazov 3mo ago
Most software in this space relies on accessibility apis, which means it only works in apps that have accessibility support. Neverclick uses computer vision. There are several advantages to using computer vision, the biggest one is that it works in every app. It also produces really good bounding boxes around text so it's great for editing text. And counterintuitively, it runs significantly faster than the traditional accessibility api approach. The hints show up instantaneously, even on old hardware.
drdexebtjl 3mo ago
The Mario Kart youtuber?

gorgoiler 3mo ago

I bought an iBook G3 specifically for running Debian back in the early 2000s. It worked absolutely perfectly and one of the huge features I miss was that the small option key to the right of the space bar was converted to a middle mouse button :)

wingi 3mo ago

I tested also https://omarchy.org/ https://omarchy.org/ ...

okeefe77 3mo ago

I understand the app is only installable on Windows, but what is the thinking behind not allowing downloads on non-Windows devices? I was going to save a copy of the installer to my Google Drive from my phone and the download button doesn't work. Seems like an odd choice that could hinder adoption. (This is partially an assumption on my part as I currently don't have an unrestricted Windows machine to verify, but considering the non-functional button and "download on Windows" message it displays it seemed safe to assume)

elzbardico 3mo ago
I think you can get the source directly from github, probably even a pre-compilled installer from the releases page in the github project.
- okeefe77 3mo ago
  You cannot. No releases and all the code files are "placeholder.cpp"
  - FinnKuhn 3mo ago
    Why even have a github if you don't actually publish anything there. To suggest that you are trustworthy and associate yourself with open source or source available without having to actually do it? Planning to do it, but being ashamed of your code? I don't know, but I definitely find it strange.
    - throw1234567891 3mo ago
      Who is “you” in this case?
      - FinnKuhn 3mo ago
        The person publishing the software. Who that is I don't know as it is not displayed anywhere on the website and the the github repo is published by "LazoVelko". Who this is I don't know either. For all I know they are a russian hacker and a future version of this software is going to add an exploit. That is why you should probably think twice about downloading and installing software published by an anonymous person without the source code being available.

deleted 3mo ago

[deleted]

zerr 3mo ago

Developing a muscle memory is the main aspect of keyboard usage. Does this app account for this? i.e. how repeatable are those key-shortcuts? Seems like it's random.

throw10920 3mo ago
It seems to implement the Avy (https://github.com/abo-abo/avy https://github.com/abo-abo/avy) model of input. It can't be improved very much, because when dealing with dynamic data, you can't make things repeatable. The idea of "muscle memory" for things that change is kind of fallacious. The best you can do is ensure some predictability in the subset of keys you're using for element selection.
lazov 3mo ago
Neverclick dev here. It's not random, a lot of thought went into how the key combinations are generated and assigned. When Neverclick generates the key combinations, it prioritizes keys in the center of the keyboard. It then sorts the key combinations from most to least convenient (so for example JJ and FF would be at the top of the list). It then assigns the most convenient key combinations to UI elements closest to the center of the screen because that's where people are most likely to want to perform a mouse action. As the other commenter already said, with dynamic data, you can't really make things repeatable. However, it's still really cozy to use and way better than if it were randomly generated and assigned (which is how it used to work before it was updated to the new system).
- FinnKuhn 3mo ago
  One interesting thing would be to have a certain section of your screen always start with the same first letter so you can already intuitively start typing it while checking the 2nd letter.

amelius 3mo ago

> Neverclick uses computer vision so it works in all applications. Ok, how does it use CV? This is not explained. Does it track eye movements? Do I need a webcam, or two?

benji-york 3mo ago
My reading is that it uses CV to identify potential click targets on the screen.
lazov 3mo ago
It uses CV to analyze the pixels on your screen to detect clickable UI elements.
Tcepsa 3mo ago
Presumably it captures a screenshot and then uses CV on the resulting image to "see" the application and identify the various things that you might want to click.

amelius 3mo ago

Anyone with experience with this keyboard which has a built-in trackpoint: https://tex.com.tw/products/shinobi https://tex.com.tw/products/shinobi

axm__ 3mo ago

Why are there 4000~ empty commits with the message "development work"?

lazov 3mo ago
The commits are a mirror of the real codebase. My intention was to show people that I've been working on this for a long time and that it's not ai slop abandonware. But after reading some of the comments here I can see how it has raised more questions than it has answered.

unregistereddev 3mo ago

This reeks of likely malware. - Fake commit history - No source code, despite being hosted on github - Windows-only binaries available for download If I weren't on my work computer I would pull down those binaries to see what is in them. It sure smells fishy.

lazov 3mo ago
The commit history is a mirror of the real codebase, I wanted to show people that I've been working on this for a long time and that it's not vibe coded ai slop. In retrospect it seems that it was a mistake to do this. Neverclick is digitally signed with Microsoft Trusted Signing (now known as Azure Artifact Signing). I had to send them my address and ID to get my identity verified so that I'd get approved for it. Is there anything else I can do to put your mind at ease?
- unregistereddev 3mo ago
  Microsoft Trusted Signing only tells me you were able to get a key. If this is intended to be an open source project, transparency would go a long way. The source code and actual history will make a big difference. If this is not intended to be an open source project, then hosting it on a source control system is misleading. Not releasing source code is fine, but then you need to build trust. Is it paid software? Is there a company or a known name behind it (in which case, can you host it on that entity's website)? At minimum, toss up a website that describes what the software does, what the usage terms are, and whether it collects information about user systems and/or uploads data anywhere. Bonus, scan it with Windows Defender and a couple other malware scanners. Advertise that it scans clean.

deleted 3mo ago

[deleted]

y3owk1n 3mo ago

[dead]