4 ms·

Robotgo v0.42.0 released

75 points by vvvway 10y ago

notheguyouthink 10y ago
I wonder, could something like this be used for automated testing? Eg, UI testing is always a concern of mine - i like full end to end testing, but UI is difficult. I wonder if we could have somewhat machine learning trained image recognition bots click a Next button, without having to specifically provide the image. Making testing much less cumbersome, and more like "click button X, require 10 list items on screen", etc. Note that this package doesn't have any ML i believe, it was just an example since this package does have bitmap comparison it looks like.
- vvvway 10y ago
  Find an image on screen, read pixels from an image will be Update,Maybe use Perceptual hash algorithm and consider other algorithms.
- yorwba 10y ago
  You could try to integrate it with whatever a screen reader does to read the text on buttons to its user. As a bonus, all your UI tests double as accessibility tests.
  - vvvway 10y ago
    Thank you for your suggestion,I will think seriously about it or add it.
- sigmonsays 10y ago
  sikuli is a project that can find things on screen and (among other things) click it. its a pretty interesting project. Would be nice to integrate some of its logic into robotgo
  - vvvway 10y ago
    Its a indeed interesting project.
drakenot 10y ago
This looks like an interesting project. I was really big into the AutoIt community back in the day. I think being able to write programs that simulate user input is a great way to learn programming. It is an amazing thing to watch an automation script you wrote fly through controls, menus and actions of 3rd party programs as fast the program can keep up. I remember all the drama surrounding the split or "fork" of AutoHotkey from AutoIt and the subsequent changing of licenses to make future AutoIt releases closed source. Being able to write AutoIt-like scripts but in a cross-platform way would be great.
- vvvway 10y ago
  Yep,Can be directly packaged binary and easy cross-platform is the reason for choosed Go.
- pmoriarty 10y ago
  Why did AutoIt split from AutoHotkey? I hadn't even heard there was a split. Is there somewhere I can read more about this? Which is the dominant user input automation tool on Windows these days? I'd thought AutoHotkey was it, but maybe not?
  - vvvway 10y ago
    Some controversy ...
  - drakenot 10y ago
    It was the other way around. AutoIt is older than AHK. My understanding was that a developer wanted to add "hotkey" functionality to AutoIt and this didn't garner much interest from the AutoIt folks. So, the developer then decided to create AutoHotkey which was source compatible with AutoIt v2. The rest I'm a little hazy on. I believe that it was this fork that led to AutoIt switching to closed source. They both have been in development for years now after the switch and I don't believe that AHK maintains scripting compatibility with AutoIt any longer. They have diverged at this point.
Walkman 10y ago
IMO Go is not a great language for Desktop automation like this. A script language, not statically typed language fits better, because it makes it possible to iterate faster. Also these kind of automation scripts are not huge usually, or if they are, they are very simple. Also don't have to be fast. The only advantage I can think of Go has here is the easy deployability for multiple platforms, but that's not even possible, because the platforms has different UI-s with different sizes, so you can't write one script and expect to work on multiple platforms.
- vvvway 10y ago
  Can be directly packaged binary and easy cross-platform is good.
- serf 10y ago
  >but that's not even possible, because the platforms has different UI-s with different sizes, so you can't write one script and expect to work on multiple platforms. game UI usually stays the same across all platforms. As someone who has written bots for video games that are cross-platform, it's nice to abstract away things like win32api when dealing with just trying to move the HIDs. Plus if your language/software allows for pixel read-in (this does), I wouldn't blame the scripting tool for non-robust applications -- make your application a bit more intelligent. Don't just blindly click at preset coordinates and expect reliable/consistent results except for the most basic automations.
  - onmobiletemp 10y ago
    Thats interesring. What games did you work on? What kinds of bugs were you trying to find?
  - vvvway 10y ago
    Really?
- zzzcpan 10y ago
  I agree, it would be better to hook it up with a javascript interpreter (I think there is one for Go) and distribute binaries for all systems to run such automation scripts. Those who need more could just extend javascript API with custom functions and recompile, as they would probably need scripting too to make it useful.
senthilnayagam 10y ago
what openAI gym and universe do with VNC , we can do it locally with robotgo now
- vvvway 10y ago
  yep.

75 points by vvvway 10y ago

I wonder, could something like this be used for automated testing? Eg, UI testing is always a concern of mine - i like full end to end testing, but UI is difficult. I wonder if we could have somewhat machine learning trained image recognition bots click a Next button, without having to specifically provide the image. Making testing much less cumbersome, and more like "click button X, require 10 list items on screen", etc. Note that this package doesn't have any ML i believe, it was just an example since this package does have bitmap comparison it looks like.

vvvway 10y ago
Find an image on screen, read pixels from an image will be Update,Maybe use Perceptual hash algorithm and consider other algorithms.
yorwba 10y ago
You could try to integrate it with whatever a screen reader does to read the text on buttons to its user. As a bonus, all your UI tests double as accessibility tests.
- vvvway 10y ago
  Thank you for your suggestion,I will think seriously about it or add it.
sigmonsays 10y ago
sikuli is a project that can find things on screen and (among other things) click it. its a pretty interesting project. Would be nice to integrate some of its logic into robotgo
- vvvway 10y ago
  Its a indeed interesting project.

drakenot 10y ago

This looks like an interesting project. I was really big into the AutoIt community back in the day. I think being able to write programs that simulate user input is a great way to learn programming. It is an amazing thing to watch an automation script you wrote fly through controls, menus and actions of 3rd party programs as fast the program can keep up. I remember all the drama surrounding the split or "fork" of AutoHotkey from AutoIt and the subsequent changing of licenses to make future AutoIt releases closed source. Being able to write AutoIt-like scripts but in a cross-platform way would be great.

vvvway 10y ago
Yep,Can be directly packaged binary and easy cross-platform is the reason for choosed Go.
pmoriarty 10y ago
Why did AutoIt split from AutoHotkey? I hadn't even heard there was a split. Is there somewhere I can read more about this? Which is the dominant user input automation tool on Windows these days? I'd thought AutoHotkey was it, but maybe not?
- vvvway 10y ago
  Some controversy ...
- drakenot 10y ago
  It was the other way around. AutoIt is older than AHK. My understanding was that a developer wanted to add "hotkey" functionality to AutoIt and this didn't garner much interest from the AutoIt folks. So, the developer then decided to create AutoHotkey which was source compatible with AutoIt v2. The rest I'm a little hazy on. I believe that it was this fork that led to AutoIt switching to closed source. They both have been in development for years now after the switch and I don't believe that AHK maintains scripting compatibility with AutoIt any longer. They have diverged at this point.

Walkman 10y ago

IMO Go is not a great language for Desktop automation like this. A script language, not statically typed language fits better, because it makes it possible to iterate faster. Also these kind of automation scripts are not huge usually, or if they are, they are very simple. Also don't have to be fast. The only advantage I can think of Go has here is the easy deployability for multiple platforms, but that's not even possible, because the platforms has different UI-s with different sizes, so you can't write one script and expect to work on multiple platforms.

vvvway 10y ago
Can be directly packaged binary and easy cross-platform is good.
serf 10y ago
>but that's not even possible, because the platforms has different UI-s with different sizes, so you can't write one script and expect to work on multiple platforms. game UI usually stays the same across all platforms. As someone who has written bots for video games that are cross-platform, it's nice to abstract away things like win32api when dealing with just trying to move the HIDs. Plus if your language/software allows for pixel read-in (this does), I wouldn't blame the scripting tool for non-robust applications -- make your application a bit more intelligent. Don't just blindly click at preset coordinates and expect reliable/consistent results except for the most basic automations.
- onmobiletemp 10y ago
  Thats interesring. What games did you work on? What kinds of bugs were you trying to find?
- vvvway 10y ago
  Really?
zzzcpan 10y ago
I agree, it would be better to hook it up with a javascript interpreter (I think there is one for Go) and distribute binaries for all systems to run such automation scripts. Those who need more could just extend javascript API with custom functions and recompile, as they would probably need scripting too to make it useful.

senthilnayagam 10y ago

what openAI gym and universe do with VNC , we can do it locally with robotgo now

vvvway 10y ago
yep.