10 ms·
Sikuli: Automate Anything You See on Screen
- brudgers 10y agoRelated: https://news.ycombinator.com/item?id=11859996 https://news.ycombinator.com/item?id=11859996
- thedeerchaser 10y agoI love sikuli! Saved me a lot of time in the past
- eb0la 10y agoFun fact: if you want to click() on something that appears more than once, Sikuli will choose it at random. Some time ago I was demoing a script to my boss and the second time I run it, it clicked on a different place with the same text (and fortunately with the same destination). We were shocked because the link wasn't where we expected to be (Ciscosecure ACS 3.x).
- erikcw 10y agoThe "region" feature helps protect against this.
- jakozaur 10y agoGreat project, but it hasn't changed since 2010? Maybe we should add it to the title? I was hoping that image based testing would eventually overtake Selenium/Appium for web/mobile testing. In principle there are several advantages: + It works the way human works. If your button id has change from #submit to #submit-application, your UI test should work just fine. However, if your button became one pixel at top-left corner, test should break. Right now most frameworks do the opposite. + If it's based on visual, it would be easier to maintain tests. E.g. Your button change color from light grey to dark grey, would you like to update your tests. + Tests would be much easier and faster to read and write. Even less technical folks like current manual QA testing can contribute to automate testing tools.
- charford 10y agoYou might be interested in http://www.sikulix.com/ http://www.sikulix.com/, which was forked from this project and is more up to date.
- adamhepner 10y agoOh, you beat me to it. But the good news is, that SikuliX maintainer seems to be actively developing the next major release version now, and last release is not that old (end of 2015, where the technology don't have to change that rapidly)
- hatsunearu 10y agoLink is a 404.
- blacksmith_tb 10y agoA better link would be https://github.com/RaiMan/SikuliX-2014 https://github.com/RaiMan/SikuliX-2014
- dack 10y agoI mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. It's too brittle to rely on exact pixel locations in the face of added features and small design tweaks. Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy.
- tlarkworthy 10y agoThe visual matchers are fuzzy matchers, not pixel perfect.
- jakozaur 10y agoIt can be some nice fuzzy algorithm with learning capabilities. E.g. if you change background, but button is in same place, you can just update the test.
- patrickk 10y agoAnother great project in a similar vein is Pulover's Macro Creator, combined with Autohotkey: http://www.macrocreator.com/ http://www.macrocreator.com/ http://ahkscript.org/ http://ahkscript.org/ Sadly Windows only. It's one area where Windows has a leg up on Macs.
- kawera 10y agoKeyboard Maestro is a good option for Macs. Fake is another one but only for automated web browsing. I've been using both for years. http://www.keyboardmaestro.com/ http://www.keyboardmaestro.com/ http://fakeapp.com/ http://fakeapp.com/
- patrickk 10y agoThanks for this!
- hobs 10y agoI am primarily a Windows user, but Automator blew me away when I first saw it a few years ago(mainly due to the OSX Applescript dictionaries? which it could access meant it had a lot of additional functionality.) A cursory glance at macrocreator doesn't really give me the impression of what it does better, do you have any additional insight?
- patrickk 10y agoEase of use is a big one. I tried used Automator for simple tasks in the past, and it's got a complicated syntax and a steeper learning curve compared to AHK which meant I gave up. Look at the list of keyboard commands for an example of what I mean: https://autohotkey.com/docs/KeyList.htm#Keyboard https://autohotkey.com/docs/KeyList.htm#Keyboard Another example, this is a macro to jump to the end of a file name in windows explorer and append today's date, then jump to the next item in the list (F3 is the trigger): F3:: Macro1: Send, {Alt}{f}{m}{End} ; shortcut to rename, jump to end of filename SetKeyDelay, 0 SendRaw, _09-06-2016 ; this is what is sent Send, {Enter}{Down} ; jump to next file Return This is extremely readable and easy to understand. Perhaps it's possible to do pixel searches in Automator (click on certain buttons for example), but AHK + Pulover's Macro creator make this extremely easy - this is why I mentioned both, as these combined make a perfect drop-in replacement for Sikuli - and they're far more stable than Sikuli, which was quite unstable when I tried it at year or two ago.
- arca_vorago 10y agoSikulix is the current fork, and I have used it to surprising effect in all kinds of situations, really awesome as long as you can ignore some bugs, but the good news is the dev is super responsive to reports.
- hyuuu 10y agoI used this to automate my unemployment benefits, no joke. You have to keep reporting hours (every week I think), so I just set this up and run it, the closest thing I get to having a piece of software that makes money #sosad
- fit2rule 10y agoCongratulations. You've made Kafka proud! :)
- gk1 10y agoDid you set it up once and set it on a schedule, or did you have to run it each time? I'm curious if the tasks can be scheduled in advance, and how.
- popey456963 10y agoI managed to setup a schedule with a simple batch script + Windows Task Scheduler (OS: Windows). Imagine you could do the same on other OS'
- hyuuu 10y agoI ran it manually, and yes it can be automated even further but I'd like to keep an eye on the process + the sense of gratification of double clicking an icon == making it rain $$$
- fizzbatter 10y agoDon't you have to report application/locations/etc too? Or did you actually do the applications and log them in some type of database (flatfile/etc), and have this script auto fill in the benis every week?
- SteveNuts 10y agoIn my state you have a few weeks or a month of grace period before they start to require resumé and interview skill classes, as well as you reporting on any applications you filled out.
- eirikb 10y agoMade a small bash script to do something similar: https://gist.github.com/eirikb/c189a7d8406b2897dad0e86086be1b23 https://gist.github.com/eirikb/c189a7d8406b2897dad0e86086be1...
- yoo1I 10y agoI ... was expecting something more complicated than that. What a pleasant surprise. Thanks you for sharing!
- roddux 10y agoThis is really nice, great hack.
- CaptSpify 10y agoNice. Any demos of it in action?
- eirikb 10y agoCertainly, it should be pretty easy. Here is a demo: http://i.imgur.com/1BpMKpe.gif http://i.imgur.com/1BpMKpe.gif Script is here: https://gist.github.com/eirikb/ac8196beb0b57577a8fc47eb18427bd4 https://gist.github.com/eirikb/ac8196beb0b57577a8fc47eb18427... Images are here: http://imgur.com/a/Uv3NH http://imgur.com/a/Uv3NH What it does: Opens Google translate in Chrome, inserts some Norwegian text, copies the translated text and inserts it into this comment. To make the images I used gnome-screenshot, it let me grab areas.
- flarg 10y agoSikulix is great and comes with a Python IDE where you drag and drop screen captures - so a command to click on an icon has the icon itself written into the code. Robotic process automation (e.g. Blue Prism) is coming to the fore in commercial settings - a mixture of API and computer vision (probably a horrible mix) so there's a market out there for developing something commercially.
- leni536 10y agoWell, this is certainly more advanced than the xmacro* utilities. Once I had to use xmacro to do stuff like this, it's a good idea to dedicate a separate X server in the background though so you don't have to leave your other tasks while it's running.
- arank 10y agoI used Sikuli to automate app downloads from app store. It opens iTunes, goes to the featured apps page, enters the password if required and downloads 10 new featured apps every day.
- jimmcslim 10y agoWonder if an integration with Screenflow or Camtasia is possible?
- Shish2k 10y agoFWIW I used sikuli(x) and found it very useful, but I have a chronic allergy to Java[1], so I've been re-implementing the same API in pure Python --> https://github.com/shish/sikulpy https://github.com/shish/sikulpy [1] more specifically, I found sikuli scripts pretty painful to debug, since neither java debuggers nor python debuggers seem to work quite right.
- ludamad 10y agoHow do you tackle the debuggablity problem?
- Shish2k 10y agoI'm just using standard python tools (pudb and pycharm depending if I'm running on a remote server or locally). Also being pure python makes it easier to stick instrumentation into the core library when I need it :)
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- holtalanm 10y agoThis is just plain incorrect. Yes, the original API was written in java, but Sikuli scripts are written in Jython. I did an entire automation project using Sikuli written in jython without any hiccups a couple of years ago.
- Shish2k 10y ago> This is just plain incorrect. Did you somehow manage to read my comment and interpret it as "Sikuli scripts are written in java"? Because that's not at all what I said (or meant) :P Sikuli itself was (and still is) written in java - which I find a pain to work with when I want to modify the internals of the library; and then scripts are written in jython, which means I don't have access to my cpython-specific tools and extensions.
- SpaceCadetJones 10y agoReminded me of the SCAR tool many of us used to use back in the day for botting on Runescape. Will have to look into this for QA automation, as Selenium isn't always too fun when you've got a hairball of ASP.NET WebForms, Vanilla JS, and Angular for a code base.
- kaitnieks 10y agoSurprised to see SCAR mentioned. I have some connection to it ;) I still use it now and then to automate some flash games. Helps me to relax.
- SpaceCadetJones 10y agoWow, it really is a small world, I can't believe you replied to my comment. You're one of the only aliases I still recognize from those days as I was about 12 at the time. SCAR and the other scriptable clients were a huge spark for my interest in programming as a kid. Seeing something that could automate actions in a game I played all day blew my mind and I had to learn how it worked. Cheers to you for that!
- jzymbaluk 10y agoWhen I was working at a video editing shop, I used SikuliX to automate a lot of common, repetitive tasks (like inputting chapter names to chapter markers in Adobe Premiere). It's really great for automating easy repetitive tasks
- brunoqc 10y agoAnyone knows if this could be used with Go or Rust? like a c lib.
- holtalanm 10y agoGreat tool. Used it to automate SAP in a project a couple of years ago. Was using it with a .NET framework though, so it is was a total pain. That led to me writing Sikuli4NET about a year later. https://sourceforge.net/projects/sikuli4net/ https://sourceforge.net/projects/sikuli4net/
- projectileboy 10y agoI once worked on a project where we needed a tool like this. Sikuli definitely worked, but for us it only handled relatively simple cases, and it felt a bit like a science project (at the time, anyway... this was 2010). Good commercial alternatives at the time included eggPlant and T-Plan.
- darkFunction 10y agoUsed Sikuli for some casino fairness testing https://pbs.twimg.com/media/B4wlbnhIAAEti1z.jpg:large https://pbs.twimg.com/media/B4wlbnhIAAEti1z.jpg:large
- jawshie 10y agoI'd love to read more about this. Do you have a writeup somewhere?
- om42 10y agoUsed Sikuli for automating systems testing a few years ago (last I heard they stopped working on it). We had to use a VNC because it was affecting system performance. Overall really good and glad SikuliX was able to continue development of it. I used it again a few months ago and there same issues with the IDE and some other setup/runtime bugs exist that existed a few years ago. Otherwise it has huge potential for automating QA/systems testing if someone takes the time and has the resources to do so.
- paulryanrogers 10y agoIn my experience Sikuli/X was too slow for image search on high definition desktops. UIs also tend to change more than keyboard shortcuts.
- Contraptor 10y agoWhen I started at my current job (back in 2011) they were using Tevron's CitraTest product to automate software hosted on citrix. The first task I was given was to see if we could find an alternative to CitraTest, well Sikuli was the answer. All of the existing code was written in VB.NET, but with a simple wrapper class and use of XML-RPC, we dropped CitraTest completely for a free MIT-Licensed software. It has worked great for years actually.
- WaxProlix 10y agoAlso great for automating repetitive/grindy tasks in computer games! (Not a bot, promise)
- brazzledazzle 10y agoLast time I tried it on OS X the experience was pretty painful but it did a decent job when all of the pieces fell into place. Has anyone tried both Sikuli and RobotJS [1] and can compare/contrast them? [1] https://github.com/octalmage/robotjs https://github.com/octalmage/robotjs
- kon1 10y agoWe use Sikuli for automatiom tests on both Windows and Mac. Other than occasional issues with its image recognition features it works great and serves us well. We have Jenkins automation jobs spin up VMs in VirtualBox, load build artifacts on the VM, and run Sikuli tests. All test output gets pulled back into the Jenkins output and parsed for successes/failures. Makes it possible for us to test across different OSes use the same Sikuli scripts.