5 ms·

Great project, but it hasn't changed since 2010? Maybe we should add it to the title? I was hoping that image based testing would eventually overtake Selenium/

Great project, but it hasn't changed since 2010? Maybe we should add it to the title? I was hoping that image based testing would eventually overtake Selenium/Appium for web/mobile testing. In principle there are several advantages: + It works the way human works. If your button id has change from #submit to #submit-application, your UI test should work just fine. However, if your button became one pixel at top-left corner, test should break. Right now most frameworks do the opposite. + If it's based on visual, it would be easier to maintain tests. E.g. Your button change color from light grey to dark grey, would you like to update your tests. + Tests would be much easier and faster to read and write. Even less technical folks like current manual QA testing can contribute to automate testing tools.

charford 10y ago
You might be interested in http://www.sikulix.com/ http://www.sikulix.com/, which was forked from this project and is more up to date.
- adamhepner 10y ago
  Oh, you beat me to it. But the good news is, that SikuliX maintainer seems to be actively developing the next major release version now, and last release is not that old (end of 2015, where the technology don't have to change that rapidly)
- hatsunearu 10y ago
  Link is a 404.
- blacksmith_tb 10y ago
  A better link would be https://github.com/RaiMan/SikuliX-2014 https://github.com/RaiMan/SikuliX-2014
dack 10y ago
I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. It's too brittle to rely on exact pixel locations in the face of added features and small design tweaks. Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy.
- tlarkworthy 10y ago
  The visual matchers are fuzzy matchers, not pixel perfect.
  - jakozaur 10y ago
    It can be some nice fuzzy algorithm with learning capabilities. E.g. if you change background, but button is in same place, you can just update the test.
  - marxidad 10y ago
    you can adjust the fuzziness
  - mmebane 10y ago
    From my experience, the fuzzy matching works well enough to handle picking out text or icons on interfaces with semi-transparent backgrounds that change on scrolling (e.g., due to a parallax effect). It's pixel-based and not scale-invariant, though, so it won't automatically handle multiple zoom levels/resolutions.
- IanCal 10y ago
  > I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. True, although for some things I do still like a visual test. Your users may be used to clicking a button that looks a certain way in a certain place, and suddenly shifting that might be worthy of requiring a change to a test. You wouldn't want all your behavioural tests done like this though. > Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy. I agree. Perhaps there's a nice integration between the two? If you were able to give, say, a css selector and a template image rather than just one or the other to look for you'd be able to report these errors: * I can't find the button I was looking for, but there is something that matches the css selector [blah] that looks like this: [image] instead of [image]. Do you want to update the image in the tests? Y/N * I can't find the css selector [blah] but the button appears here [image] with [class] and [id], do you want to update the css selector in the tests? Y/N If the program is actually still fine then it'd be a quick auto-update, and if it's not then these bits of information are probably the first bit of debug you'd want to do anyway to figure out what broke the tests.
deleted 10y ago
[deleted]
mherrmann 10y ago
If you're only worried about your tests breaking because of #submit vs #submit-application you might be interested in http://heliumhq.com http://heliumhq.com. It's a commercial wrapper around Selenium and I'm one of the authors.
- lewich 10y ago
  What is the difference between other wrappers? E.g. Selenide. It looks like you are selling shitty wrapper, not more...
  - mherrmann 10y ago
    The only reason you say shitty is because you don't like that it's commercial. ¯\_(ツ)_/
geoelectric 10y ago
Automation lead with a specialty in GUI here, You do realize we standardize on testing to ID so that you can change everything else (location, parentage, etc.) because that's the stuff that benignly changes during development or for localization, right? The ID is supposed to be semi-permanent and not change in the normal case, so don't make trivial changes like that. Otherwise, we're fully capable of finding buttons by coordinate, parentage, attributes, whatever. It's just crappy practice to do so. We intentionally lock down some things and not others because we know how software development works and what changes are likely to indicate greater chances of bugs and what aren't. We also code to accommodate different resolutions, adaptive interfaces, visual changes for different languages--your button location and boundaries change when your English 5 character caption becomes a German 25 character caption--etc. As for the 1 pixel button, it's not just that it might be 1 pixel. If all you care about is static visual correctness, you can do simple bitmap comparison to get that, though that has its own issues. And we can build in basic implicit checks into our frameworks for sane boundaries, z-order occlusion, etc. It's common for control selection to also validate the control could actually be accessed by the user. The bigger problem is that the human interaction details might not work right, It might not visibly depress when you hit it, for example, or might have a hit target that doesn't correspond to its boundaries, or have a janky delay after tapping, or whatever. Even injecting the most user-like of user events won't always catch those so there's no substitute for actually exercising the interface. And since you're in there anyway, the visual checks become trivial and mostly automatic. On another subject, there's no way in hell I'll trust a system that takes guesses as to what it's looking at for a regression test suite, which is what nearly all GUI automation suites are. Even if I did use this tool for something less deterministic like model-based GUI fuzzing, I'd need to have another suite built that verified beyond a doubt that the button I expect to be there is there and at least superficially behaves how I expect it to behave. That requires discrete selection and determinism, not fuzzy identification and non-determinism. The point of a regression suite is NOT to try to make the tests pass, it's to verify that absolutely none of the assumptions I coded into it changed and alert me to manually explore that functionality if they have by failing. Then I can bless the new assumptions with an automation code change or file a bug and xfail the test until the bug resolves. But the assumptions are supposed to be crystalized. You can't trust an alert system unless you know it'd make its decisions to alert the way you would. Unpredictability isn't welcome there. In this particular case, an ID changing means you removed the control and added another one, conceptually (and possibly practically) speaking, so the automation is supposed to break to trigger a re-examination. Your own code might reference that ID too, you know, so just you having done that means you greatly increased the chances of there being new bugs.

charford 10y ago

You might be interested in http://www.sikulix.com/ http://www.sikulix.com/, which was forked from this project and is more up to date.

adamhepner 10y ago
Oh, you beat me to it. But the good news is, that SikuliX maintainer seems to be actively developing the next major release version now, and last release is not that old (end of 2015, where the technology don't have to change that rapidly)
hatsunearu 10y ago
Link is a 404.
blacksmith_tb 10y ago
A better link would be https://github.com/RaiMan/SikuliX-2014 https://github.com/RaiMan/SikuliX-2014

dack 10y ago

I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. It's too brittle to rely on exact pixel locations in the face of added features and small design tweaks. Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy.

tlarkworthy 10y ago
The visual matchers are fuzzy matchers, not pixel perfect.
- jakozaur 10y ago
  It can be some nice fuzzy algorithm with learning capabilities. E.g. if you change background, but button is in same place, you can just update the test.
- marxidad 10y ago
  you can adjust the fuzziness
- mmebane 10y ago
  From my experience, the fuzzy matching works well enough to handle picking out text or icons on interfaces with semi-transparent backgrounds that change on scrolling (e.g., due to a parallax effect). It's pixel-based and not scale-invariant, though, so it won't automatically handle multiple zoom levels/resolutions.
IanCal 10y ago
> I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. True, although for some things I do still like a visual test. Your users may be used to clicking a button that looks a certain way in a certain place, and suddenly shifting that might be worthy of requiring a change to a test. You wouldn't want all your behavioural tests done like this though. > Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy. I agree. Perhaps there's a nice integration between the two? If you were able to give, say, a css selector and a template image rather than just one or the other to look for you'd be able to report these errors: * I can't find the button I was looking for, but there is something that matches the css selector [blah] that looks like this: [image] instead of [image]. Do you want to update the image in the tests? Y/N * I can't find the css selector [blah] but the button appears here [image] with [class] and [id], do you want to update the css selector in the tests? Y/N If the program is actually still fine then it'd be a quick auto-update, and if it's not then these bits of information are probably the first bit of debug you'd want to do anyway to figure out what broke the tests.

deleted 10y ago

[deleted]

mherrmann 10y ago

If you're only worried about your tests breaking because of #submit vs #submit-application you might be interested in http://heliumhq.com http://heliumhq.com. It's a commercial wrapper around Selenium and I'm one of the authors.

lewich 10y ago
What is the difference between other wrappers? E.g. Selenide. It looks like you are selling shitty wrapper, not more...
- mherrmann 10y ago
  The only reason you say shitty is because you don't like that it's commercial. ¯\_(ツ)_/

geoelectric 10y ago

Automation lead with a specialty in GUI here, You do realize we standardize on testing to ID so that you can change everything else (location, parentage, etc.) because that's the stuff that benignly changes during development or for localization, right? The ID is supposed to be semi-permanent and not change in the normal case, so don't make trivial changes like that. Otherwise, we're fully capable of finding buttons by coordinate, parentage, attributes, whatever. It's just crappy practice to do so. We intentionally lock down some things and not others because we know how software development works and what changes are likely to indicate greater chances of bugs and what aren't. We also code to accommodate different resolutions, adaptive interfaces, visual changes for different languages--your button location and boundaries change when your English 5 character caption becomes a German 25 character caption--etc. As for the 1 pixel button, it's not just that it might be 1 pixel. If all you care about is static visual correctness, you can do simple bitmap comparison to get that, though that has its own issues. And we can build in basic implicit checks into our frameworks for sane boundaries, z-order occlusion, etc. It's common for control selection to also validate the control could actually be accessed by the user. The bigger problem is that the human interaction details might not work right, It might not visibly depress when you hit it, for example, or might have a hit target that doesn't correspond to its boundaries, or have a janky delay after tapping, or whatever. Even injecting the most user-like of user events won't always catch those so there's no substitute for actually exercising the interface. And since you're in there anyway, the visual checks become trivial and mostly automatic. On another subject, there's no way in hell I'll trust a system that takes guesses as to what it's looking at for a regression test suite, which is what nearly all GUI automation suites are. Even if I did use this tool for something less deterministic like model-based GUI fuzzing, I'd need to have another suite built that verified beyond a doubt that the button I expect to be there is there and at least superficially behaves how I expect it to behave. That requires discrete selection and determinism, not fuzzy identification and non-determinism. The point of a regression suite is NOT to try to make the tests pass, it's to verify that absolutely none of the assumptions I coded into it changed and alert me to manually explore that functionality if they have by failing. Then I can bless the new assumptions with an automation code change or file a bug and xfail the test until the bug resolves. But the assumptions are supposed to be crystalized. You can't trust an alert system unless you know it'd make its decisions to alert the way you would. Unpredictability isn't welcome there. In this particular case, an ID changing means you removed the control and added another one, conceptually (and possibly practically) speaking, so the automation is supposed to break to trigger a re-examination. Your own code might reference that ID too, you know, so just you having done that means you greatly increased the chances of there being new bugs.