3 ms·

Interesting tool! And you've definition given a lot of thought to many of the problems encountered while attempting to do UI testing automation. I've written m

by cmehdy 6y ago

Interesting tool! And you've definition given a lot of thought to many of the problems encountered while attempting to do UI testing automation. I've written my fair share of extensive selenium stuff (and appium, but.. let's forget about those painful memories) and one thing that I found fairly "easy' to add to the suites I had written was accessibility testing (using Deque's Axe[0] tools). It literally took a few lines of code and a flag at runtime to enable the accessibility report on my Jenkins/TestNG/Selenium combo. However WCAG is constantly changing and it's hard to keep up, and even Deque is not always up to date AFAIK. Do you have plans to have accessibility testing with your tool ? (even a subset of WCAG's rules) Another thing I've noticed is the jump in pricing from Free to 100 USD/month which goes from 30mins to 6hours. This might be steep for a team attempting to test out the validity of the tool against the competition - perhaps offering a first-month discounted trial or something like that would be appealing. I also haven't really seen if it is possible to enable concurrency (for example, is testing on three platforms at the same time for 10 minutes on that free tier possible? I would imagine so if one is doing the lifting with CI like Jenkins - but perhaps you have your own solutions). Tangentially related but you say your tests integrate with various CI solutions, does this mean one can extract the test results in a way that allows further processing and integration into other tools? (I'm thinking of the XMLs coming out of TestNG there). Lastly, I don't know if the time used is counted from the moment you start your VMs or browser starts, or the page is loaded, or the first step is done or something else. Clarifying that might help with a team's estimates (I had internal tests where the struggle was entirely on having the virtual environment/device/browser ready and the test was then a breeze, so the significant metric was the boot-up time). [0] https://github.com/dequelabs/axe-core https://github.com/dequelabs/axe-core

fitzn 6y ago
Thank you for the thoughtful comment! Let me know if I miss any points: 1) We capture all attributes for elements and although we don't yet let you fail a test if the attributes are incorrect, that's where we're headed. So, you could imagine enabling/disabling accessibility validation at the account or per-test level. This is great feedback. 2) Re: pricing - thanks for this idea! Pricing is still evolving, will keep in mind. 3) We support parallel execution at no extra charge; it's only test execution time. So if you want to spin up 10 tests at once, that's fine. (We may need to scale up our infra at the outset, obviously.) There is no hard cap on the number of concurrent tests. 4) Our API (https://reflect.run/docs/developer-api/documentation/ https://reflect.run/docs/developer-api/documentation/) is in the early stages, but includes endpoints for executing tests and fetching results for tests in JSON. 5) Runtime is the time the browser begins navigating to the page. You are not charged for time spent spinning up infrastructure or saving the test run. And just to be clear, you are not charged for any time spent recording tests. Thanks again!
- cmehdy 6y ago
  Thanks for answering those questions. I think it's all great news, and for customers it's reassuring to know that running time is really just about the test itself (which I noticed is in the FAQ, so sorry for wasting your time there). This makes me think about a couple more things: can clients select zones from which to run the tests? It's probably not a huge focus if your current target customers are in North America, but can become useful if a European company considers the tool and all loading times become noticeable. Or just if a company expects clients from anywhere and wants to test for that (which is basically the hardest bit for a small team, as it starts to involve having some devops experience to spin up VMs on the cloud in different zones and all that jazz). Another tangentially related setting would be the locale for tests execution and scenarios (I'm thinking of date formats, monetary symbols, etc.. with expectations being different if you're in the US or in the UK for example) The lack of cross-browser testing is something I can see scaring away some teams, especially with mobile being restricted to profiles of older iPhones only. IIRC over 70-80% of iPhone users are on the latest major iOS version. But it all ties in to other issues that I'm sure you're already dealing with (iPhone users using Safari, something like 20% of desktop users using Firefox/Safari, and the odd clients here and there who would have very specific annoying requirement like "I must be able to test on Internet Explorer - yes I know it's deprecated but my startup is B2B and I'm the small fish there"). Your docs might be short but they are already good and readable, especially for a target that might include a mix of pure devs and automation QAs/SDET/etc. All the best!

by cmehdy 6y ago

fitzn 6y ago

Thank you for the thoughtful comment! Let me know if I miss any points: 1) We capture all attributes for elements and although we don't yet let you fail a test if the attributes are incorrect, that's where we're headed. So, you could imagine enabling/disabling accessibility validation at the account or per-test level. This is great feedback. 2) Re: pricing - thanks for this idea! Pricing is still evolving, will keep in mind. 3) We support parallel execution at no extra charge; it's only test execution time. So if you want to spin up 10 tests at once, that's fine. (We may need to scale up our infra at the outset, obviously.) There is no hard cap on the number of concurrent tests. 4) Our API (https://reflect.run/docs/developer-api/documentation/ https://reflect.run/docs/developer-api/documentation/) is in the early stages, but includes endpoints for executing tests and fetching results for tests in JSON. 5) Runtime is the time the browser begins navigating to the page. You are not charged for time spent spinning up infrastructure or saving the test run. And just to be clear, you are not charged for any time spent recording tests. Thanks again!

cmehdy 6y ago
Thanks for answering those questions. I think it's all great news, and for customers it's reassuring to know that running time is really just about the test itself (which I noticed is in the FAQ, so sorry for wasting your time there). This makes me think about a couple more things: can clients select zones from which to run the tests? It's probably not a huge focus if your current target customers are in North America, but can become useful if a European company considers the tool and all loading times become noticeable. Or just if a company expects clients from anywhere and wants to test for that (which is basically the hardest bit for a small team, as it starts to involve having some devops experience to spin up VMs on the cloud in different zones and all that jazz). Another tangentially related setting would be the locale for tests execution and scenarios (I'm thinking of date formats, monetary symbols, etc.. with expectations being different if you're in the US or in the UK for example) The lack of cross-browser testing is something I can see scaring away some teams, especially with mobile being restricted to profiles of older iPhones only. IIRC over 70-80% of iPhone users are on the latest major iOS version. But it all ties in to other issues that I'm sure you're already dealing with (iPhone users using Safari, something like 20% of desktop users using Firefox/Safari, and the odd clients here and there who would have very specific annoying requirement like "I must be able to test on Internet Explorer - yes I know it's deprecated but my startup is B2B and I'm the small fish there"). Your docs might be short but they are already good and readable, especially for a target that might include a mix of pure devs and automation QAs/SDET/etc. All the best!