2 ms·

Playwright (and axe) is a good option, but how do I have confidence it's performing the test correctly and repeatably? If the test seems flaky how do I know it'

Playwright (and axe) is a good option, but how do I have confidence it's performing the test correctly and repeatably? If the test seems flaky how do I know it's the software and not randomness of the AI part? I want tests to be predictable.

wewtyflakes 2y ago
For running the axe test itself, once the agent has decided to run it, is a dedicated tool to run an off-the-shelf axe script. The axe script itself does not change from run to run, so assuming you are running the test on the same page, you should get the same result.
- sahmeepee 2y ago
  The axe element will be the same each time, but you can do that without any AI shenanigans - just run it in a normal Playwright test. My question was really about the page interactions and the assertions being driven by AI: if they are going to be generating different code every time the test runs, how can you have any confidence in the test not having false positives and false negatives at least some of the time, unless you read the generated script each time? That sounds like a lot more work than just writing the test once in the traditional way (codegen or manually) and tweaking it only when there's a breaking change to the page. If people are genuinely using this approach then there must be something I'm missing.
  - wewtyflakes 2y ago
    We are going to be adding the ability to trigger more actions (beyond the normal clicks/keyboard) without AI by using the in-browser control panel. We wanted to add it for this ShowHN, but we ran out of time on our self-imposed deadline. :( Regarding variability of flows, you can cement a given flow by pressing the `rerun` button. That takes AI out of the driver's seat and the flow will rerun the set of actions decided on in the original flow as if it's on rails. Regarding creating a test manually, that will be a best fit for pages that have consistent selector logic for elements, though we found that as soon as a page starts randomizing element IDs, this approach starts to struggle. We get around this by creating a prioritized list of selectors for every action that touches the DOM, so that if `document.querySelector("#shenanigansId")` fails, the run can still continue by choosing the next-best selector, and so on. Thankfully this logic requires no AI at all, though it is heuristic at the end of the day.
    - sahmeepee 2y ago
      Thanks for the detailed response. The Playwright dev team would probably say that you should avoid using IDs as selectors and instead favour the use of selectors based on user-visible aspects, e.g. "a link including the text 'cat'" or "a button with the label 'register now'". That way your tests are immune to under-the-hood changes the user is oblivious to. The range of selector options (locators in their world) are a real strength of Playwright.

wewtyflakes 2y ago

For running the axe test itself, once the agent has decided to run it, is a dedicated tool to run an off-the-shelf axe script. The axe script itself does not change from run to run, so assuming you are running the test on the same page, you should get the same result.

sahmeepee 2y ago
The axe element will be the same each time, but you can do that without any AI shenanigans - just run it in a normal Playwright test. My question was really about the page interactions and the assertions being driven by AI: if they are going to be generating different code every time the test runs, how can you have any confidence in the test not having false positives and false negatives at least some of the time, unless you read the generated script each time? That sounds like a lot more work than just writing the test once in the traditional way (codegen or manually) and tweaking it only when there's a breaking change to the page. If people are genuinely using this approach then there must be something I'm missing.
- wewtyflakes 2y ago
  We are going to be adding the ability to trigger more actions (beyond the normal clicks/keyboard) without AI by using the in-browser control panel. We wanted to add it for this ShowHN, but we ran out of time on our self-imposed deadline. :( Regarding variability of flows, you can cement a given flow by pressing the `rerun` button. That takes AI out of the driver's seat and the flow will rerun the set of actions decided on in the original flow as if it's on rails. Regarding creating a test manually, that will be a best fit for pages that have consistent selector logic for elements, though we found that as soon as a page starts randomizing element IDs, this approach starts to struggle. We get around this by creating a prioritized list of selectors for every action that touches the DOM, so that if `document.querySelector("#shenanigansId")` fails, the run can still continue by choosing the next-best selector, and so on. Thankfully this logic requires no AI at all, though it is heuristic at the end of the day.
  - sahmeepee 2y ago
    Thanks for the detailed response. The Playwright dev team would probably say that you should avoid using IDs as selectors and instead favour the use of selectors based on user-visible aspects, e.g. "a link including the text 'cat'" or "a button with the label 'register now'". That way your tests are immune to under-the-hood changes the user is oblivious to. The range of selector options (locators in their world) are a real strength of Playwright.