3 ms·

Hey! Product looks great! My main question would be, for a rapidly evolving product would it generate a lot of false positives? I really enjoy the idea of busi

by Trufa 6y ago

Hey! Product looks great! My main question would be, for a rapidly evolving product would it generate a lot of false positives? I really enjoy the idea of business/marketing people collaborating with tests. Congrats. I also really like your business model so will give it a try.

tmcneal 6y ago
False positive failures are a really common issue with existing E2E testing tools, so we try to do a number of things to prevent them in Reflect tests: - We generate multiple selectors for each element you interact with. So if in the future you change a class or id, we'll fallback to other selectors when running your test. We also only choose a selector if (1) it matches a single element in the DOM, and (2) that element is interactable (e.g. not completely hidden), and (3) if it has text associated with it then we'll only choose it if the text captured at recording time matches the text at run time. This helps prevent us from selecting the wrong element when running a test. - For React apps, we use heuristics to ignore classes generated by CSS-in-JS classes like Styled Components and EmotionJS. We also have the ability to target elements based on React component name (it requires a one-time setup on your side to enable this) - For Visual Testing failures (e.g. you've screenshotted an element and now that element has a different UI) we have a simple workflow to 'Accept Changes' and mark the new UI as the new baseline for tests going forward. Certainly more to do here but this is one of the key problems we're looking to tackle.
- Trufa 6y ago
  Thank you, there are two things that are either a bug, or hard to figure out. 1) I can't delete steps 2) I can't hover. Great product guys, keep it up!
  - fitzn 6y ago
    Thanks for this feedback! 1) You can delete steps on a recorded test but not during the recording. This is to ensure that we have an exact copy of the initial recording. Thereafter, you can click on the test step and click "Delete" on the middle pane. 2) These should be captured out of the box. Can you email me your test URL so I can investigate? fitz at Thank you!

by Trufa 6y ago

tmcneal 6y ago

False positive failures are a really common issue with existing E2E testing tools, so we try to do a number of things to prevent them in Reflect tests: - We generate multiple selectors for each element you interact with. So if in the future you change a class or id, we'll fallback to other selectors when running your test. We also only choose a selector if (1) it matches a single element in the DOM, and (2) that element is interactable (e.g. not completely hidden), and (3) if it has text associated with it then we'll only choose it if the text captured at recording time matches the text at run time. This helps prevent us from selecting the wrong element when running a test. - For React apps, we use heuristics to ignore classes generated by CSS-in-JS classes like Styled Components and EmotionJS. We also have the ability to target elements based on React component name (it requires a one-time setup on your side to enable this) - For Visual Testing failures (e.g. you've screenshotted an element and now that element has a different UI) we have a simple workflow to 'Accept Changes' and mark the new UI as the new baseline for tests going forward. Certainly more to do here but this is one of the key problems we're looking to tackle.

Trufa 6y ago
Thank you, there are two things that are either a bug, or hard to figure out. 1) I can't delete steps 2) I can't hover. Great product guys, keep it up!
- fitzn 6y ago
  Thanks for this feedback! 1) You can delete steps on a recorded test but not during the recording. This is to ensure that we have an exact copy of the initial recording. Thereafter, you can click on the test step and click "Delete" on the middle pane. 2) These should be captured out of the box. Can you email me your test URL so I can investigate? fitz at Thank you!