4 ms·

While what you have made looks cool it seems like you are leaving out a lot of what is at the basis of many hard to maintain flaky e2e tests: a fundamental wron

While what you have made looks cool it seems like you are leaving out a lot of what is at the basis of many hard to maintain flaky e2e tests: a fundamental wrong approach to testing resulting in too many tests in the development process where they simply do not belong. The [test automation pyramid](https://www.google.com/search?q=test+automation+pyramid&sxsrf=ALiCzsZWChhGCToG0fDx6OqsK3MJbLJKug:1659526078421&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiJ8pizyKr5AhXbnP0HHWc-AXwQ_AUoAXoECAEQAw&cshid=1659526095972504&biw=1920&bih=957 https://www.google.com/search?q=test+automation+pyramid&sxsr...) or even the [testing trophy](https://twitter.com/kentcdodds/status/960723172591992832 https://twitter.com/kentcdodds/status/960723172591992832) doe exist with very good reason. The earlier you can write your tests the easier it is to break them up in the relevant individual functionality without tests being flaky. Now you can argue the merits of Unit tests, hence the testing trophy also being a thing. But the fact of the matter is that 90% of the time e2e tests are flaky because teams try to test things in them that shouldn't be tested at that level. Having said that, the ability to inspect the DOM, network requests, etc at various places in history is a very neat ability.

mikeshi42 4y ago
Thank you for the feedback! Totally agree that the testing pyramid/trophy and the reasoning behind them are 100% the correct way to go with testing today - but it relies on the assumption that writing higher level tests inherently means you're going to have slower tests that are more unreliable (and we all hate slow and unreliable things!). The reason we're tackling this space is because we don't think this needs to be true of end to end, or even heavier integration tests. They have really awesome qualities such as being able to test close to things users actually care about (which is the most important part of testing at all!), can identify broken pieces across or in between parts of your stack, and are fairly intuitive to write/understand (imo). But what if we could then build tooling to minimize some of the bad stuff - make tests more stable by providing better debugging, enforcing best practices automatically (imagine we can tell your tests are behaving non deterministically because of X async action, or use X conditional wait instead of a time-based one), and even giving you knobs to trade off between test fidelity and speed (ex. click a button and you can turn on/off network mocking in specific tests so you only exercise a part of your system instead of spinning up/testing the world). Hopefully if we can nail our vision, you might be saying in the future "why are there so many unit tests? They cover so little surface, don't test how our app fits together at all, and require a lot of maintenance since they tend to test some form of implementation detail or another. We should be doing a lot more integration and e2e tests :)". It'll be a long road until then, but I think developers and users will be much happier the better we can make testing.
- creesch 4y ago
  To be frank, I think you are now overselling a product where it really doesn't need to be sold as much as it already has a lot of value in itself. E2E tests deserve better tooling to debug them, not because they are flaky, but because they need to be analyzed anyway when they fail. That's basically what rubbed me slightly the wrong way initially and what mostly inspired me to comment on the flaky E2E test remark. E2E tests will always be more "flaky" compared to tests further down the pyramid. That doesn't mean they are less reliable, but is inherent to the place where they are executed and is absolutely fine. What your tooling here promises to do is make debugging of failed tests much more convenient and streamlined, which is awesome and in itself is already a huge selling point if I am being honest. Some points you mention: > but it relies on the assumption that writing higher level tests inherently means you're going to have slower tests that are more unreliable Unreliable, to some degree, as there are just more components involved, so any disruption in any of them has the potential to make your test fail. Slower, most certainly, but that is the nature of e2e tests and not necessarily a bad thing when used within the right context. As an extremely simplified example when faced with a login screen in your e2e tests you should just want to test two things: - Happy flow where the user ends up being logged in. - Unhappy flow where a feedback message is shown. This because loading the page with the login windows does cost time. Sending the request and waiting for the page to process the response also costs time. So anything that doesn't test the UI should not be tested in the UI. There likely are a lot of different ways login in can fail, but those can easily be tested at the API level. The same is true for functionality only available for when users are logged in. Sure, you can set up an automatic session so you don't need to go through the login screen and all that during your test. But, you still need to set all that up while the logic behind it can likely much more easily be tested further down limiting the need to do as many UI based e2e tests. > to things users actually care about (which is the most important part of testing at all!) Is it? It does depend on the application of course, but I'd say that it only becomes important once the underlying logic functions correctly. Certainly for applications handling and processing critical/important data. A proper foundation there is needed to be able to provide the user facing parts in a way that is reliable. If you don't focus on that underlying base you are effectively doing unit and api tests through a detour. Again, I think you made a really cool product. Just nitpicking on some of the arguments you made here :)

mikeshi42 4y ago

Thank you for the feedback! Totally agree that the testing pyramid/trophy and the reasoning behind them are 100% the correct way to go with testing today - but it relies on the assumption that writing higher level tests inherently means you're going to have slower tests that are more unreliable (and we all hate slow and unreliable things!). The reason we're tackling this space is because we don't think this needs to be true of end to end, or even heavier integration tests. They have really awesome qualities such as being able to test close to things users actually care about (which is the most important part of testing at all!), can identify broken pieces across or in between parts of your stack, and are fairly intuitive to write/understand (imo). But what if we could then build tooling to minimize some of the bad stuff - make tests more stable by providing better debugging, enforcing best practices automatically (imagine we can tell your tests are behaving non deterministically because of X async action, or use X conditional wait instead of a time-based one), and even giving you knobs to trade off between test fidelity and speed (ex. click a button and you can turn on/off network mocking in specific tests so you only exercise a part of your system instead of spinning up/testing the world). Hopefully if we can nail our vision, you might be saying in the future "why are there so many unit tests? They cover so little surface, don't test how our app fits together at all, and require a lot of maintenance since they tend to test some form of implementation detail or another. We should be doing a lot more integration and e2e tests :)". It'll be a long road until then, but I think developers and users will be much happier the better we can make testing.

creesch 4y ago
To be frank, I think you are now overselling a product where it really doesn't need to be sold as much as it already has a lot of value in itself. E2E tests deserve better tooling to debug them, not because they are flaky, but because they need to be analyzed anyway when they fail. That's basically what rubbed me slightly the wrong way initially and what mostly inspired me to comment on the flaky E2E test remark. E2E tests will always be more "flaky" compared to tests further down the pyramid. That doesn't mean they are less reliable, but is inherent to the place where they are executed and is absolutely fine. What your tooling here promises to do is make debugging of failed tests much more convenient and streamlined, which is awesome and in itself is already a huge selling point if I am being honest. Some points you mention: > but it relies on the assumption that writing higher level tests inherently means you're going to have slower tests that are more unreliable Unreliable, to some degree, as there are just more components involved, so any disruption in any of them has the potential to make your test fail. Slower, most certainly, but that is the nature of e2e tests and not necessarily a bad thing when used within the right context. As an extremely simplified example when faced with a login screen in your e2e tests you should just want to test two things: - Happy flow where the user ends up being logged in. - Unhappy flow where a feedback message is shown. This because loading the page with the login windows does cost time. Sending the request and waiting for the page to process the response also costs time. So anything that doesn't test the UI should not be tested in the UI. There likely are a lot of different ways login in can fail, but those can easily be tested at the API level. The same is true for functionality only available for when users are logged in. Sure, you can set up an automatic session so you don't need to go through the login screen and all that during your test. But, you still need to set all that up while the logic behind it can likely much more easily be tested further down limiting the need to do as many UI based e2e tests. > to things users actually care about (which is the most important part of testing at all!) Is it? It does depend on the application of course, but I'd say that it only becomes important once the underlying logic functions correctly. Certainly for applications handling and processing critical/important data. A proper foundation there is needed to be able to provide the user facing parts in a way that is reliable. If you don't focus on that underlying base you are effectively doing unit and api tests through a detour. Again, I think you made a really cool product. Just nitpicking on some of the arguments you made here :)