4 ms·
To be frank, I think you are now overselling a product where it really doesn't need to be sold as much as it already has a lot of value in itself. E2E tests des
by creesch 4y ago
To be frank, I think you are now overselling a product where it really doesn't need to be sold as much as it already has a lot of value in itself. E2E tests deserve better tooling to debug them, not because they are flaky, but because they need to be analyzed anyway when they fail. That's basically what rubbed me slightly the wrong way initially and what mostly inspired me to comment on the flaky E2E test remark.
E2E tests will always be more "flaky" compared to tests further down the pyramid. That doesn't mean they are less reliable, but is inherent to the place where they are executed and is absolutely fine. What your tooling here promises to do is make debugging of failed tests much more convenient and streamlined, which is awesome and in itself is already a huge selling point if I am being honest.
Some points you mention:
> but it relies on the assumption that writing higher level tests inherently means you're going to have slower tests that are more unreliable
Unreliable, to some degree, as there are just more components involved, so any disruption in any of them has the potential to make your test fail.
Slower, most certainly, but that is the nature of e2e tests and not necessarily a bad thing when used within the right context.
As an extremely simplified example when faced with a login screen in your e2e tests you should just want to test two things:
- Happy flow where the user ends up being logged in.
- Unhappy flow where a feedback message is shown.
This because loading the page with the login windows does cost time. Sending the request and waiting for the page to process the response also costs time. So anything that doesn't test the UI should not be tested in the UI. There likely are a lot of different ways login in can fail, but those can easily be tested at the API level.
The same is true for functionality only available for when users are logged in. Sure, you can set up an automatic session so you don't need to go through the login screen and all that during your test. But, you still need to set all that up while the logic behind it can likely much more easily be tested further down limiting the need to do as many UI based e2e tests.
> to things users actually care about (which is the most important part of testing at all!)
Is it? It does depend on the application of course, but I'd say that it only becomes important once the underlying logic functions correctly. Certainly for applications handling and processing critical/important data. A proper foundation there is needed to be able to provide the user facing parts in a way that is reliable. If you don't focus on that underlying base you are effectively doing unit and api tests through a detour.
Again, I think you made a really cool product. Just nitpicking on some of the arguments you made here :)
- mikeshi42 4y agoSorry my response was geared more towards what I hope the future of e2e (and bleeding into integration) testing could entail, independent of what we manage to accomplish - as opposed to trying to sell our product. I do get excited thinking about where testing can improve - with the end goal that developers get to ship reliable apps for their users with less frustration of testing along the way. Our product is very much just a minuscule fraction of our way there with our current iteration. I also totally agree - e2e tests deserve better tooling to debug, and that alone we've found quite compelling of a product to be working on already, and I appreciate you find it cool as well :D With regards to picking the right level of testing - I think there's a few km long rabbit hole to go down with that one, and I totally admit I hold strong beliefs that are different from most people in testing today. I agree though - there's always going to be a different preferred tool for different problems. It'd be pretty wild to expect e2e testing to accomplish the bulk of it! That being said, I think if e2e tests were better in a few dimensions, developers might find it useful for more use cases than they do today (some teams we talk to want to do more e2e tests but find the shortfalls as a real barrier to justify the investment).