4 ms·

You can find a lot of bugs with e2e but they're frequently not actionable because you can't figure out which part of your stack is causing them. Being non-actio

You can find a lot of bugs with e2e but they're frequently not actionable because you can't figure out which part of your stack is causing them. Being non-actionable is one of the least-understood reasons to skip testing, and ironically the most important one for keeping focus. They also require very high maintenance, if they're going to be sensitive enough to be useful, as any stack change potentially means re-validating them. Read up on test pyramid for more. E2E are the least useful during dev stage, more useful for exercising canary P0 use cases that can never be compromised by any stack bug or it's a stop-ship until you find it. They aren't diagnostic or robust enough for dev, but for a canary role their fragility is a plus.

pydry 6y ago
>you can't figure out which part of your stack is causing them If your debugging tools are halfway decent you certainly can.
- geoelectric 6y ago
  On a multi-collaborator large system with half the behavior off in AWS? You're either dreaming, or haven't worked on anything complex enough to see it yet. E2E means system test, whether or not you think it is, including back end. In almost any real software in 2020 that means multiple debuggers, monitors, logs, god knows what. But if you're testing a monolith, sure. Your system is one unit, so unit test your monolith and enjoy your debugger. If you're testing a monolith outside of something like embedded or basic app testing, quit, because the company you're working for built a monolith. Inactionable doesn't mean "not possible", it means "more trouble than it's worth." That's why it's advisable to do E2E on P0 cases and not really advisable to do it for P3 cases. You can do tooling work to move the bar for how much trouble you have to put in, but "try harder" misses the point.
  - pydry 6y ago
    In such a system there's typically two kinds of E2E. E2E for the team's service and end to end for the entire system. If you can't write the former without acceptable debugging tools your tests suck and you should rewrite them. If you are a team writing a micro service in an ecosystem of micro services, you need to write E2E tests that mock systems external to yours and have very clear agreements/contracts about how they interact as well as a QA team to test the system as a whole coz otherwise each bug that crosses a layer degenerates into an orgy of finger pointing. Ideally debugging at that scale should be rare and done with multiple members from multiple teams.
    - geoelectric 6y ago
      In almost every org I’ve been in, E2E meant full thickness UI to datastore and back tests. That’s where I was coming from. I’ve never considered the single service tests E2E, more either a component test suite or a public API unit test suite (same thing). I agree you’d want those with collaborators mocked, and it’s where I’d lean the bulk of my test strategy for a system like that from a QA POV (as well as promoting internal API unit testing from a dev POV).

pydry 6y ago

>you can't figure out which part of your stack is causing them If your debugging tools are halfway decent you certainly can.

geoelectric 6y ago
On a multi-collaborator large system with half the behavior off in AWS? You're either dreaming, or haven't worked on anything complex enough to see it yet. E2E means system test, whether or not you think it is, including back end. In almost any real software in 2020 that means multiple debuggers, monitors, logs, god knows what. But if you're testing a monolith, sure. Your system is one unit, so unit test your monolith and enjoy your debugger. If you're testing a monolith outside of something like embedded or basic app testing, quit, because the company you're working for built a monolith. Inactionable doesn't mean "not possible", it means "more trouble than it's worth." That's why it's advisable to do E2E on P0 cases and not really advisable to do it for P3 cases. You can do tooling work to move the bar for how much trouble you have to put in, but "try harder" misses the point.
- pydry 6y ago
  In such a system there's typically two kinds of E2E. E2E for the team's service and end to end for the entire system. If you can't write the former without acceptable debugging tools your tests suck and you should rewrite them. If you are a team writing a micro service in an ecosystem of micro services, you need to write E2E tests that mock systems external to yours and have very clear agreements/contracts about how they interact as well as a QA team to test the system as a whole coz otherwise each bug that crosses a layer degenerates into an orgy of finger pointing. Ideally debugging at that scale should be rare and done with multiple members from multiple teams.
  - geoelectric 6y ago
    In almost every org I’ve been in, E2E meant full thickness UI to datastore and back tests. That’s where I was coming from. I’ve never considered the single service tests E2E, more either a component test suite or a public API unit test suite (same thing). I agree you’d want those with collaborators mocked, and it’s where I’d lean the bulk of my test strategy for a system like that from a QA POV (as well as promoting internal API unit testing from a dev POV).