5 ms·

This seems a bit overdone right? If I understand, Fullstory and the like are cool because they let you reproduce unknown user scenarios and bugs. But for tests,

This seems a bit overdone right? If I understand, Fullstory and the like are cool because they let you reproduce unknown user scenarios and bugs. But for tests, the inputs are all known (that's the point of a test) so it should be trivial to reproduce. In other words, how is this advantageous over just rerunning the failing test locally and opening your browser developer tools?

mikeshi42 4y ago
That's a great question! While in theory, yes that is absolutely true tests should be reproducible anywhere, unfortunately in practice, tests can unfortunately fail in hard to reproduce ways (very much like production issues!), especially when testing a complex app or working in a large test suite written by many different people. Common examples of this entails race conditions, your test server might be super snappy on your local machine, but run much slower in CI, and expose race conditions that were unaccounted for. Other times it may be due to subtle difference between environments (service doesn't behave properly in CI due to build step differences or service is initialized differently in CI than locally). The list of scenarios where local reproduction can be hard is unfortunately quite long, which is the downside of running end to end tests, the surface area they test (and can fail at) is really wide and complex, much like production failures. Enough so that teams find it useful to have proper debugging tools to debug what when wrong when tests fail in CI, like how you'd want proper debugging tools when apps fail in production :) Of course there's the other side of rerunning failing tests locally, even if they are reproducible. E2E tests and their associated environment can take time to set up locally. Why bother spending 20 minutes checking out the failed commit, searching up internally how to spin up the local e2e environment to reproduce the failure, and then running the test(s) to debug. Instead you can just click through a UI and get everything you need, it's much smoother!
Aeolun 4y ago
> it should be trivial to reproduce Yeah, no. It should be trivial to reproduce. Practically there’s hundreds of actions and saga’s firing at the same time, and the order of events is anything but guaranteed. That’s absolutely a problem we wouldn’t have if everything was well-architected, but yeah, meet the real world.

mikeshi42 4y ago

That's a great question! While in theory, yes that is absolutely true tests should be reproducible anywhere, unfortunately in practice, tests can unfortunately fail in hard to reproduce ways (very much like production issues!), especially when testing a complex app or working in a large test suite written by many different people. Common examples of this entails race conditions, your test server might be super snappy on your local machine, but run much slower in CI, and expose race conditions that were unaccounted for. Other times it may be due to subtle difference between environments (service doesn't behave properly in CI due to build step differences or service is initialized differently in CI than locally). The list of scenarios where local reproduction can be hard is unfortunately quite long, which is the downside of running end to end tests, the surface area they test (and can fail at) is really wide and complex, much like production failures. Enough so that teams find it useful to have proper debugging tools to debug what when wrong when tests fail in CI, like how you'd want proper debugging tools when apps fail in production :) Of course there's the other side of rerunning failing tests locally, even if they are reproducible. E2E tests and their associated environment can take time to set up locally. Why bother spending 20 minutes checking out the failed commit, searching up internally how to spin up the local e2e environment to reproduce the failure, and then running the test(s) to debug. Instead you can just click through a UI and get everything you need, it's much smoother!

Aeolun 4y ago

> it should be trivial to reproduce Yeah, no. It should be trivial to reproduce. Practically there’s hundreds of actions and saga’s firing at the same time, and the order of events is anything but guaranteed. That’s absolutely a problem we wouldn’t have if everything was well-architected, but yeah, meet the real world.