5 ms·

We run ordered Integration Tests to save time as the set-up is onerous. Is that bad?

by 6LLvveMx2koXfwn 4y ago

readthenotes1 4y ago
It's bad, no doubt. The question is What are the costs of the alternatives?
daniel-s 4y ago
I think that your ordered tests are the setup.
teeray 4y ago
Like everything in tech, it’s a tradeoff. You’re trading the potential for side-effects from one test to influence the behavior of subsequent tests without you knowing for decreased setup time. This might be a good thing or a bad thing—it depends! Engineering is balancing these tradeoffs and mitigating the risks their downsides bring.
evil-olive 4y ago
not necessarily - the key is that you want to make the ordering dependencies explicit. the real issue comes from tests that have accidental or implicit order dependencies, such that adding a test to the middle of a file might cause subsequent tests to fail because they expected something from the previous test. or, refactoring a large file into two smaller files might result in a bunch of test failures, because the tests happened to depend on system state specific to the original larger file. implicit ordering dependencies also tends to assume single-threaded test execution. a common strategy to speed tests up is to run them in parallel, and that can cause havoc if the tests aren't carefully scoped to act only on resources created by that test (such as, a test of user creation that in its teardown phase says "eh, just delete all users with a username starting with "test_")
zamalek 4y ago
Now you'll struggle to run them completely in parallel (in complete isolation, i.e. unique database server and what-have-you). This will probably matter if you want to run them in CI. It's a trade-off though, and itests are a nasty monster to deal with no matter what.
andreareina 4y ago
I don't know if they exist in this library but in Python I would use subtests: def test_ordered_integration() # setup with subtest("thing one"): assert something with subtest("thing two"): assert something_else # etc
KMag 4y ago
Can you do setup single-threaded (or get back to being single-threaded at the end of setup) and fork before starting each test case? If you need multiple service processes up and running, can you make them less stateful to get around some of your test limitations?

We run ordered Integration Tests to save time as the set-up is onerous. Is that bad?

readthenotes1 4y ago

It's bad, no doubt. The question is What are the costs of the alternatives?

daniel-s 4y ago

I think that your ordered tests are the setup.

teeray 4y ago

Like everything in tech, it’s a tradeoff. You’re trading the potential for side-effects from one test to influence the behavior of subsequent tests without you knowing for decreased setup time. This might be a good thing or a bad thing—it depends! Engineering is balancing these tradeoffs and mitigating the risks their downsides bring.

evil-olive 4y ago

not necessarily - the key is that you want to make the ordering dependencies explicit. the real issue comes from tests that have accidental or implicit order dependencies, such that adding a test to the middle of a file might cause subsequent tests to fail because they expected something from the previous test. or, refactoring a large file into two smaller files might result in a bunch of test failures, because the tests happened to depend on system state specific to the original larger file. implicit ordering dependencies also tends to assume single-threaded test execution. a common strategy to speed tests up is to run them in parallel, and that can cause havoc if the tests aren't carefully scoped to act only on resources created by that test (such as, a test of user creation that in its teardown phase says "eh, just delete all users with a username starting with "test_")

zamalek 4y ago

Now you'll struggle to run them completely in parallel (in complete isolation, i.e. unique database server and what-have-you). This will probably matter if you want to run them in CI. It's a trade-off though, and itests are a nasty monster to deal with no matter what.

andreareina 4y ago

I don't know if they exist in this library but in Python I would use subtests: def test_ordered_integration() # setup with subtest("thing one"): assert something with subtest("thing two"): assert something_else # etc

KMag 4y ago

Can you do setup single-threaded (or get back to being single-threaded at the end of setup) and fork before starting each test case? If you need multiple service processes up and running, can you make them less stateful to get around some of your test limitations?