4 ms·

> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isol

by mrkeen 2mo ago

> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same. Completely backward. This definition requires isolation, rather than granting it. In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.

aleksiy123 2mo ago
I don’t think that’s right? The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel? Nothing about that “requires” isolation. It is isolation.
- mrkeen 2mo ago
  Maybe I misread. When I see a setUp() in a test suite, 9 times out of 10 it's to handle something shared and bulky, like a database. Otherwise you'd just do the thing - assert(expected, myService.run(input)); If I saw an extra myService.reset() or myService.setUp() I would suspect it's either in anticipation of a bad state (from a previous test) or to be a good neighbour for the next test which will run. In the cases where each unit test does have it's own individual Postgres or whatever, sure.
  - aleksiy123 2mo ago
    I guess it all depends, isolation isn’t binary. you can share some things and be isolated across other dimensions. You could share a Postgres dB connection but just isolate the data logically. You can isolate or share across individual tests, across suites, across envs, across runs, across time. More isolation is generally better unless the complexity or cost is too high. But I guess I was just confused as to how a test requires isolation rather than have it.
- locknitpicker 2mo ago
  > The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
  - solarengineer 2mo ago
    In JUnit, tests run sequentially in a single thread by default [1]. Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread. Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation. Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer. [1] https://docs.junit.org/6.1.3/writing-tests/parallel-execution.html https://docs.junit.org/6.1.3/writing-tests/parallel-executio...
    - aleksiy123 2mo ago
      This

by mrkeen 2mo ago

aleksiy123 2mo ago

I don’t think that’s right? The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel? Nothing about that “requires” isolation. It is isolation.

mrkeen 2mo ago
Maybe I misread. When I see a setUp() in a test suite, 9 times out of 10 it's to handle something shared and bulky, like a database. Otherwise you'd just do the thing - assert(expected, myService.run(input)); If I saw an extra myService.reset() or myService.setUp() I would suspect it's either in anticipation of a bad state (from a previous test) or to be a good neighbour for the next test which will run. In the cases where each unit test does have it's own individual Postgres or whatever, sure.
- aleksiy123 2mo ago
  I guess it all depends, isolation isn’t binary. you can share some things and be isolated across other dimensions. You could share a Postgres dB connection but just isolate the data logically. You can isolate or share across individual tests, across suites, across envs, across runs, across time. More isolation is generally better unless the complexity or cost is too high. But I guess I was just confused as to how a test requires isolation rather than have it.
locknitpicker 2mo ago
> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation. I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
- solarengineer 2mo ago
  In JUnit, tests run sequentially in a single thread by default [1]. Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread. Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation. Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer. [1] https://docs.junit.org/6.1.3/writing-tests/parallel-execution.html https://docs.junit.org/6.1.3/writing-tests/parallel-executio...
  - aleksiy123 2mo ago
    This