3 ms·

I don't understand why he jumps straight from 'one test per process' to 'one test per thread' as the alternative. I'm not actually clear what he means by 'test

by marky1991 2y ago

I don't understand why he jumps straight from 'one test per process' to 'one test per thread' as the alternative. I'm not actually clear what he means by 'test' to be honest, but I assume he means 'a single test function that can either pass or fail' Eg in python (nose) class TestSomething: def test_A(): ... def test_B(): ... I'm assuming he means test_A. But why not run all of TestSomething in a process? Honestly, I think the idea of having tests have shared state is bad to begin with (for things that truly matter, eg if the outcome of your test depends on the state of sys.modules, something else is horribly wrong), so I would never make this tradeoff to benefit a scenario that I never think should be done. Even if we were being absolute purists, this still hasn't solved the problem, the second your process communicates with any other process (or server). And that problem seems largely unsolveable, short of mocking. Basically, I'm not convinced this is a good tradeoff, because the idea of creating thousands and thousands of processes to run a test suite, even on linux, sounds like a bad idea. (And at work, would definitely be a bad idea, for performance reasons)

cbarrick 2y ago
> I'm not actually clear what he means by 'test' to be honest, but I assume he means 'a single test function that can either pass or fail' I assume so as well. Unit testing in Rust is based around functions annotated with #[test], so it's safe to assume that when the author says "test" they are referring to one such function. It's up to the user to decide what they do in each function. For example, you could do a Go-style table-driven test, but the entire function would be a single "test", _not_ one "test" per table entry.
saghm 2y ago
I think most of the context that might explain your confusion is the way that tests work out of the box in Rust. The default test harness when invoking `cargo test` runs one test per thread (and by default parallelizes based on the number of cores available, although this is configurable with a flag). In Rust, there isn't any equivalent to the `TestSomething` class you give; each test is always a top-level function. Since `cargo nextest` is a mostly drop-in replacement for `cargo test`, I imagine the author is using one test per thread as an alternative because that's the paradigm that users will be switching from if they start using cargo nextest. While enforcing no shared state in tests might be useful, that wouldn't be feasible in Rust without adding quite a lot of constraints that would be tough if not impossible to enforce in a drop-in replacement for cargo test. There's certainly room for alternatives in the testing ecosystem in Rust that don't try to maintain compatibility with the built-in test harness, but I don't think the intention of cargo nextest is to try to do that. One other point that might not be obvious is that right now, there's no stable way to hook into Rust's libtest. The only options to provide an alternative testing harness in Rust are to either only support nightly rather than stable, break compatibility with tests written for the built-in test harness, or provide a separate harness that still supports existing tests. I'm sure there are arguments to be made for each of the other alternatives, but personally, I don't think there's any super realistic chance for adoption of anything that picks the first two options, so the approach cargo nextest is taking the most viable approach available (at least until libtest stabilizes, but it's not obvious when that will happen).
jclulow 2y ago
NB: the author's pronouns are they/she, not he.
sunshowers 2y ago
FYI I use they/she pronouns (thanks jclulow!) > Honestly, I think the idea of having tests have shared state is bad to begin with (for things that truly matter, eg if the outcome of your test depends on the state of sys.modules, something else is horribly wrong), so I would never make this tradeoff to benefit a scenario that I never think should be done. I don't disagree as a matter of principle, but the reality really is different. Some of the first nextest users outside of myself and my workplace were graphical libraries and engines. > Basically, I'm not convinced this is a good tradeoff, because the idea of creating thousands and thousands of processes to run a test suite, even on linux, sounds like a bad idea. (And at work, would definitely be a bad idea, for performance reasons) With Rust or other native languages it really is quite fast. With Python I agree not as much, so this tradeoff wouldn't make as much sense there yes. But note that things like test cancellation are a little easier to do in an interpreted model.
- vlovich123 2y ago
  Have you considered having pre warmed zygote processes hot that you then fork for each test rather than launching the process from scratch? That might mitigate the performance issue even more since less has to be initialized.
  - sunshowers 2y ago
    Zygote processes for the test binary itself, you mean? That will probably require some coordination from the test because fork is not compatible with most Rust code. Nextest is designed to work with arbitrary Rust test binaries, though it imposes some conditions on them.
hinkley 2y ago
> Honestly, I think the idea of having tests have shared state is bad to begin with I blame this partially on our notions of code reuse. We conflate it with several other things, and in the case of tests we conflate it with state reuse. And the availability of state reuse leads to people writing fakes when they should be using mocks, and people not being able to tell the difference between mocks and fakes and thus being incapable to have a rational discussion about them. To my thinking, and the thinking of pretty much all of the test experts I’ve called Mentor (or even Brother), beforeEaches should be repeatable. Add a test, it repeats one more time. Delete a test, one less. And if they’re repeatable, they don’t have to share the same heap. One heap is as good as another. Lots of languages can only do that segregation at the process level. In NodeJS it would be isolates (workers). If you’re very careful about global state you could do it per thread. But that doesn’t happen very often because “you” includes library writers, language designers, and your coworker Steve who is addicted to in-memory caching. I can say, “don’t be Steve” until I’m blue in the face but nearly every team hires at least one Steve, and some are rotten with them.

cbarrick 2y ago

> I'm not actually clear what he means by 'test' to be honest, but I assume he means 'a single test function that can either pass or fail' I assume so as well. Unit testing in Rust is based around functions annotated with #[test], so it's safe to assume that when the author says "test" they are referring to one such function. It's up to the user to decide what they do in each function. For example, you could do a Go-style table-driven test, but the entire function would be a single "test", _not_ one "test" per table entry.

saghm 2y ago

I think most of the context that might explain your confusion is the way that tests work out of the box in Rust. The default test harness when invoking `cargo test` runs one test per thread (and by default parallelizes based on the number of cores available, although this is configurable with a flag). In Rust, there isn't any equivalent to the `TestSomething` class you give; each test is always a top-level function. Since `cargo nextest` is a mostly drop-in replacement for `cargo test`, I imagine the author is using one test per thread as an alternative because that's the paradigm that users will be switching from if they start using cargo nextest. While enforcing no shared state in tests might be useful, that wouldn't be feasible in Rust without adding quite a lot of constraints that would be tough if not impossible to enforce in a drop-in replacement for cargo test. There's certainly room for alternatives in the testing ecosystem in Rust that don't try to maintain compatibility with the built-in test harness, but I don't think the intention of cargo nextest is to try to do that. One other point that might not be obvious is that right now, there's no stable way to hook into Rust's libtest. The only options to provide an alternative testing harness in Rust are to either only support nightly rather than stable, break compatibility with tests written for the built-in test harness, or provide a separate harness that still supports existing tests. I'm sure there are arguments to be made for each of the other alternatives, but personally, I don't think there's any super realistic chance for adoption of anything that picks the first two options, so the approach cargo nextest is taking the most viable approach available (at least until libtest stabilizes, but it's not obvious when that will happen).

jclulow 2y ago

NB: the author's pronouns are they/she, not he.

sunshowers 2y ago

FYI I use they/she pronouns (thanks jclulow!) > Honestly, I think the idea of having tests have shared state is bad to begin with (for things that truly matter, eg if the outcome of your test depends on the state of sys.modules, something else is horribly wrong), so I would never make this tradeoff to benefit a scenario that I never think should be done. I don't disagree as a matter of principle, but the reality really is different. Some of the first nextest users outside of myself and my workplace were graphical libraries and engines. > Basically, I'm not convinced this is a good tradeoff, because the idea of creating thousands and thousands of processes to run a test suite, even on linux, sounds like a bad idea. (And at work, would definitely be a bad idea, for performance reasons) With Rust or other native languages it really is quite fast. With Python I agree not as much, so this tradeoff wouldn't make as much sense there yes. But note that things like test cancellation are a little easier to do in an interpreted model.

vlovich123 2y ago
Have you considered having pre warmed zygote processes hot that you then fork for each test rather than launching the process from scratch? That might mitigate the performance issue even more since less has to be initialized.
- sunshowers 2y ago
  Zygote processes for the test binary itself, you mean? That will probably require some coordination from the test because fork is not compatible with most Rust code. Nextest is designed to work with arbitrary Rust test binaries, though it imposes some conditions on them.

hinkley 2y ago

> Honestly, I think the idea of having tests have shared state is bad to begin with I blame this partially on our notions of code reuse. We conflate it with several other things, and in the case of tests we conflate it with state reuse. And the availability of state reuse leads to people writing fakes when they should be using mocks, and people not being able to tell the difference between mocks and fakes and thus being incapable to have a rational discussion about them. To my thinking, and the thinking of pretty much all of the test experts I’ve called Mentor (or even Brother), beforeEaches should be repeatable. Add a test, it repeats one more time. Delete a test, one less. And if they’re repeatable, they don’t have to share the same heap. One heap is as good as another. Lots of languages can only do that segregation at the process level. In NodeJS it would be isolates (workers). If you’re very careful about global state you could do it per thread. But that doesn’t happen very often because “you” includes library writers, language designers, and your coworker Steve who is addicted to in-memory caching. I can say, “don’t be Steve” until I’m blue in the face but nearly every team hires at least one Steve, and some are rotten with them.