5 ms·

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found

by pfdietz 2mo ago

I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order. This found new bugs involving unintended persistent state.

natbennett 2mo ago
After a bunch of experiences like this I will now basically only use test runners that automatically run the tests in randomized order. Gotta catch that stuff early.
neuronexmachina 2mo ago
Reversing the test order is also pretty effective: https://pypi.org/project/pytest-reverse/ https://pypi.org/project/pytest-reverse/
sholladay 2mo ago
Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues. https://github.com/avajs/ava https://github.com/avajs/ava
crabbone 2mo ago
While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms). Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.
- pkhuong 2mo ago
  > Any interface with thousands of individual pieces is difficult to use. Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.
  - crabbone 2mo ago
    If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?
- pfdietz 2mo ago
  > While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug. (This was the standard compliance test suite for Common Lisp, btw.)
  - crabbone 2mo ago
    Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc). I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test. The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar). This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?
    - pfdietz 2mo ago

by pfdietz 2mo ago

natbennett 2mo ago

After a bunch of experiences like this I will now basically only use test runners that automatically run the tests in randomized order. Gotta catch that stuff early.

neuronexmachina 2mo ago

Reversing the test order is also pretty effective: https://pypi.org/project/pytest-reverse/ https://pypi.org/project/pytest-reverse/

sholladay 2mo ago

Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues. https://github.com/avajs/ava https://github.com/avajs/ava

crabbone 2mo ago

While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use. Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms). Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.

pkhuong 2mo ago
> Any interface with thousands of individual pieces is difficult to use. Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.
- crabbone 2mo ago
  If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?
pfdietz 2mo ago
> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy. The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug. (This was the standard compliance test suite for Common Lisp, btw.)
- crabbone 2mo ago
  Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc). I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test. The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar). This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?
  - pfdietz 2mo ago