3 ms·

This is exactly my point though, you wouldn't catch this in an e2e test either unless you try with an "11". If you're explicitly trying the "11" in an e2e test,

by Sodman 6y ago

This is exactly my point though, you wouldn't catch this in an e2e test either unless you try with an "11". If you're explicitly trying the "11" in an e2e test, why not just do it in a unit test instead? Once you hit this bug once, you can add an "assert(A(11) == 12);" and move on with confidence. If you want to test this specific scenario in an extra e2e test, you could potentially be adding another 2+ mins to every other CI/CD run that ever happens on the project. Any good unit tests should at a minimum test the min/max/expected cases, as well as any known special cases. If there are unknown special cases, you're probably not any more likely to find them in e2e tests than you are in unit tests.

asdfasgasdgasdg 6y ago
To be clear, I was trying to indicate that there would be an interaction between two modules that is not tested via mocks, but is common in production. That is the most common cause of bugs, and also the type that e2e tests tend to catch quite well.
- jolux 6y ago
  This is a great example of a case where property-based testing is a good idea.
crpatino 6y ago
The G.P. was a terrible example. In reality, Module A crashes on inputs 11, 10305, 5556430, 6867534029289, -88989, -101010101010, and -3. In Module A's Unit test, there is no rational reason to test for 2 or 6, but not for 11 (or either of the other options outlined above. So in order to have any confidence at all that the module will not crash you will have to add, assuming the input is a 32bit integer, one billion test cases. An your chance of still missing at least one of the bad cases will be 99%+ anyways. On the other hand. The regression test should look something like A(B(C(D(10)))) and A(B(C(D(20)))), where A, B, C and D are modules and 10, 20 are typical user inputs. This approach will still miss most of the bad cases, but there is a chance that it will be able to pick up either 10305 or -3, which are the failure modes a user most likely would trigger in production. And yes, once identified, it is good to add cases 10305 and -3 to future tests. Though it is unclear if they will belong to the unit test suite for A, B, C or D.

by Sodman 6y ago

asdfasgasdgasdg 6y ago

To be clear, I was trying to indicate that there would be an interaction between two modules that is not tested via mocks, but is common in production. That is the most common cause of bugs, and also the type that e2e tests tend to catch quite well.

jolux 6y ago
This is a great example of a case where property-based testing is a good idea.

crpatino 6y ago

The G.P. was a terrible example. In reality, Module A crashes on inputs 11, 10305, 5556430, 6867534029289, -88989, -101010101010, and -3. In Module A's Unit test, there is no rational reason to test for 2 or 6, but not for 11 (or either of the other options outlined above. So in order to have any confidence at all that the module will not crash you will have to add, assuming the input is a 32bit integer, one billion test cases. An your chance of still missing at least one of the bad cases will be 99%+ anyways. On the other hand. The regression test should look something like A(B(C(D(10)))) and A(B(C(D(20)))), where A, B, C and D are modules and 10, 20 are typical user inputs. This approach will still miss most of the bad cases, but there is a chance that it will be able to pick up either 10305 or -3, which are the failure modes a user most likely would trigger in production. And yes, once identified, it is good to add cases 10305 and -3 to future tests. Though it is unclear if they will belong to the unit test suite for A, B, C or D.