4 ms·

If my unit tests are testing all of my public interfaces with "Given input X, expect output Y" and they're all passing, that gives me pretty good confidence tha

• asdfasgasdgasdg 6y ago
  > If my unit tests are testing all of my public interfaces with "Given input X, expect output Y" and they're all passing, that gives me pretty good confidence that the system is in good shape. Unfortunately, in my experience, most production bugs look more like this: Module A: Returns X + 1 Implementation int A(int x) { if (x == 11) crash(); return x + 1; } Unit test: assert(A(1) == 2); assert(A(5) == 6); Production: A(11); The problem compounds as the parameter space of your system grows. Unless your unit tests exercise the power set of your parameters and results for each module, there is a likelihood of missing bugs like this.
  • Sodman 6y ago
    This is exactly my point though, you wouldn't catch this in an e2e test either unless you try with an "11". If you're explicitly trying the "11" in an e2e test, why not just do it in a unit test instead? Once you hit this bug once, you can add an "assert(A(11) == 12);" and move on with confidence. If you want to test this specific scenario in an extra e2e test, you could potentially be adding another 2+ mins to every other CI/CD run that ever happens on the project. Any good unit tests should at a minimum test the min/max/expected cases, as well as any known special cases. If there are unknown special cases, you're probably not any more likely to find them in e2e tests than you are in unit tests.
    • asdfasgasdgasdg 6y ago
      To be clear, I was trying to indicate that there would be an interaction between two modules that is not tested via mocks, but is common in production. That is the most common cause of bugs, and also the type that e2e tests tend to catch quite well.
      • jolux 6y ago
        This is a great example of a case where property-based testing is a good idea.
    • crpatino 6y ago
      The G.P. was a terrible example. In reality, Module A crashes on inputs 11, 10305, 5556430, 6867534029289, -88989, -101010101010, and -3. In Module A's Unit test, there is no rational reason to test for 2 or 6, but not for 11 (or either of the other options outlined above. So in order to have any confidence at all that the module will not crash you will have to add, assuming the input is a 32bit integer, one billion test cases. An your chance of still missing at least one of the bad cases will be 99%+ anyways. On the other hand. The regression test should look something like A(B(C(D(10)))) and A(B(C(D(20)))), where A, B, C and D are modules and 10, 20 are typical user inputs. This approach will still miss most of the bad cases, but there is a chance that it will be able to pick up either 10305 or -3, which are the failure modes a user most likely would trigger in production. And yes, once identified, it is good to add cases 10305 and -3 to future tests. Though it is unclear if they will belong to the unit test suite for A, B, C or D.