3 ms·
I'll prefix by saying that this is exactly how I write my tests, too. But, let's do a little critical thinking here and ask "Why do we write integration tests?
by jwalton 6y ago
I'll prefix by saying that this is exactly how I write my tests, too. But, let's do a little critical thinking here and ask "Why do we write integration tests?" If the goal is to improve software quality, I'm afraid I have some disappointing results for you.
Back when my dad was working at a huge software company (BigCorp, let's say), he looked at how much effort the manual test team spent over a two week period, and how many defects they found. Then he did the same over the next two week period. Now, logically, in that second period, some of the earlier defects had been found and fixed, so it should now be harder to find defects, so the total defect count per unit of test effort should be lower, right? Armed with two data points, he did a regression and worked out how many defects they'd find if they did an infinite amount of testing; effectively the undiscovered defect count left in the product. The number he got was astoundingly huge - no one believed this was possible. So, he went over to the plotter and plotted out a giant effort/defect count curve, and then every two weeks he'd put a pin in his plot to show reality vs. his prediction, and for months and months until he got tired of doing it, he was pretty dead on. And he didn't just do this for one project, he did it for lots of projects, across lots of different teams of various sizes.
On all of these projects, all the manual testing they could possibly hope to accomplish if they had their entire testing staff spend 100 years testing would have reduced the overall defect count by a tiny tiny fraction of a percent. So testing and then fixing bugs found in tests (at least in all the projects at BigCorp) didn't really have a huge impact on software quality.
And this should not really be a surprise; if you were manufacturing cars, you might test the power seats on every 100th car, and use this to figure out what the quality of power seats in your cars is. You might discovery only 95% of power seats are working, and you might think that's unacceptable. If you do, though, you're not going to "solve" the problem by fixing all the broken power seats you test; you're going to go figure out where in your manufacturing process/supply chain things are going wrong and fix the problem there, and improve your quality. The testing is a measure of your process.
Software is not so different - by the time it gets to integration testing, the code has been written. The level of quality of the code has largely already been set at this point - all the defects that are going to be introduced have already been introduced. The quality level is dependent upon your process and the skill level of your developers. So testing some arbitrary fraction of the lines-of-code is going to find problems in some percentage of those lines-of-code, but fixing those particular problems? Is this going to have a huge impact on quality?
- krab 6y ago> you're not going to "solve" the problem by fixing all the broken power seats you test; you're going to go figure out where in your manufacturing process/supply chain things are going wrong I think the point is that examining the failing seats will lead you to the points in process that should be fixed. Therefore you can fix the issue more efficiently than going through the whole process. Similar as with knowing details about failing software tests. Imagine I hide the information about seats and tell you 10% of the finished cars have "some" defect. Where would you even start looking in the factory?