5 ms·
100% test coverage
- sghiassy 3y agoI don’t get the 100% branch coverage requirement. If the code is off of ‘master’ why would you have to go and fix it?
- hinkley 3y agoConditional branches, not repository branches.
- kristiandupont 3y agoI think he's referring to branching in the code (if-then-else blocks).
- anotherhue 3y agoI was confused also but decided they meant if/else branching
- hooksfordays 3y agoBranch coverage means something different. If you have a function with 2 if statements, there are 4 branches. 100% code coverage could be 1 test and would run the code before + after the if statements, and enter each if statement block. 100% branch coverage would test this code 4 times: once without entering either if statement, once only entering the first, once only entering the second, and once entering both.
- bewaretheirs 3y agoBranch coverage refers to branch instructions (in the underlying machine code), not a branch in whatever source control system you're using.
- hinkley 3y ago> We run one of our projects not only with 100% test coverage, but 100% branch coverage. It’s deeply unpleasant.
- anotherhue 3y agoI've seen bad codebases rightfully elevated to greatness because of their test suite. The confidence it adds allows new people to make changes much faster.
- DragonStrength 3y agoI've seen great teams grind to a halt because of edicts. The elevation of process over results is demoralizing to builders.
- jameshart 3y agoBut builder morale is not a product that generates revenue
- DragonStrength 3y agoNeither does process. Product velocity to zero for the sake of a management metric is a good way to kill revenue while getting some managers promoted.
- anotherhue 3y agoSure, but I've seen enough premadonnas walk away from a tyre fire that fed their ego to build that I don't take their egos too seriously. Stifling process? No, unwavering diligence? Yes.
- DragonStrength 3y agoSure, but I've seen enough prima donna middle managers who fed their ego by enforcing process to know the pragmatism of builders is less harmful. I'll take a pragmatist who understands when 94% or 60% is good enough test coverage over the manager who demands 100%. I'm positive I know who has the bigger ego there. Perhaps you're reading your own experience into this, but I pretty clearly set up a scenario, and yes, I do think there's a lot of ego in anyone demanding 100% coverage, especially if they aren't writing the code.
- ramraj07 3y agoHow do we test for 100% branch coverage on python that covers list comprehensions?
- DragonStrength 3y ago> But 100% coverage definitely helps, especially in languages like Ruby where an untaken branch can contain a serious error that’s accepted by the interpreter, but will take down production if hit. And if you're in a language providing more constraints, the benefits can quickly turn into a cargo cult. This is certainly the mindset of Ruby folks, and such a mindset left a native client team with 0 system devs and all web dev converts who refused to engage with our actual issues and focused only on dogma within 2 years of hiring a manager with that background. Color me skeptical of this applying to all projects.
- kristiandupont 3y agoI don't think I've ever worked with a code base that had 100% coverage so I can't speak from experience per se. But my general feeling is that the cost increases exponentially and the return on investment diminishes exponentially with those last bits of coverage. Also, I don't think I need to tell anyone here but "100% branch coverage" still doesn't mean "100% branch combination coverage", which would quickly make your test code base grow to the size of the universe anyway.
- morelisp 3y agoThe term you're looking for is "path coverage".
- kristiandupont 3y agoSo it is, thank you!
- Veuxdo 3y agoYou probably can get to 100% branch test coverage while being able to tie all tests to business requirements. It'll be expensive, though.
- AndrewKemendo 3y agoHow do you manage breaking changes from changes in third party APIs and libraries then? Woof
- iron2disulfide 3y agoI started my career off in the chip design world, where 100% line, branch, FSM transition, functional group, and toggle (i.e. individual bits in a bus switching 0 -> 1 _and_ 1 -> 0) was "table stakes". There are a lot of comments in here saying that achieving 100% coverage would be expensive - and they're right. The majority of headcount in modern chip design houses is taken up by verification engineers, whose sole job is to architect, implement and maintain a minimal test corpus that achieves that high bar. The cost of failure is simply too high to omit this step. It was unsettling to me after moving to a SWE job where coverage was kind of... not as important.
- rTX5CMRXIfFG 3y agoI can think of certain types of software that are so critical that they must be 100% thoroughly verified to be logically correct, e.g. software in surgical machines, airplanes, or high-frequency trading. The vast majority of applications, though, would survive with less coverage, though I still think that the sweet spot is somewhere between 70-80%.
- pictur 3y ago%100 test coverage is absolutely essential for dependencies that are used in too many places and can affect too many places if something goes wrong. but it is necessary to decide whether it is really necessary to approach all addictions in this way. what I'm talking about is of course valid for a large codebase. And yes, 100% test coverage does not eliminate the possibility of error :D
- jameshart 3y agoHow do you get 100% branch coverage in a dynamic language? Every method dispatch or property access is a branch to an arbitrary piece of code, or a source of an arbitrary error to handle, depending on what object it happens to end up dealing with.
- pawelduda 3y agoI wouldn't worry too much about getting to 100%... With tests the spectrum can be ALWAYS extended to the right side. E.g. you could argue that you should also do mutation testing because 100% coverage doesn't mean tests are sensitive enough to changes in code branches, etc... Once you realize this you never really arrive at 100% That said having coverage target is more often positive than not having one. Coverage target kind of sends "we care about test suite" message
- morelisp 3y agoI mean, this is the kind of thing I agree with, and would even say I'd trust a project with 90% coverage but no surviving test mutants vs. 100% and lots of mutants. But in reality I'm usually looking at piles of like... 60-70%ish? Thousands of mutants. Maybe a round-trip fuzz or property test if I'm lucky. So I keep aiming for 100% and hoping anyone else follows...
- pawelduda 3y agoThese are also more expensive kinds of tests to setup and run so it's often not feasible, maybe in extreme cases. In a few Rails codebases I tried to mutation test something critical only to see that getting it to work would take more time than worth the effort
- lucumo 3y agoThe tradeoff between the cost of many tests and the benefits of them is why I usually use a lower coverage requirement. Something like 80% or sometimes 60% seems to give most of the benefits with less of the cost.
- TheChaplain 3y agowell, in my experience 100% line and branch coverage is not enough. Countless times I see people write tests "to pass", but nothing to make them fail. I speak about null-pointers, invalid dates, strings with garbage, out of bounds indices etc.
- nathants 3y agoquickcheck is eating popcorn with 3d glasses on.