4 ms·
The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said
by mikestew 10y ago
The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definition of "compiled") language? Meh, you don't need to check every use of a variable at runtime to make sure the data types didn't go haywire.
The real Real Tragedy of 100% coverage is the number of shops who think they're done testing when they hit 100%. I've heard words to that effect out of the mouth of a test manager at Microsoft, as one example. No, code coverage is a metric, not the metric. Code coverage doesn't catch the bugs caused by the code you didn't write but should have, for example. Merely executing code is a simplistic test at best.
- ruleabidinguser 10y ago70% coverage... I don't know if I'm going to be able to give up my 0% coverage 1-man projects...
- BigJono 10y agoI'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out. Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them. Anything past that is for companies with customers that need to maintain a certain level of quality and service. Worry about it when you get there.
- dvirsky 10y agoAgreed. If you're a small start-up in a hurry and <4 devs, the best strategy would be to write top-down smoke tests on your API that will actually tell you when things break, even if not exactly where. Their actual derived coverage might be pretty high.
- plaguuuuuu 10y agoAs long as that 10-20% is the more complex bits, and you are capable of picking out which bits are complex, it's IMO better to do that and spend the time saved on other shit that increases maintainability and reliability. Killing tech debt, killing your teammates, whatever.
- tumblen 10y agoHow do you know what 10-20% to test? This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.
- jakewins 10y agoSince what to test is a sliding scale, determining when and what to test is perhaps something that comes with exposure to good testers.. For me, a few references I use are "tests are a thinking help for specifying behavior", "tests hold behavior in place" and "test until you feel comfortable". The first is a note that TDD thoughts are written by developers that write APIs, not by developers that write applications. TDD is a fantastic tool for an API designer, because they force you to think about the experience of using the API. So, whenever I design APIs, I like TDD. This is also a good argument for why you should minimize setup code that "goes behind the scenes" - if you're testing, say, a REST API, do as much of the setup and assertions as you can via the REST API as well! The second helps me remember to think "a year from now, are all the behaviors this code needs to have obvious, or is someone likely to unintentionally break it?". I try to write tests that will flag if someone broke an important edge case - or the main use case! Tests can be used as the programmers equivalent of a carpenters' clamps, kind of. The third is why I don't write as many tests anymore. I normally try to write one workflow-oriented feature test up front, like "When a user creates a new invoice, then that invoice should show up in the users list of invoices with the values they entered, plus x,y,z auto-generated parameters". As I implement the feature, if I come across a piece of logic that makes me feel uncomfortable - lots of branching, or code that's very important that it stays intact - I'll write a unit test or two to hold that in place; sometimes I don't write any unit tests, meaning I'd have written just one or two tests over the course of two or three days of implementation.
- jdmichal 10y agoYou should test the thing that makes you money first, and should delay testing supporting functionality. For instance, if I am writing a document converter, the thing that makes me money is the AST -> AST conversion. Testing that should come before testing parsing (bytes -> AST) and rendering (AST -> bytes). The place where you make money is the place that will have the largest demand for new and changing functionality. And where things change the most is where you need tests to protect against regressions.
- threatofrain 10y agoBeing able to scale-up is interesting, both human logistics-wise and machine performance-wise, but the ability to scale down is also interesting.
- nathan_f77 10y agoI've been working on a 1-man project with maybe 5% test coverage, just for some critical libraries that I ended up refactoring a few times. There's actually one little library that has 100% test coverage and super detailed error messages, because that's where many of the bugs seemed to happen. I also have a simple integration test that just clicks through everything and makes sure nothing crashes. Not having a lot of tests can be painful. Especially when you're learning new languages or frameworks, you almost always want to go back and rewrite some code (or just reorganize it into different files), and it's really nice to have tests when you do that. So sometimes that gives me the motivation to start writing up a bunch of tests, and then after that I dive in and refactor everything.
- sqeaky 10y agoDoesn't it also depend on th application? I mean I wouldn't settle for less than 200% test coverage on an automatic pilot for landing airplanes. If it is a one off script that happen to become a part of a temporary business process perhaps just a sample set of data, a desired output and small tool for comparing the results is enough.