18 ms·
Unit Testing Doesn’t Affect Codebases the Way You Would Think
- fulafel 9y agoWhat about causality? It seems plausible that people would be more motivated to write unit tests for complex functions that are hard to gain cofidence in otherwise.
- dozzie 9y agoIt seems plausible that people would write shorter, clearer functions that have no possibility of being incorrect if they had no tests to gain the confidence of them being correct. Also, designing for testability only gives you testability. If you ditch this requirement, suddenly you're free to design directly for ease of use, not for some proxy like testability.
- dpark 9y ago> It seems plausible that people would write shorter, clearer functions that have no possibility of being incorrect This seems exceedingly implausible. If people were capable of writing perfectly correct functions consistently, they’d do so. Also, in my experience the people who write tests are generally the ones who write good small functions. The people writing massive dense functions are rarely writing good (or often any) tests. (This obviously diverges from the study’s results, though, so grain of salt.)
- sidlls 9y agoIn my experience the people who are the strongest advocates of unit testing write small functions. I left off "good" because I don't agree that "small functions" are "good" necessarily. Often the functions are in fact too small. They properly belong in a larger function as part of that function's implementation, but are "factored out" solely for the purpose of unit testing that piece of the algorithm inside the function. Usually these small functions are also only called once. In that case especially it leads to messy, bloated code that is in fact harder to reason about, harder to maintain and less efficient besides.
- arthur_pryor 9y ago> Usually these small functions are also only called once. In that case especially it leads to messy, bloated code that is in fact harder to reason about, harder to maintain and less efficient besides. the flipside is that if this approach is done well, you end up with things broken down along the natural abstraction lines you'd use, and if you trust that your small functions do the right thing (are tested themselves) and are called correctly (their callers are tested well), it tends to make it much easier to reason about the code, IMHO. but yeah, also, sometimes larger functions and more integration-y tests are the way to go. or the above fine breakdown + unit tests + integration tests. depends on the situation, i think.
- btilly 9y agoThe tendency is to lock in the first abstraction line that you thought of..which rarely ages well with future experience. My rule of thumb is that something needs to be a function if it is called 3x, doesn't if it is only called from one place, and if it happens 2x it is a judgement call whether to leave cross-referencing comments between the spots or to factor it out into a function. (If it is unlikely that I will need a third and the common code is difficult to extract, I will leave comments.)
- dllthomas 9y agoSure, either way sounds plausible - the point is that this study doesn't tell us how much of an effect runs each way.
- ehvatum 9y agoYou are absolutely correct. GLSL shaders were excruciatingly difficult to test and debug, before tooling caught up. As a result, I took great care to write small and easily understood functions. It did much to improve my work in Python and C++, where debugging and unit testing were available. It's irritating to devise guidelines for good development, in general, and then have people build a cult around them. People honestly believe that unit testing is the only incentive for writing good code. This is equivalent to ascribing all good things to Dear Leader and all bad things to an out group.
- mannykannot 9y agoThere is no guarantee that all complex problems can be solved in an obviously correct way if you use small enough functions. As the number of functions increases, the complexity tends to get pushed into the interactions between them - and you need integration testing to look for bugs in how they interact. If you are writing small functions - and, generally speaking, you should - integration testing becomes more important than unit testing.
- danblick 9y agoThis was my thought too. The findings are written as "unit tests cause X" when really what is shown is "unit tests are associated with X". Another explanation for the results could be common cause: Maybe packages with more unit tests are also professionally developed, or written by people interested in software methodology, or are older; and those aspects are also associated with method length/decomposition patterns/etc.
- skywhopper 9y agoI like the idea of this study. I wish there were more data behind it. I’d love to see comparisons based on language, or testing tool, for example. Or code base size, age, author count, etc. There also needs to be some weighting of confidence based on how many code bases are in each group. eg, Since there are only four code bases in the >50% group, it’s dangerous to draw conclusions based on a trend line when that group is equally weighted with the far more populous groups. Still, I’m hoping the author expands his datasets and analytical methods as time goes on and I’d be interested to see this topic revisited with a larger sample.
- crdoconnor 9y agoIf you define unit tests as tests which test non-divisible units on a low level, then by definition they are a form of tight coupling. The most loosely coupled form of test would be an end to end test (most loosely coupled, but obviously not without its disadvantages). Since the pain of dealing with one form of tight coupling is magnified by having another form present, the presence of unit tests act as a sort of canary in the coalmine: if your unit tests are extra painful today, that's probably because the part they are connecting with has a high degree of tight coupling. That might (or might not) lead you to refactor and decouple the rest of the code base in an attempt to mute that pain. Either way you'll feel the pain. I believe this effect is real. I also don't do this myself, because I don't believe in self flagellation as a means to enlightenment. The best thing to couple your tests to depends upon a range of factors, IMHO, and getting religious about where to couple anything is the worst thing you could do, especially if that religion dictates tight coupling (unit testing all the things). Method length and complexity probably go up because if you focus on loose coupling (to mute the "unit testing pain") to the exclusion of other good code qualities - less code/less complex code - then those qualities will be sacrificed.
- cbanek 9y agoLove this article. Very interesting, and I love how the results are able to be quantified. Coming from some projects that had a very heavy unit testing culture, I can agree that unit testing doesn't always necessarily have the intended results on "quality" and can really shape a codebase. I feel the root of the reason is that testing is not the same thing as development. Truly testing things involves a mentality that is different, it involves understanding how mulitple pieces fit together and operate in reality, not in some unit testing framework. A lot of developers look at unit tests the same way they look at development, once they work, it's done. But testing is proving they work in all the right conditions, not just getting it to work once. I've personally seen very simple code convoluted so that it could be unit tested, and how those convolutions can easily add bugs and additional layers of indirection that hurt performance (as well as add unnecessary code, which needs maintenance). Many times these tests could have been written with the existing code as long as it used some other component... but then it wouldn't be "unit testing" only to see these components not work together on the first runtime. (For some reason, places that rely heavily on unit testing seem to have fewer overall or integration tests, because they spend so much time and effort on unit testing) I'd love to see a study on how many lines of code change because of unit tests vs in non unit tested codebases. Both in terms of how many unit tests need to change and the size of the change itself.
- pqh 9y agoI'm curious what language and techniques you mean this applies to. In my experience, I've seen a lot of code that was written in a way that was not unit-testable, for what was the same reason that later turned out to be poorly-maintainable.
- loup-vaillant 9y agoSome seem to use (their personal idea of) unit testability as a proxy for maintainability or quality. That is often invalid. I've written some high-quality code and tested it to death¹, only to be told that it probably wasn't unit testable (because dependencies were hard coded instead of injected, IIRC). Making that code unit testable would have complicated it, thus lowering its maintainability. [1] Not a single bug has ever been found after my tests.
- danielovichdk 9y agoI believe you came to a result to quickly by the way you measure the code bases chosen. Lot's of things here that you rund over very quickly without showing anything to back up you right/wrong results. And sure, tests is of course having an impact on design. As it should have. There is consensus around, at least for the oop paradigm, that this is the case. I also belive you try to make a once size fit all for measuring tests and that is simply not fair in terms of how differently these codebases are designed and implemented.
- siliconc0w 9y agoVery cool, I'd love to see a more scientific approach to understanding software development methodologies. What might be interesting is a metric to see how 'refactorable' a codebase is - i.e once a class is created what is the probability there would be significant changes in the future for highly tested vs untested code. Also the same metrics along a dimension of typed vs untyped languages (my hypothesis would be that unit testing would show more benefits for untyped languages like Python than .NET).
- ams6110 9y agoI was curious about You can’t (without a lot of chicanery) unit test private methods. Why is that? Maybe you can't test them directly but private methods should be executed and thus indirectly testable by public methods. If there is code in private methods that can't be executed by any public method invocation, it's unnecessary code isn't it?
- taeric 9y agoI've had a similar debate. Someone kept trying to exclude some of our data objects from test coverage reports. I agreed we shouldn't bother testing getter/setters. But, those object better be used in other tests we are writing.
- pleasecalllater 9y agoAs usually: it depends. Sometimes testing getters/setters is a good thing.
- taeric 9y agoMy point is when we checked the report, the file was already getting covered. If your tests of the system aren't using the data objects of your system, you have larger problems.
- JustSomeNobody 9y agoThe typical argument for getters/setters is that there is some logic that should be performed and that's why the fields should not be public. If there's logic, they should be tested. If there's no logic, they should be public fields.
- kkapelon 9y agoInteresting article but I think the number of projects (100) is very low for valid conclusions. If you have already automated the process maybe you should try adding more projects. >Unit Testing Appears to Make Complexity and Method Length Worse I think it is the other way around. People are forced to write unit tests for complex and big methods. So not as negative as your present it.
- theptip 9y ago> I think it is the other way around. People are forced to write unit tests for complex and big methods. Intuitively this seems correct to me, at least for the code-first paradigm. I do wonder if this holds in the TDD paradigm though, since in that case there are (should be?) tests for everything regardless of complexity. It would be interesting to try to answer this question with further experimentation along the lines of the OP, though I can't think of any way of detecting/classifying projects as TDD vs. not-TDD without contacting the repository authors (which would be time-consuming, potentially prohibitively so).
- daxfohl 9y agoMy guess is that both variables in these results are almost entirely driven by external factors. Team size, whether it's pro or side project, deadlines, experience/talent of developers, whether tests are required or optional, the language. Hard to draw direct correlations given the sample size.
- vikiomega9 9y agoWhy do we expect fewer lines of code with more unit tests? I don't quite follow this.
- AdeptusAquinas 9y agoPer function, not overall. A function with more lines of code is doing more, and so is harder to test
- sidlls 9y agoIt isn't harder to test, necessarily. It's harder to unit-test. The distinction is important. Perhaps a single test that TDD people might call an "integration test" suffices for that function. Often it does, in fact.
- AdeptusAquinas 9y agoAn integration test isn't a test for a function. Its a test of the system as a whole. On a per function level, more code in a function makes it harder to test, whether you are unit testing it or somehow manually testing it.
- sidlls 9y agoIt seems agreeing on what a "unit" is is elusive. If a function calls another function, for example, testing the caller without mocking the callee, I'm told often, is not truly a unit test.
- spc476 9y agoNo, a function can be long but have a low cyclomatic complexity score [1] (being a list of do_this(), do_that(), do_somethingelse(), etc) while a short function can have a higher cyclomatic complexity score (lots of fiddly logic). [1] https://en.wikipedia.org/wiki/Cyclomatic_complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity
- blahedo 9y agoI love that this post is quantifying results over actual data—a huge improvement over the speculative/anecdotal pattern that a lot of these analyses have. But the conclusion that complexity and LOC both are higher in codebases with much unit testing... seems very weak. The complexity graph in particular looks very level, with the trendline driven almost entirely by the very low datum for 0-10%, and the LOC graph looks almost as suspicious. That low datum definitely demands further explanation before any conclusions are drawn on those two. To me, it also inspires two further questions about the LOC methodology: 1) how sensitive is the measure to different coding styles? Does it include the method header? Does it include the closing bracket for the whole function? Does it include open bracket for the function when on a line by itself? Does it include any bracket on a line by itself? With the average method length ranging from 2 to 5, coding conventions on brackets could make a substantial difference, and if coding conventions correlate with testing philosophy at all (for whatever reason), that's a possible threat to validity. 2) Could these averages be dominated by a few larger projects? That is, is the average computed as average (length of all methods, all projects) or average ( for each project, (average (length of methods))) ? If the former, larger projects would dominate the average. (True of all the other questions that involve counts and averages too, actually.)
- couchand 9y agoIn additions to the methodology concerns you raised, I have to question the use of a "linear" trend line, particularly when one axis is arbitrary buckets of unequal size.
- theptip 9y agoThe consistent bump in the 0% bucket suggests that there is a significant measurement error going on here. The author acknowledges this: > So if any of these codebases are using a different framework, they’ll be counted as non-test codebases (which I recognize as a threat to validity). I'm inclined to say the 0% bucket should be thrown out, since it's impossible to disambiguate between "no tests" and "tests I can't detect". All of the other buckets are quite likely to be correctly classified.
- andy_ppp 9y agoAwesome article, so interesting. I wonder how language design affects this, I write much better code in Elixir than I do in Node for example. Maybe though the solution is simple; if you value having low cyclometric complexity/method length/etc. don’t let people commit stuff unless it passes the required level. This is certainly working well for me with prettier+eslint... I’ll look into other tools based on the code smells in this article I think!
- fnl 9y agoInteresting article, but too many uncertainties: 100 examples only, the 30 code bases w/o unit tests might have tests the author didn't check or see, and there is likely a cluster effect in this sample to be found. But interesting enough to warrant a large scale investigation to see if those results would really hold.
- josteink 9y agoDisclaimer: unit-test proponent hypothesizing. > But taking the data at face value, the trend lines speak clearly. More test methods in codebases predict more cyclomatic complexity per method and more lines of code per method My initial reaction to this is that having more tests means your code accommodates more and increasingly complex cases and edge-cases in general, because you’re not going to write 10 test-cases testing the same logic (or combination of logic). That’s going to mean more code and more complex code either you like it or not. The projects with fewer test-cases may not handle as many edge-cases, and maybe they don’t have this complexity (and the associated tests) because they don’t need it yet? If so, tests themselves are not bad, but may just be indicative of the real world cost of real world complexity?
- milesvp 9y agoQuite possibly. I will assert, though, from my experience, and the wisdom I've garnered from talking to people around unit testing, that as code assypmtotically approach 100% code coverage, the complexity of said code will tend to go up exponetially (or worse). I don't have significant proof other than to say that things like DI and IoC start to get used in places that make little sense, which means complexity that wouldn't exist without tests. I also have anecdotes of defensive code I've written, with code branches that 'will never execute'. Branches that are there in case I missed something subtle, that simply log an error, or otherwise throw some exception. I can verify the code works before walling it off, but if you ask me to unit test those lines of code, now I need ways to set the internals of a function externally, or otherwise add code to allow variables to be set in a way that shouldn't be possible. This leads to complexity. And if you asked me to test it, I'd just quietly delete the defensive code rather than poison internal state variables.
- josteink 9y ago> as code assypmtotically approach 100% code coverage, the complexity of said code will tend to go up exponetially (or worse). I think attempting to reach 100% coverage is a good direction to steer against while you have low coverage, but not worth it as a an ultimate goal. As you get closer (even as as TDD-proponent), I'm pretty sure the costs outweighs the gains, and you get problems like the ones you describe here. Where a "reasonable" cutoff point may be... Now that's a harder discussion :)
- zebraflask 9y agoUnit testing tells you what you already know. It's kind of pointless, when you think about it (unpopular opinion). If you have a handle on your code there's rarely any need for it.
- sinhpham 9y agoNo. Software evolve and there's value in knowing your assumptions still hold after the change/refactor.
- zebraflask 9y agoI don't understand that but thank you for your opinion.
- spc476 9y agoI've been using a program I wrote for 18 years now. The code today is not the same as the code 18 years back. I have slowly modified the code as ideas for new features come up, old features removed because they aren't used anymore, and foundational code rewritten as I found better implementations (or the previous implementation was a mistake [1][2]. Making a sweeping change (like re-implementing some core code) is dangerous because of the threat of new bugs, which is why tests are important (although I tent to prefer integration tests over unit tests). [1] I used to have an error logging mechanism in place that would record where in the code the error happened (in addition to other information) and how it propagated up through the code. While it wasn't that hard to use, per se, it was bothersome when I had to add an error (each error had a unique ID and because of language issues and tooling, that was a manual process). And it really never paid back its implementation cost in terms of reporting, so I finally ripped it out. [2] I had my own version of C's FILE* [3] that ended up being a horrible abstraction---it was so confusing that I could never remember how to use it, and I wrote the code. I got fed up with it, and ripped that out, replacing it with native IO calls. [3] For stupid reasons now that I think back on it.
- kovacs 9y agoThe simple way to say it is that once your code gets too complicated to keep all the execution paths in your head (sooner than you think), tests are what enable you to make sure you didn't forget an assumption you made about how it should work. Do you like going back to manually verify things still worked after you make a foundational change? Or do you just trust yourself to know you've not impacted anything negatively? Do you honestly believe that's good use of your brain power even if true? Do you enjoy being married to that code? Because that's exactly what happens as nobody but you can work on it. That strategy works out well for employees to become key men and get big retention bonuses so there is some merit to your approach :-) Not to mention the team impact to cowboy coding. It's a selfish approach to software development and is often undertaken by bully developers that say things like "use the source" as a way to make you feel dumb for not wanting to spend all day trying to untangle their likely insane code. All because they were too lazy to use a little empathy and be a good teammate. To recap: Tests document and improve your design, verify functionality, provide leverage to accelerate development, and reduce "bus factor" risk. I don't care what a small sample data set says to the contrary there's really little debate that when wielded properly these tools and techniques do improve software on multiple dimensions that go beyond the software itself and heavily impact the business. 20+ years of product development experience in lots of teams/products inform this opinion. Let's see the data on how your business grinds to a halt when these tools aren't used and the code base gets increasingly larger and complex and people leave. Tell me about how you don't need those automated tests as you approach technical debt bankruptcy and all your key people have left. "Use the source" is a terrible response to that problem. Rewrites can be fun though so maybe that's the answer. It sure worked out well for Netscape. ;-)
- ysleepy 9y agoA generally interesting study subject. But it seems not quite thought through. Projects might only invest in unit tests once a project exceeds a certain complexity, or has seen refactorings with age which made the need for tests clear. I'm not sure how to normalize properly, but maybe projects solving a similar problem could be compared. Like different ORMs or Databases or Kernels.
- spjt 9y agoI can speculate as to why this would happen. People expect unit-tested code to have shorter methods because it's easier to write tests for shorter methods. However, writing a new test for a new method can be harder than just stuffing that code in some other method that already has a test. Depending on how testing is enforced, you might be able to at get away with not having to do anything at all if the existing test still passes. If you're in a hurry and the test fails, just change the assertion to whatever happens. Is it right? I dunno, the test says it's working.
- kgr 9y agoThe results are compatible with "risk homeostasis" or "risk compensation" theory. People who use unit tests have an increased sense of safety, so are more comfortable living with less-safe designs and code-bases. "Risk compensation is a theory which suggests that people typically adjust their behavior in response to the perceived level of risk, becoming more careful where they sense greater risk and less careful if they feel more protected. Although usually small in comparison to the fundamental benefits of safety interventions, it may result in a lower net benefit than expected" -- https://en.wikipedia.org/wiki/Risk_compensation https://en.wikipedia.org/wiki/Risk_compensation http://injuryprevention.bmj.com/content/4/2/89 http://injuryprevention.bmj.com/content/4/2/89
- megamindbrian2 9y agoThis is exactly what I want to talk about.
- ajmurmann 9y agoI would love analysis like this based on all the data that codeclimate must have. The fact that the test coverage buckets and at 50%+ makes me think that the number of projects that were actually TDDed is very low. I've never seen a project that was truly TDDed with less than 98%. Typically the 2% come from a config file or something like that.
- davidbanham 9y agoThis is a great direction for research, but wake me when there's been any measurement of significance in the results. Trendlines are interesting but until we've proven that the variation we're seeing is actually statistically significant with regard to the sample size, these aren't actionable results.
- zo7 9y agoThe author mentions that they only analyzed the collection of C# repositories that they used in their study on Singletons, but I would expect the results to be different depending on the language used. In my experience with Python, unit testing is essential to catch careless errors (typos, bad ducks, etc. that can't be caught by a linter) that would cause your program to crash at runtime, so there's a slightly different motivation for writing unit tests. It's a less restrictive language so you don't have the issues with private methods like the author mentioned, and is (at least to me) a bit less of a strain to write tests. Unit testing might correlate more with good hygiene for a Python codebase because testing is vital to check correctness and there are fewer situations where you have to mangle your code for the tests, whereas it may not in C# because you get guarantees from the type system that makes testing less vital and writing tests is more of a nuisance.
- mrlyc 9y agoWhen I was writing device drivers for the middleware people, I found that unit tests were useful for testing the code and making sure the hardware was doing what it was supposed to. Along with the drivers, I released the test code to middleware so they could use it in their own code.
- iamleppert 9y agoThis is just my experience, but the engineers and managers who are the biggest proponents of unit tests don't write tests themselves for their own work. I've noticed this many times in many different jobs and I'm curious as to why it is?
- DmitryOlshansky 9y agoWhile an interesting read, I can’t help but wonder about two things: 1. This is obviously .NET projects only 2. 100 is a drop in the bucket even if we take C#-only projects. How good are statistical results, is it representative?