9 ms·
The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is ho
by alkonaut 9y ago
The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday.
In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word document in describing how code should behave. I very much prefer a massive, hard to maintain set of poorly written unit tests, to a large number of outdated documents describing how every bit of the system should work.
So is a big/bad test suite a burden? Sure. But is it a burden compared to maintaining specifications of other kinds? Is it a burden compared to working in a system with neither type of specification?
Further, the people writing long articles like this are (or at least were) very good developers. There is often an element of "good developers write good code, so just use good developers" in them. But writing good software with good developers was never the problem. The problem is making software that isn't terrible, with developers ranging from good to terrible, and most being mediocre.
- alfiedotwtf 9y agoThis. We tend to put a high value on code review, yet some don't value unit testing. Lean in and I'll tell you a secret: ... unit testing is automated, repeatable, code review!
- fvdessen 9y agoOnly if the tests and business code are not written by the same engineer.
- dorfsmay 9y agoWhich they typically no longer are when fixing a bug or refactoring.
- mannykannot 9y agoHow many times do you need to repeat the review of a given unit of code? (actually, that would have more chance of catching additional bugs than would rerunning the same unit tests.) Once you change the code, the existing unit tests are, by definition, invalidated (i.e. there is no value to being able to rerun them), while most higher-level testing will retain their validity. You have actually created an argument for automated higher-level testing (the author mentions that his position is often mistaken for an attack on test automation.)
- nradov 9y agoNo, changing the code doesn't invalidate the unit tests unless the API design contact changes.
- mannykannot 9y agoBy definition, unit tests test the implementation. A test at the API design contract level is an integration test. The only place where these are the same thing are pure (side-effect-free) functions that do not call other functions.
- blub 9y agoI don't see such a large overlap between code review and unit testing. Here's an example from one of my code reviews, where I discovered two issues: 1) A relaxed requirement was agreed to by the PO, but the remote developer wasn't aware of that and wrote code to fulfil the original complex requirement. The implementation was harder to understand and touched more areas of the code, leading to issue number 2. 2) By analysing the interactions between multiple units, I was able to determine that the code as implemented was in fact reading a setting too early, when it was not available, thereby having no effect at all compared to the already existing code. Ironically, this exact concern had prompted the negotiation with the PO which resulted in the relaxed requirement. Interestingly the error was not caught by unit tests, because it didn't happen in a single unit. It wasn't caught by integration tests, because that particular online component was mocked and it passed code review by two other developers.
- eric_h 9y agoStatic analysis is another tool in the toolbox, and invaluable as well.
- jryan49 9y agoYour unit tests check the architecture of the code and the implementation is maintainable? I say that cause that's what I use code reviews for :P
- kafkaesq 9y ago... unit testing is automated, repeatable, code review! Except when it isn't. "Unit tests can be one form of code review (that also happens to have the advantage of being automated and repeatable). But should never be mistaken as substitute for actually understanding what the code does, why it does it and how it got that way. Which requires an incompressible amount of effort, analysis, and plain and pure grit", would be another take on the matter.
- crdoconnor 9y ago>That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Unit tests are better than nothing, but integration tests usually serve as a better specification still. Unit tests are ok at specifying low level side-effect-less modules though.
- mannykannot 9y agoYes, and furthermore, due to the combinatorial explosion, and as explained in the article, unit tests can only cover an infinitesimal fraction of all the possibilities at that level. The sort of interfaced-based testing favored in the article ameliorates the problem because it uses the inherent abstractions of the code to prune the problem space. In addition, if you depend on unit tests, you cannot tell if a change has broken the system - you can expand or rewrite your tests to cover the change (if we put aside the question of what requirements these tests are testing), but that cannot tell you if you have, for example, violated a system-wide constraint. Only testing at higher levels of abstraction, up to the complete-system level, can help you with that. Therefore, to the extent that the "the code is the specification" argument is a valid one, it actually leads to the conclusion that Coplien is right.
- ZenoArrow 9y agoIn my opinion, the way that most unit tests are written is mostly a waste of effort, but there are a class of unit tests that are much better suited to the task. The class of unit tests I'm referring to is property-based tests. The most famous property-based testing framework is QuickCheck, but there are plenty of others, e.g. FsCheck for C#/F#, ScalaCheck for Scala, Hypothesis for Python, etc... The general idea being that you write out a specification that the code should meet, and the test parameters are determined at runtime, rather than being hardcoded. There's more to it than that, but that's the general idea. If you want a simple introduction, I'd recommend this video: https://www.youtube.com/watch?v=AfaNEebCDos https://www.youtube.com/watch?v=AfaNEebCDos
- mattmanser 9y agoThe article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point?
- alkonaut 9y agoMy point was that even bad unit tests are better than both good/bad word docs or no unit tests. So even bad tests written by bad programmers have a value. They have a large COST too (which is what he's arguing), my argument was merely that the value might not be less than the cost, which is his assertion. > Software engineering research has shown that the most costeffective > places to remove bugs are during the transition from > analysis to design, in design itself, and in the disciplines of > coding. It's much easier to avoid putting bugs in than to take > them out. Everyone knows that. It's just beating a dead horse. But companies demonstrably want to ship buggy and feature rich software fast, rather than ship well designed and lean software. And b) don't want to pay for developers of the kind that write good code from the beginning. So with that out of the way, the whole point of real world development methodology is to make sure that the feature-bloated code hastily written by average developers doesn't lack a specification, doesn't deteriorate over time into something that has to be abandoned, and doesn't have such a high risk of modification/refactoring that it can't be maintained. Now there are some good points to 1.4 "code is better than tests" actually applies: make asserts in the code rather than in the tests, where possible. Fully agreee with that. Or even better I'd say: don't "assert" things, just make the invalid state impossible. This is what types are for. If you can't take a null into a method, don't even accept an object that could be null by accident. Accept an Option type. Yes even in Java. Even in bloody C. Anything that a compiler could have caught shouldn't be either a runtime assert nor a test. > if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Well firstly they are usually better than no tests. And second, they tend to fail too often, rather than too little. This is what causes the enormous cost incurred by these tests. But apart from that 90% unnecessary cost (test failures are failures of the code rather than the costs, so after changing the code you have to change the test) - they still do add a lot of confidence for refactoring. Because in my experience with a large but bad test suite, there are often false positives but false negatives are much rarer.
- brightball 9y agoThat’s actually one of the reasons I like Elixirs approach to inline documentation so much. If you include a code example, the example is run as part of the test suite. Makes is dead simple to automatically include simple unit tests right where the code is along with documentation that had to remain up to date. http://whatdidilearn.info/2017/10/23/writing-documentation-in-elixir.html http://whatdidilearn.info/2017/10/23/writing-documentation-i...
- floatboth 9y agoThis is also somewhat popular in Python, Haskell and Rust. Nice, but doesn't seem applicable to all the situations. Works quite well for collections of simple independent functions e.g. I did that for a regex library https://github.com/myfreeweb/pcre-heavy/blob/9873fb019873340b212780d12595c0f66ec88613/library/Text/Regex/PCRE/Heavy.hs#L177-L196 https://github.com/myfreeweb/pcre-heavy/blob/9873fb019873340... — but didn't use much for other projects…
- brightball 9y agoYea, definitely not for everything. It's more of a complement to the rest of the test suite and also verifies that the documentation/examples in code comments are still accurate.
- yorwba 9y ago> Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Maybe, but there's no reason to write such tests by hand. Just automatically test each method against the previous version and see what changed. I think the generated results would be too noisy and developers would stop paying attention to test failures, but maybe a strict enough code review process could prevent that.
- dragonwriter 9y ago> The normal practice for large scale codebases in complex domains is "the code is the spec". ”The code is the spec” is just a way of saying “we don't know what the code is supposed to do, and the intent was never, even initially, properly validated by domain experts, or, if it was, we tossed the knowledge generated in that process.” This is especially the case if that phrase is used about a large system in a complex domain.
- sametmax 9y agoI agree. Espacially since most projects I worked on didn't have a doc at all. When the doc existed, it was always obsolete. Unit tests are not a silver bullet, but if you request passing tests and coverage to put in prod, you basically request a minimal & always up to date documentation. And even with all the (very real) downsides of it, it still a huge plus. Besides, a lot of persons tend to see tests in a very rigid way. You don't have to do tdd. You don't have to do have full coverage. You don't even need all your tests to be unit tests, you can mix end2end and functional tests with it. You can have hacky mocks in it too. It doesn't need to be perfect for your team to benefit from it. Actually, I would say that you probably want it not to be perfect so that the benefits outweight the costs. Cause they, tests have a huge cost.
- JamesBarney 9y agoMost unit tests don't test functionality at the business requirement level. Most unit tests I've seen are more fine grained. I think keeping your requirement documentation as comments on unit tests that are a literal translations of those documentations is amazing and very helpful. But I would argue that represents in my experience just 10% of the unit tests I see. Instead I see a lot more testing of implementation details.
- derefr 9y ago> That is, the only specification for how the system should work is how it worked yesterday. You'd think, if this was the desired model, you could partially automate the "writing unit tests" part of this process. (Integration tests no, but unit tests yes.) The "spec" for the unit tests is already there, in the form of the worktree of the previous, known-good commit. That means that, in a dynamic language, you'd just need a "test suite" consisting of a series of example calls to functions. No outputs specified, no assertions—just some valid input parameters to let the test-harness call the functions. (In a static language, you wouldn't even need that; the test-harness could act like a fuzzer, generating inputs from each function's domain automatically.) The tooling would then just compare the outputs of the functions in the known-good-build, to the outputs of the same functions from your worktree. Anywhere they differ is an "assertion failure." You'd have to either fix the code, or add a pragma above the function to specify that the API has changed. (Though, hopefully, such pragmas would be onerous enough to get people to mostly add new API surface for altered functionality, rather than in-place modifying existing guarantees.) A pre-commit hook would then strip the pragmas from the finalized commit. (They would be invalid as of the next commit, after all.) Interestingly, given such pragmas, the pre-commit hook could also automatically derive a semver tag for the new commit. No pragmas? Patch. Pragmas on functions? Minor version. Pragmas on entire modules? Major version.
- njharman 9y ago> The "spec" for the unit tests is already there Ah, but the spec is often really poorly defined. That is all sorts of edge cases and uncommon code paths do not work as imagined. And, of course, it is full of bugs. I have found countless "bugs" and issues when writing tests for existing code. I also often have to refactor the code to make it clearer what it's doing or to make it testable. Unit tests are a development tool more than they are a testing tool. They are a means to produce correct, well documented, well speced, cleanly separated code.