3 ms·
I have found test-first development valuable sometimes. It's a proper subset of the times that automated testing in general is worthwhile; there's fairly stron
by kragen 2y ago
I have found test-first development valuable sometimes. It's a proper subset of the times that automated testing in general is worthwhile; there's fairly strong empirical support in the academic literature for automated testing improving productivity and quality, but test-first seems to be a wash overall.
Test-first development has some advantages in my experience:
- You know the test isn't vacuous because you saw it fail when you first ran it.
- You know you have thorough test coverage because you stopped adding code to the implementation when all the tests passed (at least temporarily).
- You know the interface you're implementing isn't a huge pain in the ass to use because you would have noticed that halfway through writing the test case and redesigned it.
- You have a shorter feedback loop between writing the implementation and seeing a test start passing, which can be a useful motivation hack and can also speed up your learning of the language and libraries you're using to build the implementation.
But there are also disadvantages:
- You can't use snapshot testing. I didn't know about syrupy, but doctest is also very amenable to snapshot testing.
- The rapid feedback loop can be a motivational trap, getting you lost in tactical details (as Ousterhout famously worried in his debate with Martin) at the expense of attention to more strategic concerns that are ultimately more valuable.
- Sometimes it's a bigger concern to define an interface that it's feasible to implement than one that's a pleasure to use. Test-first development backfires in that case, since first you wrote a test that's impossible to pass, then an article implementation that doesn't pass it, and then a second test that are tests the actually feasible implementation. A common case of this is where you left some necessary resource or information out of the interface.
With respect to your piece about snapshot testing, which is excellent, I actually disagree about using tests to verify security properties. I mean, you should definitely have automated security tests, yes, but you should never rely on them. Code reviews, logical arguments, and ideally formal proofs are much better. Attackers can be counted on to try things you didn't think about when you were writing the tests. Randomized testing like Hypothesis or AFL can help, but not enough.
But, at the other extreme, there are some programs for which automated testing itself is of no value.
- simonw 2y agoI find test-first development useful for very specific things. If I'm building a Markdown parser for example, where the inputs and outputs are very clear, I may break out test-first development. In place of test-first for test verification I'll often deliberately break my new test (either in the test or the implementation), run it once to make sure it fails, then fix it again.
- kragen 2y agoYeah, that makes a lot of sense to me.