2 ms·

Sometimes you're writing code because you want to see what happens. Richard P. Gabriel wrote a wonderful, moving essay about this recently, "I Throw Itching Po

by kragen 2y ago

Sometimes you're writing code because you want to see what happens. Richard P. Gabriel wrote a wonderful, moving essay about this recently, "I Throw Itching Powder at Tulips": https://www.dreamsongs.com/Files/Tulips.pdf https://www.dreamsongs.com/Files/Tulips.pdf > Software engineering is not what I do when I program. I am programming. I write software as part of doing science. I use software as a machine or instrument to explore how the mind / brain might work. Not when the mind’s thinking—that’s old AI. I mean when people are creating. I find that it's pretty common when I'm, say, tweaking gameplay in a game, that I have no trouble making the code behave the way I expect, but impossible to predict ahead of time how I will feel about different possible behaviors. I have to try them on to see how I feel in them. Adding tests to that feedback loop doesn't help; it just slows it down, dramatically slowing down the process of learning and exploration. On the other hand, when I am writing code to meet someone else's needs rather than as a form of self-expression or exploration, I usually want the code to behave in a specific way, at least in certain situations, and to keep behaving that way as I make other changes to it in the future, unless I change my mind. In those cases I think it's very rare for automated tests to slow me down instead of speeding me up. In languages with expressive static type systems like OCaml, the crossover happens a little later, because the static type system already constrains the program's behavior a lot. But automated tests still become very valuable very soon when I want a specific behavior. I've also, unfortunately, worked on projects where the test suite we had was worse than nothing. Invariably it wasn't because we were throwing itching powder at tulips so that the optimal test suite would be empty; it was because we'd done a bad job writing the tests.

simonw 2y ago
This is why I'm not at all a fan of pure test-driven development where you try to write the tests before the implementation. I think that's often a big time sink and can put people off writing tests entirely. Instead, I like tests-in-the-same-commit development. I write the implementation, tweak it until it works, then add a test that proves it does what it's supposed to before I commit the code. I wrote about that here: https://simonwillison.net/2022/Oct/29/the-perfect-commit/ https://simonwillison.net/2022/Oct/29/the-perfect-commit/ I often use snapshot testing for this, which is very fast to implement. I have a short note about one way to do that here: https://til.simonwillison.net/pytest/syrupy https://til.simonwillison.net/pytest/syrupy - and a longer piece here: https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/ https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests...
- kragen 2y ago
  I have found test-first development valuable sometimes. It's a proper subset of the times that automated testing in general is worthwhile; there's fairly strong empirical support in the academic literature for automated testing improving productivity and quality, but test-first seems to be a wash overall. Test-first development has some advantages in my experience: - You know the test isn't vacuous because you saw it fail when you first ran it. - You know you have thorough test coverage because you stopped adding code to the implementation when all the tests passed (at least temporarily). - You know the interface you're implementing isn't a huge pain in the ass to use because you would have noticed that halfway through writing the test case and redesigned it. - You have a shorter feedback loop between writing the implementation and seeing a test start passing, which can be a useful motivation hack and can also speed up your learning of the language and libraries you're using to build the implementation. But there are also disadvantages: - You can't use snapshot testing. I didn't know about syrupy, but doctest is also very amenable to snapshot testing. - The rapid feedback loop can be a motivational trap, getting you lost in tactical details (as Ousterhout famously worried in his debate with Martin) at the expense of attention to more strategic concerns that are ultimately more valuable. - Sometimes it's a bigger concern to define an interface that it's feasible to implement than one that's a pleasure to use. Test-first development backfires in that case, since first you wrote a test that's impossible to pass, then an article implementation that doesn't pass it, and then a second test that are tests the actually feasible implementation. A common case of this is where you left some necessary resource or information out of the interface. With respect to your piece about snapshot testing, which is excellent, I actually disagree about using tests to verify security properties. I mean, you should definitely have automated security tests, yes, but you should never rely on them. Code reviews, logical arguments, and ideally formal proofs are much better. Attackers can be counted on to try things you didn't think about when you were writing the tests. Randomized testing like Hypothesis or AFL can help, but not enough. But, at the other extreme, there are some programs for which automated testing itself is of no value.
  - simonw 2y ago
    I find test-first development useful for very specific things. If I'm building a Markdown parser for example, where the inputs and outputs are very clear, I may break out test-first development. In place of test-first for test verification I'll often deliberately break my new test (either in the test or the implementation), run it once to make sure it fails, then fix it again.

by kragen 2y ago

simonw 2y ago

This is why I'm not at all a fan of pure test-driven development where you try to write the tests before the implementation. I think that's often a big time sink and can put people off writing tests entirely. Instead, I like tests-in-the-same-commit development. I write the implementation, tweak it until it works, then add a test that proves it does what it's supposed to before I commit the code. I wrote about that here: https://simonwillison.net/2022/Oct/29/the-perfect-commit/ https://simonwillison.net/2022/Oct/29/the-perfect-commit/ I often use snapshot testing for this, which is very fast to implement. I have a short note about one way to do that here: https://til.simonwillison.net/pytest/syrupy https://til.simonwillison.net/pytest/syrupy - and a longer piece here: https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/ https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests...

kragen 2y ago
I have found test-first development valuable sometimes. It's a proper subset of the times that automated testing in general is worthwhile; there's fairly strong empirical support in the academic literature for automated testing improving productivity and quality, but test-first seems to be a wash overall. Test-first development has some advantages in my experience: - You know the test isn't vacuous because you saw it fail when you first ran it. - You know you have thorough test coverage because you stopped adding code to the implementation when all the tests passed (at least temporarily). - You know the interface you're implementing isn't a huge pain in the ass to use because you would have noticed that halfway through writing the test case and redesigned it. - You have a shorter feedback loop between writing the implementation and seeing a test start passing, which can be a useful motivation hack and can also speed up your learning of the language and libraries you're using to build the implementation. But there are also disadvantages: - You can't use snapshot testing. I didn't know about syrupy, but doctest is also very amenable to snapshot testing. - The rapid feedback loop can be a motivational trap, getting you lost in tactical details (as Ousterhout famously worried in his debate with Martin) at the expense of attention to more strategic concerns that are ultimately more valuable. - Sometimes it's a bigger concern to define an interface that it's feasible to implement than one that's a pleasure to use. Test-first development backfires in that case, since first you wrote a test that's impossible to pass, then an article implementation that doesn't pass it, and then a second test that are tests the actually feasible implementation. A common case of this is where you left some necessary resource or information out of the interface. With respect to your piece about snapshot testing, which is excellent, I actually disagree about using tests to verify security properties. I mean, you should definitely have automated security tests, yes, but you should never rely on them. Code reviews, logical arguments, and ideally formal proofs are much better. Attackers can be counted on to try things you didn't think about when you were writing the tests. Randomized testing like Hypothesis or AFL can help, but not enough. But, at the other extreme, there are some programs for which automated testing itself is of no value.
- simonw 2y ago
  I find test-first development useful for very specific things. If I'm building a Markdown parser for example, where the inputs and outputs are very clear, I may break out test-first development. In place of test-first for test verification I'll often deliberately break my new test (either in the test or the implementation), run it once to make sure it fails, then fix it again.