3 ms·

I've found that this depends heavily on the software in question. For most of what I do (basic line-of-business type webapps & servers) the vast majority of th

I've found that this depends heavily on the software in question. For most of what I do (basic line-of-business type webapps & servers) the vast majority of the code works on the first try and is obvious enough that I never break it; during development there tends to be a few sections that break repeatedly while I'm changing other things, and those are the sections I write tests for. (Generally this is a few percent of the entire codebase.) On the other hand, more 'computer-sciencey' projects tend to need a correspondingly larger proportion of test cases. The most dramatic example of this was when I built a Lisp interpreter as a side project: this was the only software I've ever written fully test-first, and I don't think I would have been able to get it working at all without a full test suite that hit every line of code at least twice.

marcosdumay 10y ago
As a follow-up question, did you (or could you have) develop that lisp interpreter on the purest TDD approach of: - Write a few tests that fail - Write code until your tests stop failing - Repeat until the program is done? In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests.
- wolfgang42 10y ago
  > did you (or could you have) develop that lisp interpreter on the purest TDD approach Yes, that's what I meant by "fully test-first". I was following along with Paul Graham's The Roots of Lisp[1], so I had a convenient set of pre-made test cases. I would start by copying his examples into a test case: it('exec atom on a symbol is true', () => { var symbol = Symbols.get_symbol_named('foo') expect(exec(empty_scope, [natives.atom, [natives.quote, symbol]])).to.equal(true) }) it('exec atom on a non-empty list is false', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, ['a']]])).to.equal(false) }) it('exec atom on an empty list is true', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, []]])).to.equal(true) }) And then write my code: natives.atom = new Native(function (item) { var i = exec(this, item) return (i instanceof Array && i.length == 0) || Symbols.is_symbol(i) }) Once I got through the paper that way, I had a complete Lisp interpreter working, and wrote a few small programs in it with no trouble at all. > In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests. This has been my experience as well. As I say, this is the only program I've ever written fully test-first; I think it was only possible because I already knew exactly what the inputs and outputs were going to be (having them well-specified by virtue of being a Lisp) and which ones I needed to implement to be able to get later ones working (with the help of the paper). Generally I find it's not possible to work this way; my usual approach for solving problems where the code isn't immediately obvious is to write (and generally rewrite) in parallel: - A set of usage examples and/or documentation, to clarify what exactly I'm trying to do and make sure the interface makes sense. - The actual implementation (or, in early stages, some pseudocode). - The tests (if any), to check that what I've written actually works. Often these are the same as the usage examples. Frequently questions raised while writing one of these will influcence the others, often significantly; often I'll only think of an edge case while I'm writing the code and then have to go back and add it to the tests, or realize that the code could be simplified by changing the way the API works. [1]: http://www.paulgraham.com/rootsoflisp.html http://www.paulgraham.com/rootsoflisp.html
  - kazinator 10y ago
    I'd wait until the Lisp interpreter works significantly and then write reasonable code in Lisp like: (it (atom 'foo) t) If it fails, it just says something like: test failed: (atom 'foo) returned nil; expected t. So no need to have a string there. Save the "blub-level" testing for things that are not testable through Lisp. (Or, really, things you are justified in not wanting to expose such that they are testable.) (Sure, it could be the case that both the interpreter are wrong, and the atom function are wrong such that the test passes. That level of breakage isn't likely going to pass much of a significantly detailed test suite.)
    - wolfgang42 10y ago
      If I'd waited until the interpreter to work before I wrote tests, I'd never have gotten a working interpreter. For atom the implementation is obvious, but the more complex functions were riddled with bugs on the first implementation, and the lexical scope handling took about three complete rewrites before I got it right. If this had gone any further than the three days of off-hours time I spent, I certainly would have rewritten the test suite in Lisp; however it was primarily an academic exercise to really 'get' how Lisps worked and push my comfort level; writing all of my tests in the language I was trying to write would have pushed things a bit to far. In fact, I never got around to writing a parser, so even if I'd written them in Lisp it would have looked like this: var Symbols = require('../Symbols') var natives = require('../natives') var foo = Symbols.get_symbol_named('foo') module.exports = [ natives.tests, [natives.it, [natives.atom, [natives.quote, foo]], true], [natives.it, [natives.atom, [natives.quote, []]], true], ]
- steveklabnik 10y ago
  I did this with brainfuck.

marcosdumay 10y ago

As a follow-up question, did you (or could you have) develop that lisp interpreter on the purest TDD approach of: - Write a few tests that fail - Write code until your tests stop failing - Repeat until the program is done? In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests.

wolfgang42 10y ago
> did you (or could you have) develop that lisp interpreter on the purest TDD approach Yes, that's what I meant by "fully test-first". I was following along with Paul Graham's The Roots of Lisp[1], so I had a convenient set of pre-made test cases. I would start by copying his examples into a test case: it('exec atom on a symbol is true', () => { var symbol = Symbols.get_symbol_named('foo') expect(exec(empty_scope, [natives.atom, [natives.quote, symbol]])).to.equal(true) }) it('exec atom on a non-empty list is false', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, ['a']]])).to.equal(false) }) it('exec atom on an empty list is true', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, []]])).to.equal(true) }) And then write my code: natives.atom = new Native(function (item) { var i = exec(this, item) return (i instanceof Array && i.length == 0) || Symbols.is_symbol(i) }) Once I got through the paper that way, I had a complete Lisp interpreter working, and wrote a few small programs in it with no trouble at all. > In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests. This has been my experience as well. As I say, this is the only program I've ever written fully test-first; I think it was only possible because I already knew exactly what the inputs and outputs were going to be (having them well-specified by virtue of being a Lisp) and which ones I needed to implement to be able to get later ones working (with the help of the paper). Generally I find it's not possible to work this way; my usual approach for solving problems where the code isn't immediately obvious is to write (and generally rewrite) in parallel: - A set of usage examples and/or documentation, to clarify what exactly I'm trying to do and make sure the interface makes sense. - The actual implementation (or, in early stages, some pseudocode). - The tests (if any), to check that what I've written actually works. Often these are the same as the usage examples. Frequently questions raised while writing one of these will influcence the others, often significantly; often I'll only think of an edge case while I'm writing the code and then have to go back and add it to the tests, or realize that the code could be simplified by changing the way the API works. [1]: http://www.paulgraham.com/rootsoflisp.html http://www.paulgraham.com/rootsoflisp.html
- kazinator 10y ago
  I'd wait until the Lisp interpreter works significantly and then write reasonable code in Lisp like: (it (atom 'foo) t) If it fails, it just says something like: test failed: (atom 'foo) returned nil; expected t. So no need to have a string there. Save the "blub-level" testing for things that are not testable through Lisp. (Or, really, things you are justified in not wanting to expose such that they are testable.) (Sure, it could be the case that both the interpreter are wrong, and the atom function are wrong such that the test passes. That level of breakage isn't likely going to pass much of a significantly detailed test suite.)
  - wolfgang42 10y ago
    If I'd waited until the interpreter to work before I wrote tests, I'd never have gotten a working interpreter. For atom the implementation is obvious, but the more complex functions were riddled with bugs on the first implementation, and the lexical scope handling took about three complete rewrites before I got it right. If this had gone any further than the three days of off-hours time I spent, I certainly would have rewritten the test suite in Lisp; however it was primarily an academic exercise to really 'get' how Lisps worked and push my comfort level; writing all of my tests in the language I was trying to write would have pushed things a bit to far. In fact, I never got around to writing a parser, so even if I'd written them in Lisp it would have looked like this: var Symbols = require('../Symbols') var natives = require('../natives') var foo = Symbols.get_symbol_named('foo') module.exports = [ natives.tests, [natives.it, [natives.atom, [natives.quote, foo]], true], [natives.it, [natives.atom, [natives.quote, []]], true], ]
steveklabnik 10y ago
I did this with brainfuck.