3 ms·

Doctest.js: a humane Javascript test framework

45 points by mikexstudios 14y ago

skrebbel 14y ago
I'm impressed. I really thought I'd seen all approaches to unit testing, but apparently I hadn't. This really appeals to me! Is there anyone here who's been using DocTest and has some experiences to share? I love how, as a side effect, this approach to unit testing encourages implementing a good toString / __str__ on each class. In those languages that have that, of course.
- bryanlarsen 14y ago
  jashkenas touches on it, but here's my rule of thumb for doctests: Test your documentation, but tests aren't documentation. In other words, it's great practice to ensure that code snippets inside your documentation execute correctly. Your primary benefit is that your documentation is correct, complete and up to date. Any testing benefit is purely secondary; you need a complete test suite regardless. You can use a doctest framework to create this test suite for convenience, but recognize that your doctests are going to be either illustrative or complete, but cannot be both. Examples: Here's a tutorial written in git comments so that each step is fully executable: http://cookbook.hobocentral.net/tutorials/agility http://cookbook.hobocentral.net/tutorials/agility Here's a manual page written in rubydoctest: http://cookbook.hobocentral.net/manual/scopes http://cookbook.hobocentral.net/manual/scopes
bergie 14y ago
If you could combine this with a Literate Programming approach like Literate CoffeeScript[1], then the examples in your API documentation could also become the executable tests. 1: https://github.com/jashkenas/coffee-script/issues/1786 https://github.com/jashkenas/coffee-script/issues/1786
- bergie 14y ago
  Thinking this a bit further, it would be possible to write a simple xdoctest command that would extract all code blocks from a JavaScript document and save them in the format doctest.js supports. Then you could do stuff like: ## Capitalizing text The `capitalize` function receives a string and returns it in capitalized format. Example: ```javascript print(capitalize('some words')); // => Some Words ``` ...and generate both your API docs and the runnable tests from the same file. This way you'd only need to keep one set of examples/tests up to date.
  - ianb 14y ago
    With doctest.js potentially the HTML docs/examples are runnable tests, and you wouldn't have to generate both (or the difference would just be whether you included doctest.js) If you could just do: ```javascript;commenttest print(capitalize('some words')); // => Some Words ``` and if javascript;commenttest got translated into <pre class="commenttest">, then you'd be good to go right now, without any special support from doctest. I believe if the element is syntax highlighted that doctest will just ignore (and wipe) that highlighting, but it depends on the details, I've never personally tried it. If you could add multiple classes you could use javascript;commenttest.hidden for setup code that you want to run as part of the tests, but doesn't add to the narrative. (There's a class that makes setup blocks compact, but I suppose I could extend it to do hidden blocks too: http://doctestjs.org/reference.html#compact http://doctestjs.org/reference.html#compact) I don't think there's an advantage for doctest to parse the Markdown directly, as opposed to parsing what is rendered. And extracting just the code also seems unnecessary, as it is helpful to see the context of any failures.
- jashkenas 14y ago
  We've actually talked about adding this to CoffeeScript in the past, as a regular language feature. (You've probably already read the ticket.) The reason we don't is because good tests are very rarely the same thing as good examples. Good examples often elide bits of code that aren't pertinent to the API function being demonstrated ... and good tests often involve the edge cases that are poorly suited to learning examples. While testing your examples is all well and good, it's also limiting -- you can no longer write: makeRequest({ url: 'http://example.com/endpoint', data: ..., }) ... unless, come to think of it, you had some sort of Yada-yada operator that served as a silent no-op.
  - bergie 14y ago
    I'm waiting eagerly for the Literate CoffeeScript support to land. My main concern with it is that it forces you to write the document in the same order you want the code to be extracted, which may not be the best order for explaining things. That is why classic Literate Programming tools like noweb[1] allow you to name the chunks of code and then arrange them into the generated files as you wish. You can see an example of this in action when I'm assembling noweb.php[2]. Also, agreed on the point that not every piece of example code is suitable for running as a test, so you'd optimally want some sort of annotation for that. Github-flavored Markdown[3] has the code block fencing syntax that would allow metadata like this. Something like: ```coffeescript # Just a regular, non-testable example ``` ```coffeescript;test # Example that is also a test ``` Actually, this same approach could be used for working with named chunks like noweb does. 1: http://en.wikipedia.org/wiki/Noweb http://en.wikipedia.org/wiki/Noweb 2: https://github.com/bergie/noweb.php/blob/master/README.txt#L324 https://github.com/bergie/noweb.php/blob/master/README.txt#L... 3: http://github.github.com/github-flavored-markdown/ http://github.github.com/github-flavored-markdown/
    - jashkenas 14y ago
      > My main concern with it is that it forces you to write the document > in the same order you want the code to be extracted, which may not > be the best order for explaining things. ... Yep -- this is probably the main complaint raised when talking about tools like Docco or Markdown-as-source-code as "literate programming". But, it's a concern that I believe is entirely outdated. Modern dynamic languages make it easy to sequence your code as you like. The methods in a class may be listed in any logical order, helper functions can be listed in an appendix after the functions that make use of them, and so on. I find it hard to imagine an example where changing the order of the codebase would make the prose version more readable -- and wouldn't also make the code version more readable as well. I think that, unfortunately, the tangle/weave paradigm is a large part of why literate programming is barely used -- it introduces an entire level of build-time complexity that doesn't have any direct effect or benefit on your code. Here's hoping that the simplicity of having a file that is at once both executable code and publishable markdown is something that feels more usable to more programmers.
j4mie 14y ago
This blog post is essential reading for Python developers who see doctest.js and think "ew, doctest": http://blog.ianbicking.org/2012/10/02/why-doctest-js-is-better-than-pythons-doctest/ http://blog.ianbicking.org/2012/10/02/why-doctest-js-is-bett...
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]

45 points by mikexstudios 14y ago

I'm impressed. I really thought I'd seen all approaches to unit testing, but apparently I hadn't. This really appeals to me! Is there anyone here who's been using DocTest and has some experiences to share? I love how, as a side effect, this approach to unit testing encourages implementing a good toString / __str__ on each class. In those languages that have that, of course.

bryanlarsen 14y ago
jashkenas touches on it, but here's my rule of thumb for doctests: Test your documentation, but tests aren't documentation. In other words, it's great practice to ensure that code snippets inside your documentation execute correctly. Your primary benefit is that your documentation is correct, complete and up to date. Any testing benefit is purely secondary; you need a complete test suite regardless. You can use a doctest framework to create this test suite for convenience, but recognize that your doctests are going to be either illustrative or complete, but cannot be both. Examples: Here's a tutorial written in git comments so that each step is fully executable: http://cookbook.hobocentral.net/tutorials/agility http://cookbook.hobocentral.net/tutorials/agility Here's a manual page written in rubydoctest: http://cookbook.hobocentral.net/manual/scopes http://cookbook.hobocentral.net/manual/scopes

bergie 14y ago

If you could combine this with a Literate Programming approach like Literate CoffeeScript[1], then the examples in your API documentation could also become the executable tests. 1: https://github.com/jashkenas/coffee-script/issues/1786 https://github.com/jashkenas/coffee-script/issues/1786

bergie 14y ago
Thinking this a bit further, it would be possible to write a simple xdoctest command that would extract all code blocks from a JavaScript document and save them in the format doctest.js supports. Then you could do stuff like: ## Capitalizing text The `capitalize` function receives a string and returns it in capitalized format. Example: ```javascript print(capitalize('some words')); // => Some Words ``` ...and generate both your API docs and the runnable tests from the same file. This way you'd only need to keep one set of examples/tests up to date.
- ianb 14y ago
  With doctest.js potentially the HTML docs/examples are runnable tests, and you wouldn't have to generate both (or the difference would just be whether you included doctest.js) If you could just do: ```javascript;commenttest print(capitalize('some words')); // => Some Words ``` and if javascript;commenttest got translated into <pre class="commenttest">, then you'd be good to go right now, without any special support from doctest. I believe if the element is syntax highlighted that doctest will just ignore (and wipe) that highlighting, but it depends on the details, I've never personally tried it. If you could add multiple classes you could use javascript;commenttest.hidden for setup code that you want to run as part of the tests, but doesn't add to the narrative. (There's a class that makes setup blocks compact, but I suppose I could extend it to do hidden blocks too: http://doctestjs.org/reference.html#compact http://doctestjs.org/reference.html#compact) I don't think there's an advantage for doctest to parse the Markdown directly, as opposed to parsing what is rendered. And extracting just the code also seems unnecessary, as it is helpful to see the context of any failures.
jashkenas 14y ago
We've actually talked about adding this to CoffeeScript in the past, as a regular language feature. (You've probably already read the ticket.) The reason we don't is because good tests are very rarely the same thing as good examples. Good examples often elide bits of code that aren't pertinent to the API function being demonstrated ... and good tests often involve the edge cases that are poorly suited to learning examples. While testing your examples is all well and good, it's also limiting -- you can no longer write: makeRequest({ url: 'http://example.com/endpoint', data: ..., }) ... unless, come to think of it, you had some sort of Yada-yada operator that served as a silent no-op.
- bergie 14y ago
  I'm waiting eagerly for the Literate CoffeeScript support to land. My main concern with it is that it forces you to write the document in the same order you want the code to be extracted, which may not be the best order for explaining things. That is why classic Literate Programming tools like noweb[1] allow you to name the chunks of code and then arrange them into the generated files as you wish. You can see an example of this in action when I'm assembling noweb.php[2]. Also, agreed on the point that not every piece of example code is suitable for running as a test, so you'd optimally want some sort of annotation for that. Github-flavored Markdown[3] has the code block fencing syntax that would allow metadata like this. Something like: ```coffeescript # Just a regular, non-testable example ``` ```coffeescript;test # Example that is also a test ``` Actually, this same approach could be used for working with named chunks like noweb does. 1: http://en.wikipedia.org/wiki/Noweb http://en.wikipedia.org/wiki/Noweb 2: https://github.com/bergie/noweb.php/blob/master/README.txt#L324 https://github.com/bergie/noweb.php/blob/master/README.txt#L... 3: http://github.github.com/github-flavored-markdown/ http://github.github.com/github-flavored-markdown/
  - jashkenas 14y ago
    > My main concern with it is that it forces you to write the document > in the same order you want the code to be extracted, which may not > be the best order for explaining things. ... Yep -- this is probably the main complaint raised when talking about tools like Docco or Markdown-as-source-code as "literate programming". But, it's a concern that I believe is entirely outdated. Modern dynamic languages make it easy to sequence your code as you like. The methods in a class may be listed in any logical order, helper functions can be listed in an appendix after the functions that make use of them, and so on. I find it hard to imagine an example where changing the order of the codebase would make the prose version more readable -- and wouldn't also make the code version more readable as well. I think that, unfortunately, the tangle/weave paradigm is a large part of why literate programming is barely used -- it introduces an entire level of build-time complexity that doesn't have any direct effect or benefit on your code. Here's hoping that the simplicity of having a file that is at once both executable code and publishable markdown is something that feels more usable to more programmers.

j4mie 14y ago

This blog post is essential reading for Python developers who see doctest.js and think "ew, doctest": http://blog.ianbicking.org/2012/10/02/why-doctest-js-is-better-than-pythons-doctest/ http://blog.ianbicking.org/2012/10/02/why-doctest-js-is-bett...

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]