3 ms·

IMO a big portion of what a good test suite should be is documentation. Tests are examples of things that do and don't work, with some explanation of why. Examp

IMO a big portion of what a good test suite should be is documentation. Tests are examples of things that do and don't work, with some explanation of why. Examples in documentation can (for some doc frameworks & languages) be set up to run as tests, ensuring they stay current and actually work. Tests are machine-readable documentation.

deleted 3y ago
[deleted]
zilti 3y ago
Oh yes, and for languages where that is not directly possible, there are still ways - I implemented libraries for Chicken Scheme in org-mode with org-babel; the examples are simultaneously tests, runnable directly from within the org file, and live next to the code together with the docs. org-babel's tangle and org's export take care of the rest.
rqtwteye 3y ago
I subscribe this to a certain degree but there are limits. Some tests are just too complex to be used as documentation. Sample code is better in such cases.
ramijames 3y ago
This is ALWAYS TRUE. Software without good documentation pales in comparison to software with good documentation. https://www.ramijames.com/thoughts/docs-deserve-more-respect https://www.ramijames.com/thoughts/docs-deserve-more-respect
Just_Harry 3y ago
This is something I _really_ like about D. Unit-tests are built into the language, as is comment-based documentation—put those two together and you get unit-tests as documentation examples built into the language; all it takes is to put a documentation comment (which can be blank) right before a `unittest` block after a declaration. E.g. the examples for the D standard-library's `curry` function are just unit-tests: the docs: <https://dlang.org/phobos/std_functional.html#quickindex.curry https://dlang.org/phobos/std_functional.html#quickindex.curr...>; the code: <https://github.com/dlang/phobos/blob/42b8c65ccfd35c863f7cedfd20794a125615c49b/std/functional.d#L909-L1010 https://github.com/dlang/phobos/blob/42b8c65ccfd35c863f7cedf...>
- deleted 3y ago
  [deleted]
- hitchstory 3y ago
  I took the same "docs are tests and tests are docs" approach with integration testing when I created this library: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory I realized at some point that a test and a how-to guide can and probably should actually be the same thing - not just for doctests (https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html), but for every kind of test. It's not only 2x quicker to combine writing a test with writing docs, the test part and the docs part reinforce the quality of each other: * Tests are more easily understandable when you attach written context - the kind that is used when generating a readable how-to guide. * How to docs are better if they come attached to a guarantee that they're tested, not out of date and not missing crucial details. * Integration test driven development where how-to docs are created as a side effect is really nice.
abdullahkhalids 3y ago
You can put tests in python's class/method docs. They work wonderfully if you put enough examples to cover most inputs/outputs of the function.
- SAI_Peregrinus 3y ago
  Yep, also D and Rust have similar things, and likely other languages. I didn't feel it particularly useful to try to list them all, just to point out that tests & docs are two aspects of the same thing.
loup-vaillant 3y ago
That depends on what we're testing, and how thoroughly. Tests as documentation are amazing when the specs are in flux and we'd rather document stuff in one place instead of duplicating it between tests and user manual. But when the specs are fixed this is much less of a problem. Then there's how much you need to test. Good tests go way beyond readable examples, they usually involve generating tons of tests with more or less random data and running them in a loop (property based tests). I do this, and to be honest I wouldn't recommend end users read them.
- SAI_Peregrinus 3y ago
  Agreed, that's why I said it's a "big portion" and not "all" of what a test suite should be. Many tests are "how-to guides" in the Diátaxis documentation approach[1]. They're also useful as a part of tutorials. Anything over 25% is a pretty big portion of the total documentation. Automated property-based tests, fuzz tester cases, and similar things aren't documentation at all.
1-more 3y ago
> Examples in documentation can (for some doc frameworks & languages) be set up to run as tests, ensuring they stay current and actually work. As an example, in Elm (as in many languages) your publicly exposed functions in a package (a.k.a. library) are required to have documentation comments. This often includes a simple example because, well, it's easy to grasp. The build tool elm-verify-examples runs all of these examples and verifies that the output is what you say it is. It pretty cleverly uses the inline comment delimiter as the start of an arrow so that the rendered code listing in the example is still syntactically correct. https://github.com/stoeffel/elm-verify-examples https://github.com/stoeffel/elm-verify-examples

deleted 3y ago

[deleted]

zilti 3y ago

Oh yes, and for languages where that is not directly possible, there are still ways - I implemented libraries for Chicken Scheme in org-mode with org-babel; the examples are simultaneously tests, runnable directly from within the org file, and live next to the code together with the docs. org-babel's tangle and org's export take care of the rest.

rqtwteye 3y ago

I subscribe this to a certain degree but there are limits. Some tests are just too complex to be used as documentation. Sample code is better in such cases.

ramijames 3y ago

This is ALWAYS TRUE. Software without good documentation pales in comparison to software with good documentation. https://www.ramijames.com/thoughts/docs-deserve-more-respect https://www.ramijames.com/thoughts/docs-deserve-more-respect

Just_Harry 3y ago

This is something I _really_ like about D. Unit-tests are built into the language, as is comment-based documentation—put those two together and you get unit-tests as documentation examples built into the language; all it takes is to put a documentation comment (which can be blank) right before a `unittest` block after a declaration. E.g. the examples for the D standard-library's `curry` function are just unit-tests: the docs: <https://dlang.org/phobos/std_functional.html#quickindex.curry https://dlang.org/phobos/std_functional.html#quickindex.curr...>; the code: <https://github.com/dlang/phobos/blob/42b8c65ccfd35c863f7cedfd20794a125615c49b/std/functional.d#L909-L1010 https://github.com/dlang/phobos/blob/42b8c65ccfd35c863f7cedf...>

deleted 3y ago
[deleted]
hitchstory 3y ago
I took the same "docs are tests and tests are docs" approach with integration testing when I created this library: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory I realized at some point that a test and a how-to guide can and probably should actually be the same thing - not just for doctests (https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html), but for every kind of test. It's not only 2x quicker to combine writing a test with writing docs, the test part and the docs part reinforce the quality of each other: * Tests are more easily understandable when you attach written context - the kind that is used when generating a readable how-to guide. * How to docs are better if they come attached to a guarantee that they're tested, not out of date and not missing crucial details. * Integration test driven development where how-to docs are created as a side effect is really nice.

abdullahkhalids 3y ago

You can put tests in python's class/method docs. They work wonderfully if you put enough examples to cover most inputs/outputs of the function.

SAI_Peregrinus 3y ago
Yep, also D and Rust have similar things, and likely other languages. I didn't feel it particularly useful to try to list them all, just to point out that tests & docs are two aspects of the same thing.

loup-vaillant 3y ago

That depends on what we're testing, and how thoroughly. Tests as documentation are amazing when the specs are in flux and we'd rather document stuff in one place instead of duplicating it between tests and user manual. But when the specs are fixed this is much less of a problem. Then there's how much you need to test. Good tests go way beyond readable examples, they usually involve generating tons of tests with more or less random data and running them in a loop (property based tests). I do this, and to be honest I wouldn't recommend end users read them.

SAI_Peregrinus 3y ago
Agreed, that's why I said it's a "big portion" and not "all" of what a test suite should be. Many tests are "how-to guides" in the Diátaxis documentation approach[1]. They're also useful as a part of tutorials. Anything over 25% is a pretty big portion of the total documentation. Automated property-based tests, fuzz tester cases, and similar things aren't documentation at all.

1-more 3y ago

> Examples in documentation can (for some doc frameworks & languages) be set up to run as tests, ensuring they stay current and actually work. As an example, in Elm (as in many languages) your publicly exposed functions in a package (a.k.a. library) are required to have documentation comments. This often includes a simple example because, well, it's easy to grasp. The build tool elm-verify-examples runs all of these examples and verifies that the output is what you say it is. It pretty cleverly uses the inline comment delimiter as the start of an arrow so that the rendered code listing in the example is still syntactically correct. https://github.com/stoeffel/elm-verify-examples https://github.com/stoeffel/elm-verify-examples