3 ms·

One of the main bullet points on the page is automated tests. How do you write automated tests for documentation? Somehow require that blocks of code have docu

One of the main bullet points on the page is automated tests. How do you write automated tests for documentation? Somehow require that blocks of code have documentation linked to them?

brobdingnag_pp 2y ago
Or just require file/function level comments. Requiring them to be helpful can be managed interpersonally, like someone was slacking off (they are)
Spasnof 2y ago
Yes there are certain libraries that can handle this. Essentially asserting that functions documented are valid / return the proper results. See https://docs.python.org/3/library/doctest.html#module-doctest https://docs.python.org/3/library/doctest.html#module-doctes... as an example.
tpoacher 2y ago
In matlab, where functions are "always" defined in their own file, there is a tool that checks if function documentations have all the right headers expected conventionally by matlab's documentation system (e.g. header, usage, examples, see-also links, etc). So this would be one example, I guess. But I too would be interested to hear other people's insights who subscribe to this Docs as Code model.
blowski 2y ago
> Somehow require that blocks of code have documentation linked to them The Symfony (PHP) framework now does this. Code and config examples in the docs have automated regression tests.
thesuperbigfrog 2y ago
>> How do you write automated tests for documentation? Somehow require that blocks of code have documentation linked to them? It could be tests to ensure documentation "builds" into all of the desired formats (e.g. web, pdf, ebooks, etc.) correctly. Some programming languages have the idea of "documentation tests". In Rust, tests that are part of the documentation will run as part of the documentation build: https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html https://doc.rust-lang.org/rustdoc/write-documentation/docume...
- Nzen 2y ago
  If we treat specifications written in gherkin syntax [0] as documentation, then the cucumber framework can match a line or stanza of gherkin to a test function [1]. I admit that, while I write instructions for how to test specific functionality in gherkin, our company would not countenance publishing a non-narrative description of the system's behavior to our client's employees. [0] https://www.manning.com/books/writing-great-specifications https://www.manning.com/books/writing-great-specifications Given a work order xx and xx isExpedite When an operator prints the jobcard Then expect a label in the footer that says Expedite [1] https://cucumber.io/docs/cucumber/step-definitions/?lang=java https://cucumber.io/docs/cucumber/step-definitions/?lang=jav...
0cf8612b2e1e 2y ago
I believe Rust and Python have the ability to run tests defined in docstrings.
darknavi 2y ago
Loads of things! - Making sure example snippets still compile - Checking if links are dead - Check for standardized/proper formatting Basically anything you'd want to enforce manually, try to enforce with CI.
fmbb 2y ago
Yeah that’s one way. And you can test that docs don’t link to code that does not exist. Here are some other Good Ideas in a blog post I stumbled upon the other week: https://azdavis.net/posts/test-repo/ https://azdavis.net/posts/test-repo/
spondylosaurus 2y ago
Linters like Vale are pretty common for docs(-as-code).
simonw 2y ago
I wrote about my way of doing that here: https://simonwillison.net/2018/Jul/28/documentation-unit-tests/ https://simonwillison.net/2018/Jul/28/documentation-unit-tes... Short version: have tests that use introspection (listing functions and classes in a module, iterating over JSON API endpoints in the codebase etc) and then run regular expressions against your documentation searching for relevant headings or other pre-determined structures.

One of the main bullet points on the page is automated tests. How do you write automated tests for documentation? Somehow require that blocks of code have documentation linked to them?

brobdingnag_pp 2y ago

Or just require file/function level comments. Requiring them to be helpful can be managed interpersonally, like someone was slacking off (they are)

Spasnof 2y ago

Yes there are certain libraries that can handle this. Essentially asserting that functions documented are valid / return the proper results. See https://docs.python.org/3/library/doctest.html#module-doctest https://docs.python.org/3/library/doctest.html#module-doctes... as an example.

tpoacher 2y ago

In matlab, where functions are "always" defined in their own file, there is a tool that checks if function documentations have all the right headers expected conventionally by matlab's documentation system (e.g. header, usage, examples, see-also links, etc). So this would be one example, I guess. But I too would be interested to hear other people's insights who subscribe to this Docs as Code model.

blowski 2y ago

> Somehow require that blocks of code have documentation linked to them The Symfony (PHP) framework now does this. Code and config examples in the docs have automated regression tests.

thesuperbigfrog 2y ago

>> How do you write automated tests for documentation? Somehow require that blocks of code have documentation linked to them? It could be tests to ensure documentation "builds" into all of the desired formats (e.g. web, pdf, ebooks, etc.) correctly. Some programming languages have the idea of "documentation tests". In Rust, tests that are part of the documentation will run as part of the documentation build: https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html https://doc.rust-lang.org/rustdoc/write-documentation/docume...

Nzen 2y ago
If we treat specifications written in gherkin syntax [0] as documentation, then the cucumber framework can match a line or stanza of gherkin to a test function [1]. I admit that, while I write instructions for how to test specific functionality in gherkin, our company would not countenance publishing a non-narrative description of the system's behavior to our client's employees. [0] https://www.manning.com/books/writing-great-specifications https://www.manning.com/books/writing-great-specifications Given a work order xx and xx isExpedite When an operator prints the jobcard Then expect a label in the footer that says Expedite [1] https://cucumber.io/docs/cucumber/step-definitions/?lang=java https://cucumber.io/docs/cucumber/step-definitions/?lang=jav...

0cf8612b2e1e 2y ago

I believe Rust and Python have the ability to run tests defined in docstrings.

darknavi 2y ago

Loads of things! - Making sure example snippets still compile - Checking if links are dead - Check for standardized/proper formatting Basically anything you'd want to enforce manually, try to enforce with CI.

fmbb 2y ago

Yeah that’s one way. And you can test that docs don’t link to code that does not exist. Here are some other Good Ideas in a blog post I stumbled upon the other week: https://azdavis.net/posts/test-repo/ https://azdavis.net/posts/test-repo/

spondylosaurus 2y ago

Linters like Vale are pretty common for docs(-as-code).

simonw 2y ago

I wrote about my way of doing that here: https://simonwillison.net/2018/Jul/28/documentation-unit-tests/ https://simonwillison.net/2018/Jul/28/documentation-unit-tes... Short version: have tests that use introspection (listing functions and classes in a module, iterating over JSON API endpoints in the codebase etc) and then run regular expressions against your documentation searching for relevant headings or other pre-determined structures.