5 ms·
We need a way to unit test documentation to see if the code snippets still compile and the things that are referenced still exist.
by MattGaiser 6y ago
We need a way to unit test documentation to see if the code snippets still compile and the things that are referenced still exist.
- twblalock 6y agoThis is why I prefer to link to actual code in the repo, in an examples directory, which gets compiled during every build.
- dgb23 6y agoThis is not only simple and comprehensive but also doesn't require any special support from the tooling or language. Love it!
- qayxc 6y agoUnfortunately that doesn't help if the documentation is written by a different team and in a different format. Getting out of sync happens very quickly if docs aren't generated from source and aren't an integral part of the release process.
- mijoharas 6y agoobligatory "Rust docs can do this"[0] (quite a nice feature tbh). [0] https://doc.rust-lang.org/rustdoc/documentation-tests.html https://doc.rust-lang.org/rustdoc/documentation-tests.html
- m45t3r 6y agoPython can too, and I think most other languages are inspired by it. https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html
- bussierem 6y agoThis actual exists in some places! Check out moduledocs and "doctests" for languages like Elixir. Was something really cool I liked while I have been learning Elixir.
- johnc1231 6y agoWe do this for our project (https://hail.is https://hail.is) and it's a game changer. Saves so much time and so many bug reports.
- pdimitar 6y agoCan you specify how are you doing it exactly? Very interested.
- forsaken 6y agoSphinx supports this for Python: https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html https://www.sphinx-doc.org/en/master/usage/extensions/doctes...
- m45t3r 6y agoThis is actually something from Python's stdlib: https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html Really underrated.
- TT3351 6y agoI seem to recall a talk given at one point that demonstrated a scheme that would reverse engineer functions out of whole cloth based only on docstrings and doctests, but of course I can't find it now.
- MattGaiser 6y agoMy job is mostly Python. Did not know this existed.
- qayxc 6y agoThat's definitely something that's still missing in technical documentation. The tooling still isn't there yet, and that's sad. Every change in API or behaviour should be automatically blocked by CI tooling in release builds if its documentation is missing or outdated. But alas, even in 2020 technical documentation is still just an afterthought for most companies - even big ones.
- crypteasy 6y agoMy company is currently trialing this with the Open API Spec. The workflow at a high level: 1) Make code changes to a specific microservice. (note: The Open API Spec also lives in code.) 2) CI/CD pipelines get triggered. 3) New microservice is built. 4) Call every REST endpoint defined in the microservice's Open API Spec and validate it using the example requests/responses. 5) If successful, regenerate the API documentation from the Open API Spec. This ensures that our documentation and services will stay inline with each other.
- kakwa_ 6y agoIt's far less extensive than what you presented, but on the current project we are working on at work, we are using https://github.com/swaggo/swag https://github.com/swaggo/swag It's somewhat specific to golang but so far it has been relatively good for us. The spec is directly next to the code as code comments, so it has far more chances to get updated when changes are made compared to an external documentation. Also, the payloads and responses are directly derived from the Golang structs, so, at least on that aspect, changes in the code are automatically reflected in the spec (apart for description and examples). We also put the interactive documentation directly in our API under /doc. It is a useful tool for developers when implementing a new url/handler or modifying an existing one. It also creates incentives to keep the documentation matching. Overall, we had very few mismatches between the spec and the actual implementation overall, despite not having deeply tested it (be it manually or automatically). Apart from one or two mismatches that were fixed quickly, I was able to take the spec, generate client libraries from it (swagger-codegen), and use them for quite extensive demos without issues. We are still early in the project (not in production yet), and there are definitely some aspects we need to improve (integration/automated tests to be sure doc and code are 100% matching) or to completely figure out (ex: how to handle several versions of the same API). But overall, using swaggo/swag has been a pleasant experience.
- FeepingCreature 6y agoIf you write a unittest in D underneath a function, it will automatically be included in the generated docs as an example. This is how all the examples in the standard library documentation are created, so you know for a fact that those work.
- mristin 6y agoUse contracts and propery-based testing. See Hypothesis library in Python (and other laguages). I wrote a library for contracts in Python (http://github.com/Parquery/icontract http://github.com/Parquery/icontract, see its readme for further references to other libraries).
- bradwood 6y agoComes standard with Rust