11 ms·
Unit tests as documentation
- PaulHoule 2y agoActually every example in the documentation should be backed by a unit test, as in the example is transcluded from the unit test into the docs. Since you often want to show examples that don’t compile in docs you also should be able to write tests for compile errors.
- red2awn 2y agoBetter yet, use doc test as featured in Python [1] or Rust [2]. This makes sure your documentation examples are always up-to-date and runnable. [1]: https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html [2]: https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html https://doc.rust-lang.org/rustdoc/write-documentation/docume...
- eschneider 2y agoUnit tests are a _kind_ of documentation, but are rarely a complete solution to "documenting code". In general, the folks who don't do adequate code documentation are the same folks who don't do adequate unit tests. :/
- simonw 2y agoA trick I use a lot these days is to take the unit tests from an under-documented library, dump them into an LLM and ask it to write me detailed usage documentation. This works REALLY well. I've even occasionally done some of my own reviewing and editing of those docs and submitted them back to the project. Here's an example: https://github.com/pydantic/jiter/pull/143 https://github.com/pydantic/jiter/pull/143 - Claude transcript here: https://gist.github.com/simonw/264d487db1a18f8585c2ca0c68e50d1e https://gist.github.com/simonw/264d487db1a18f8585c2ca0c68e50...
- deleted 2y ago[deleted]
- tln 2y agoExtracting unit tests from your docs: great! Somehow extracting your docs from unit tests: might be ok! Pointing people at unit tests instead of writing docs: not even remotely ok. Is that really what this guy is advocating??
- bluefirebrand 2y ago> Pointing people at unit tests instead of writing docs: not even remotely ok. Couldn't agree more I'm trying to integrate with a team at work that is doing this, and I'm finding it impossible to get a full picture of what their service can do. I've brought it up with my boss, their boss, nothing happens And then the person writing the service is angry that everyone is asking him questions about it all the time. "Just go read the tests! You'll see what it does if you read the tests!" Incredibly frustrating to deal with when my questions are about the business rules for the service, not the functionality of the service
- alphanumeric0 2y agoThe code, tests and comments convey what actual business rules are implemented. While documentation is someone's non-precise natural language expression of what (to the best of their imperfect human capacity) expected the code to implement at the time of writing.
- bluefirebrand 2y agoYes, it is absolutely more valuable to know what the code "should" be doing than to know what the code is doing Otherwise there is no way to know what is expected behavior or just a mistake built into it by accident
- invaderzirp 2y agoEspecially on teams where deviance has been normalized, and broken things are just expected. I've been bitten both ways before: is this an obvious mistake? Or the lynchpin holding up the house of cards? Of course, if someone had just written some text explaining it, or perhaps a decent commit message instead of just "WIP", maybe we wouldn't have to do archaeology every single time.
- zahlman 2y agoThis isn't at all a new idea, but it's the first time I've seen it presented with this textbook AI style.
- teivah 2y agoIs there something problematic you think about the style? It's a genuine question. I wrote a book, and when I created my newsletter, I wanted to have a shift in terms of style because, on the Internet, people don't have time. You can't write a post the same way you write a book. So, I'm following some principles taken here and there. But happy to hear if you have some feedback about the style itself :)
- meindnoch 2y agoIs this "article" written by a LLM? "Tomorrow, you will receive your weekly recap on unit tests." Please, no.
- teivah 2y agoAs the post's author, no, it's not written by an LLM. The Coder Cafe is a daily newsletter for coders; we go over different topics from Monday to Thursday, and on Friday, there's a recap ;)
- rglover 2y agoJust write the docs. A simple template: - What is it? - What does it do? - Why does it do that? - What is the API? - What does it return? - What are some examples of proper, real world usage (that don't involve foo/bar but instead, real world inputs/outputs I'd likely see)?
- croes 2y agoWhy is a hard question. And what should be obvious or it’s still too complex.
- MathMonkeyMan 2y agoI was going to say that unit tests have the benefit of breaking when the truth changes. But then I realized that a lot of what makes a set of tests good documentation is comments, and those rot, maybe worse than dedicated documentation. Keeping documentation up to date is a hard problem that I haven't yet seen solved in my career.
- rglover 2y agoThe only fix for that is discipline. You can't automate away quality. The best people/teams understand that and make good docs a feature requirement, not an afterthought. My favorite example is Stripe. They've never skimped on docs and you can tell they've made it a core competency requirement for their team.
- Attummm 2y agoUnit tests as documentation have proven their worth over the years. For example this recent feature was added through unit test as documentation. https://github.com/Attumm/redis-dict/blob/main/extend_types_tests.py https://github.com/Attumm/redis-dict/blob/main/extend_types_...
- danjl 2y agoWhy just unit tests? Integration tests seem much more valuable as documentation of what the users will do in the app. Unit tests have limited benefits overall, and add a bunch of support time, slowing down development. If you have good (90%+) coverage just from integration tests, you are likely doing 90%+ coverage of the unit tests at the same time, without the extra effort or support burden. You can use the same reasoning to describe the benefits for understanding the code, you get a clear understanding of the important usage cases, plus you get the unit-level "documentation" for free.
- theLiminator 2y agoI think unit testing if you're testing in a blackbox manner. Whitebox unit testing tends to be very fragile and nowhere near as valuable as an integration test.
- smrtinsert 2y agoIf you look for edge cases in integration tests, you will have a combinatorial explosion of integration tests and you will be adding much more work. Unit tests save time, not lose it. I make this part of my filtering potential companies to work with now. I can't believe how often people avoid doing unit tests.
- danjl 2y agoThat's funny, since I wouldn't code at a place that mandates unit tests. Sure, they have a very minor role, in very specific cases, but I'd say 90% of projects can get 90% of the benefits by writing only integration tests with 90% coverage. If you'd like a more in-depth discussion of why integration testing is better: https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests
- evil-olive 2y agounit vs integration tests is not an either/or. you need both, and in appropriate coverage amounts. a common way to think about this is called the "test pyramid" - unit tests at the base, supporting integration tests that are farther up the pyramid. [0] roughly speaking, the X-axis of the pyramid is number of test cases, the Y-axis is number of dependencies / things that can cause a test to fail. as you travel up the Y-axis, you get more "lifelike" in your testing...but you also generally increase the time & complexity it takes to find the root-cause of a test failure. many times I've had to troubleshoot a failure in an integration test that is trying to test subsystem A, and it turns out the failure was caused by unrelated flakiness in subsystem B. it's good to find that flakiness...but it's also important to be able to push that testing "down the pyramid" and add a unit test of subsystem B to prevent the flakiness from reoccurring, and to point directly at the problem if it does. > Unit tests have limited benefits overall, and add a bunch of support time, slowing down development unit tests, _when done poorly_, have limited benefits, require additional maintenance, and slow down development. integration tests can also have limited benefits, require additional maintenance, and slow down development time, _when done poorly_. testing in general, _when done well_, increases development velocity and improves product quality in a way that completely justifies the maintenance burden of the additional code. 0: https://martinfowler.com/articles/practical-test-pyramid.html https://martinfowler.com/articles/practical-test-pyramid.htm...
- lucianbr 2y agoOne - unit tests explain nothing. They show what the output should be for a given input, but not why, or how you get there. I'm surprised by the nonchalant claim that "unit tests explain code". Am I missing something about the meaning of the english word "explain"? Two - so any input value outside of those in unit tests is undocumented / unspecified behavior? A documentation can contain an explanation in words, like what relation should hold between the inputs and outputs in all cases. Unit tests by their nature can only enumerate a finite number of cases. This seems like such an obviously not great idea...
- atoav 2y agoNot sure about this, but I like it the way it is done in the Rust ecosystem. In Rust, there are two types of comments. Regular ones (e.g. starting with //) and doc-comments (e.g. starting with ///). The latter will land in in the generated documentation when you run cargo doc. And now the cool thing: If you have example code in these doc comments, e.g. to explain how a feature of your library can be used, that script will automatically become part of the tests per default. That means you are unlikey to forget to update these examples when your code changes and you can use them as tests at the same time by asserting something at the end (which also communicates the outcome to the reader).
- lucianbr 2y agoYeah, combining unit tests and written docs in various ways seems fine. My reading of the article was that the tests are the only documentation. Maybe that was not the intent but just a bad interpretation on my part. Though some replies here seem to keep arguing for my interpretation, so it's not just me.
- the_af 2y agoCombining is what TFA suggests. They even go as far as closing the article with: > Note also that I’m not suggesting that unit tests should replace any form of documentation but rather that they should complement and enrich it.
- 2y ago
- kbbgl87 2y agoI believe that doctest is the best of both worlds, https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html
- Etheryte 2y agoThis is functionally not different from saying your code is your documentation. If it builds, then it's valid, etc. In other words, nonsense. Code, tests and documentation each serve a useful purpose and crucially they each serve a purpose that's distinct from the other ones, but supports them. Code is there to do the thing, tests are there to make sure the thing is done correctly, documentation is for other humans to understand what the thing is and how it's done.
- wubrr 2y agoCode as documentation is not nonsense at all. I do think high quality documentation should exist on it's own, but cleanly written and organized, well-commented code that is easy to read and understand is extremely valuable for many reasons. It IS a huge part of the documentation for the technical people that have to maintain the code and/or use it in advanced/specialized ways.
- Etheryte 2y agoYes, except this is not what people talk about when they say code is the documentation. What's meant in that context is no documentation and only code, with the idea that you can always read the code if you need to figure something out. Which, of course, is nonsense.
- wubrr 2y agoNah, that's your own (incorrect) interpretation, the first result of googling 'code as documentation' [0], starts off with: > Almost immediately I feel the need to rebut a common misunderstanding. Such a principle is not saying that code is the only documentation. [0] https://martinfowler.com/bliki/CodeAsDocumentation.html https://martinfowler.com/bliki/CodeAsDocumentation.html
- mihaigalos 2y agoIn TDD, u-tests are called "spec". Pretty much sums it up.
- lucianbr 2y agoSo again, any inputs outside of those exemplified in unit tests are unspecified behaviour? How would this work for mathematical operators for example?
- viraptor 2y agoA part of this lives in the spec name, and a part in the assumption that the Devs are not psychos. As in, if you test that sum(a,b) returns a sum of your numbers, the name/description of the test says so. And the second part means that it should hold for all numbers and the exceptions would be tested explicitly - nobody added "if a=5 & b=3 return 'foobar'" to it.
- samatman 2y agoInduction is a valid form of inference.
- advisedwang 2y agospec and documentation are things different though?
- gpmcadam 2y agoIn TDD spec is converted to behaviours that can be ran as automated tests so if the logic of the code changes, it breaks the spec and in turn the tests Whereas documentation can (inevitably) go stale with no feedback or build failures
- worik 2y agoUnit tests are valuable But they are also pricy I am interested in how people prevent unit tests becoming a maintenance burden over time. I have seen so many projects with legacy failing tests. Any proposal to invest time and money cleaning them up dies on the alter of investing limited resources in developing features that make money
- kubectl_h 2y agoI am starting to notice more and more unit tests in my org are written by AI -- I'm guessing usually after the implementation. I know this because I have, guiltily, done it and can tell when someone else has done it as well. I don't think anything can be done about this technically so it probably needs to be something discussed socially within the team.
- _thisdot 2y agoWhat is wrong with this? Tests involve a lot of hardcoding and mocking. I see this as an excellent use case for AI.
- JonChesterfield 2y agoGenerating tests that match the implementation doesn't tell you the implementation is doing the right thing. If it isn't, changing the implementation will break the tests, which in the best case wastes time and in the worst means the bugfix is abandoned. I deeply hate "regression tests" that turn red when the implementation changes, so you regenerate the tests to match the new implementation and maybe glance at the diff, but the diff is thousands of lines long so really it's not telling you anything other than "something changed".
- jaredcwhite 2y agoI very much disagree with this. Good code can be documentation, both in the way it's written and structured and obviously in the form of comments. Good tests simply verify what the author of the test believes the behavior of what is being tested should be. That's it. It's not documentation, it rarely "explains" anything, and any time someone eschews actually writing documentation in the form of good code hygiene and actual docs in favor of just writing tests causes the codebase to suffer.
- RangerScience 2y agoI disagree in a very specific and limited way: good tests show you how to use the code, which can be as simple as just “here’s some typical parameters for this function.” In more complex situations, good tests also show you the environmental set up - for example, all the various odd database records the code needs or expects. It’s not everything you’d want out of a doc, but it’s a chunk of it.
- bluefirebrand 2y ago> good tests show you how to use the code If you can't find examples of how to use the code in the code then why does the code even exist?
- esafak 2y agoCode is not always meant to be run by strangers, if it's internal to a team. The interface for customers should be documented.
- badmintonbaseba 2y agoThere are multiple problems with that: 1. The code uses internal interfaces, not meant to be used by users of the code. 2. The code might not use the high level public interfaces you are interested in. Those interfaces are meant to be used by users, and tested by tests. Having said that reading the code itself is often fruitful. Not for example usages, but to just learn how the thing is implemented.
- 2y ago
- benrutter 2y agoI've heard "tests are documentation" a lot, and even said it without thinkibg much myself. It sounds good, and I definitely like the idea of it, but I'm not sure it's true. Here's my thinking: - I've never tried to understand a code base by looking at the tlunit tests first. They often require more in depth understanding (due to things like monkeypatching) than just reading the code. I haven't seen anyone else attempt this either. - Good documentation is good as far as it aids understanding. This might be a side effect of tests, but I don't think it's their goal. A good test will catch breaks in behaviour, I'd never trade completeness for readability in tests, in docs it's the reverse. So I think maybe, unit tests are just tests? They can be part of your documentation, but calling them documentation in and of themselves I think is maybe just a category error?
- andoando 2y agoI definitely do look at tests to see examples of how library is meant to be used. But thats quite different
- sixthDot 2y ago> Good documentation is good as far as it aids understanding. This might be a side effect of tests, but I don't think it's their goal. A good test will catch breaks in behaviour, I'd never trade completeness for readability in tests, in docs it's the reverse. The D language standard library uses both. When you generate the documentation from the comments attached to a declaration, the following unittests (they are identified using a special markup, (that is just triple slashes...) are also included. Example once rendered [0], in the source you see the examples are actually unit tests [1]. [0]: https://dlang.org/phobos/std_algorithm_searching.html#.all https://dlang.org/phobos/std_algorithm_searching.html#.all [1]: https://github.com/dlang/phobos/blob/master/std/algorithm/searching.d#L112-L156 https://github.com/dlang/phobos/blob/master/std/algorithm/se...
- crabbone 2y agoYou are confused between "if" and "if and only if": "Tests are documentation" doesn't imply "documentation is tests". There are many useful tools that are documentation, none of them has to be exclusive.
- eesmith 2y agoI did not agree with most of the advice. Here are some examples: > Unit tests explain [expected] code behavior Unit tests rarely evaluate performance, so can't explain why something is O(n) vs O(n^2), or if it was supposed to be one or the other. And of course the unit tests might not cover the full range of behaviors. > Unit tests are always in sync with the code Until you find out that someone introduced a branch in the code, eg, for performance purposes (classic refactor step), and forgot to do coverage tests to ensure the unit tests exercised both branches. > Unit tests cover edge cases Note the True Scotsman fallacy there? 'Good unit tests should also cover these cases' means that if it didn't cover those cases, it wasn't good. I've seen many unit tests which didn't cover all of the edge cases. My favorite example is a Java program which turned something like "filename.txt" into "filename_1.txt", where the "_1" was a sequence number to make it unique, and ".txt" was required. Turns out, it accepted a user-defined filename from a web form, which could include a NUL character. "\x00.txt" put it in an infinite loop due to it's incorrect error handling of "", which is how the Java string got interpreted as a filename. > Descriptive test name With some test systems, like Python's unittest, you have both the test name and the docstring. The latter can be more descriptive. The former might be less descriptive, but easier to type or select. > Keep tests simple That should be 'Keep tests understandable'. Also, 'too many' doesn't contribute information as by definition it's beyond the point of being reasonable.
- latchkey 2y agoNobody is mentioning this. Tests are for change over time, they are not just for testing the output is the same. When you have a codebase sitting around rotting for years and you need to go back and refactor things to add a feature or change the behavior, how do you know you aren't breaking some dependent code down the line? What happens when you upgrade a 3rd party dependency, how do you know it isn't breaking your code? The javascript ecosystem is rife with this. You can't upgrade anything years later or you have to start over again. Tests are especially important when you've quit your company and someone else is stuck maintaining your code. The only way they can be sure to have all your ingrained knowledge is to have some sort of reliable way of knowing when things break. Tests are for preventing the next developer from cursing you under their breath.
- luisgvv 2y agoI used to work for a company that was in the fintech space and had several scenarios and rules for deciding whether a person could apply for a loan, the interest rate, prime, insurance etc. A lot of the code written was way back from the early 2000s in Visual Basic and migrated improved towards C#. I didn't knew a thing about how the business operated and the rationale behind the loans and the transactions. The parts of the application that had unit and behavior tests were easy to work on. Everyone dreaded touching the old pieces that didn't have tests.
- latchkey 2y agoI've got a typescript react component library that integrates 3 different projects together. It gets 36k downloads on npm every month. I started the project 5 years ago. When I originally wrote it, I knew that I would have to maintain it over time, so I wrote a ton of unit tests that mostly were just snapshots of the html output. I have two choices, running through my relatively complicated example app by hand and verifying things still work, or writing tests. I used this project to prove to myself that tests are indeed valuable. Over the years, I've made many releases. The 3 projects have been independently upgraded over time. The only way that I would have kept any sanity and been motivated to even work on this project (I no longer even use it myself!), is the fact that it takes almost zero effort to upgrade the dependencies, run the tests and build a release. If there are too many things to fix, I just wait for the community to eventually submit a PR. The best part is that if they break something, it is easy to spot in the snapshots (or test failures). I can almost accept PR's without having to even read them, just because the tests pass. That's pretty cool.
- danielovichdk 2y agoUnit tests is documentation of assertions. Hence it documents the result of how the code results to specification. It's of course not documentation in the sense of a manual to the detail of code it exercises, but it definitely helps if tests are proper crafted.
- timeon 2y ago"// The Coder Cafe" if it had "///" it could have test in docs: https://doc.rust-lang.org/stable/book/ch14-02-publishing-to-crates-io.html#documentation-comments-as-tests https://doc.rust-lang.org/stable/book/ch14-02-publishing-to-...
- janalsncm 2y agoCode and tests tell you what. They don’t tell you why. And if there’s a bug not covered in the tests, neither code nor tests can help you figure that out.
- bunderbunder 2y agoI share this ideal, but also have to gripe that "descriptive test name" is where this falls apart, every single time. Getting all your teammates to quit giving all their tests names like "testTheThing" is darn near impossible. It's socially painful to be the one constantly nagging people about names, but it really does take constant nagging to keep the quality high. As soon as the nagging stops, someone invariably starts cutting corners on the test names, and after that everyone who isn't a pedantic weenie about these things will start to follow suit. Which is honestly the sensible, well-adjusted decision. I'm the pedantic weenie on my team, and even I have to agree that I'd rather my team have a frustrating test suite than frustrating social dynamics. Personally - and this absolutely echoes the article's last point - I've been increasingly moving toward Donald Knuth's literate style of programming. It helps me organize my thoughts even better than TDD does, and it's earned me far more compliments about the readability of my code than a squeaky-clean test suite ever does. So much so that I'm beginning to hold hope that if you can build enough team mass around working that way it might even develop into a stable equilibrium point as people start to see how it really does make the job more enjoyable.
- tpoacher 2y agoObviously this is slightly implementation dependent but if your tests are accompanied by programmatic documentation (that is output together with the test), doesn't that eliminate the need for a descriptive test name in the first place? If anything, in this scenario, I wouldn't even bother printing the test names, and would just give them generated identifier names instead. Otherwise, isn't it a bit like expecting git hashes to be meaningful when there's a commit message right there?
- crabbone 2y agoThe article addresses this: there's no systematic enforcement of documentation currency. It may accidentally and unbeknownst to its authors become outdated. Anyways, as already mentioned earlier: unit tests are code and all quality criteria that apply to any other code apply to unit tests too. We expect identifiers used in code to help understand the code no matter if it's the name of a unit test or any other entity in our program. NB. To me this argument seems as bizarre as disputing washing your hands after using the bathroom. Why would anyone think that they should get a pass on code quality standards when writing unit tests? This just doesn't make sense...
- byyll 2y agoWrite your unit tests all you want but they are not documentation.
- tqi 2y agoWithout further documentation (beyond a descriptive test name), I fear that unit tests inevitably become a kind of Chesterton's Fence...
- ssalka 2y agoI forget where I heard this, but early in my career someone described unit tests to me as "a contract between you and your code." Which seems largely true – when I write a test, I'm saying "this is how a given function should behave, and that contract should hold true over time." If my future self wants the code to behave differently, so be it, but the contract needs to be amended so that the new code changes are also in agreement with it. Conversely, if you fail to write a unit test, there is no contract, and the code can freely diverge over time from what you think it ought to be doing.
- hannasm 2y agoI like the idea of this article but I would say that it's actually integration tests that are documentation. When learning a new codebase, and I'm looking for an example of how to use feature X I would look in the tests first or shortly after a web search. It seems to me like the second half of this article also undermines the main idea and goal of using unit tests in this way though. > Descriptive test name, Atomic, Keep tests simple, Keep tests independent A unit test that is good at documenting the system needs to be comprehensive, clear and in many cases filled with complexity that a unit test would ignore or hide. A test with a bunch of mocks, helpers, overrides and assumptions does not help anyone understand things like how to use feature X or the correct way to solve a problem with the software. There are merits to both kinds of tests in their time and place but good integration tests are really the best ones for documenting and learning.
- Terretta 2y agoExecutable documentation: Hitchstory is a type-safe StrictYAML python integration testing framework exploring some interesting ideas around this. https://hitchdev.com/hitchstory/ https://hitchdev.com/hitchstory/ Example: https://hitchdev.com/hitchstory/using/behavior/run-single-named-story/ https://hitchdev.com/hitchstory/using/behavior/run-single-na... Source: https://github.com/hitchdev/hitchstory/blob/master/hitch/story/success.story https://github.com/hitchdev/hitchstory/blob/master/hitch/sto... See also the explanation of self-rewriting tests: https://hitchdev.com/hitchstory/why/rewrite/ https://hitchdev.com/hitchstory/why/rewrite/
- hombre_fatal 2y agoI like how they couldn't be bothered to show examples of this ideal unit test code they think is just as good as documentation, just like people who can't be bothered to write docs. In reality, except for the most trivial projects or vigilant test writers, tests are too complicated to act as a stand in for docs. They are usually abstract in an effort to DRY things up such that you don't even get to see all the API in one place. I'd rather keep tests optimized for testing rather than nerfing them to be readable to end users.
- exabrial 2y agoIf you want to see how to do this right, go look at the CDI specification for Java. Every statement in the spec has a corresponding unit test, and it’s unbelievably incredible. Hats of to everyone that worked on this.
- 1980phipsi 2y agoD has documented unit tests. https://dlang.org/spec/unittest.html#documented-unittests https://dlang.org/spec/unittest.html#documented-unittests Nice when combined with CI since you’ll know if you accidentally break your examples.
- youainti 2y agoSomething I've been thinking of is that unit tests may now become useful as examples to be input into LLMs. If each function has a couple of tests with appropriate documentation, that may be useful as RAG input.
- lihaoyi 2y agoI make heavy use of this idea in many of my open source projects. I've tried a variety of approaches: * ScalaSql, where the reference docs (e.g. https://github.com/com-lihaoyi/scalasql/blob/main/docs/reference.md https://github.com/com-lihaoyi/scalasql/blob/main/docs/refer...) are generated by running unit tests (e.g. https://github.com/com-lihaoyi/scalasql/blob/53cbad77f7253f39688069850409cf4eb09c208c/scalasql/test/src/api/DbApiTests.scala#L17-L25 https://github.com/com-lihaoyi/scalasql/blob/53cbad77f7253f3...) * uPickle, where the documentation site (https://com-lihaoyi.github.io/upickle/#GettingStarted https://com-lihaoyi.github.io/upickle/#GettingStarted) is generated by the document-generator which has syntax to scrape (https://github.com/com-lihaoyi/upickle/blob/004ed7e17271635d9e80fcfe8c46e4a5a9f1423c/upickleReadme/Readme.scalatex#L84 https://github.com/com-lihaoyi/upickle/blob/004ed7e17271635d...) the unit tests without running them (https://github.com/com-lihaoyi/upickle/blob/main/upickle/test/src/upickle/example/ExampleTests.scala#L113-L124 https://github.com/com-lihaoyi/upickle/blob/main/upickle/tes...) * OS-Lib, where the documentation examples (e.g. https://github.com/com-lihaoyi/os-lib?tab=readme-ov-file#osreadchunks https://github.com/com-lihaoyi/os-lib?tab=readme-ov-file#osr...) are largely manually copy-pasted from the unit tests (e.g. https://github.com/com-lihaoyi/os-lib/blob/9e7efc36355103d71788c14671a7ded892c74ce2/os/test/src/ReadingWritingTests.scala#L38-L49 https://github.com/com-lihaoyi/os-lib/blob/9e7efc36355103d71...) into the readme.md/adoc It's a good idea overall to share unit tests and documentation, but there is a lot of subtlety around how it must be done. Unit tests and documentation have many conflicting requirements, e.g. * Unit tests prefer thoroughness to catch unintuitive edge cases whereas documentation prefers highlighting of key examples and allowing the reader to intuitively interpolate * Unit tests examples prefer DRY conciseness whereas documentation examples prefer self-contained-ness * Unit tests are targeted at codebase internal developers (i.e. experts) whereas documentation is often targeted at external users (i.e. non-experts) These conflicting requirements mean that "just read the unit tests" is a poor substitute for documentation. But there is a lot of overlap, so it is still worth sharing snippets between unit tests and examples. It just needs to be done carefully and with thought given handling the two sets of conflicting requirements
- valenterry 2y agoOkay, I'll go with it: statically defined types are also documentation! And the method names are equivalent to the test names. Of course, only if you don't wildly throw around exceptions or return null (without indicating it clearly in the type signature).
- matheusmoreira 2y agoI did something like this for my language. I built an interpreter and tested it by writing input programs and expected outputs. The test suite feeds the program through the interpreter and compares the actual and expected outputs. It's just like Ruby's executable RSpec. It's so nice. Every time I add a feature, I write an example program for it and that program automatically tests the language, its features, its semantics... With good comments they could conceivably be used to teach the language. I eventually added support for real unit tests to my test suite as well. I started testing parts of the runtime through them. Those turned out to be a lot messier than I'd hoped. Hopefully I'll be able to improve them over time by applying the principles outlined in the article.
- jillesvangurp 2y agoTests and documentation are things that I'm increasingly using LLMs for. Writing exhaustive tests and documentation is somewhat tedious work. You do it to help others. Or if you are smart enough to realize it, your future self. But LLMs take the tedium away. Example: I wrote a little fast api endpoint and realized that having some decent openapi documentation would be nice. Support for that is built in. So, copy paste into chat gpt, "add openapi documentation to these endpoints" paste it back. And then "write me an integration test that exercises these endpoints and tests all the exceptional responses". Simple stuff. But the point is that the generated documentation is pretty good and helpful. It's exhaustive. It documents all the essentials. I could sit down and do it manually for an hour. Or I could just generate it. I also generated a README for the same project with instructions on how to setup all the tools and do all the key things (run tests, run a dev server, build a docker container, etc. The Dockerfile was generated too. Dockerfiles are also great as documentation artifacts because it precisely defines how to build and run your software. LLMs are really good at documenting and summarizing things.
- nissarup 2y agoFor one narrow definition of documentation, yes. I'm pretty sure our end-users would get no value out of reading the unit tests in the code of the application they are using.
- noobermin 2y agoThis article is almost jumping the shark. It's fine to say it's in the constellation of things that help document code, but the end where you're starting to limit the tests you can do to highlight an ideal, for example, the "atomic" bullet, that's where you're now doing things for ideology, not really for their utility. A similar thing occurred with function purity, etc, etc. I don't know who this helps but if you're a young developer, always beware what you read on substack about how you should constrain yourself. Take them with a grain of salt.
- player1234 2y agoJust read the code.
- Jeff_Brown 2y agoFirst type signatures, then good function names, then tests. Between those three I rarely need comments -- except, perhaps ironically, in the tests, to explain what kind of corner condition I'm testing where.
- alfonsodev 2y agoI wonder if something like Jupyter notebook would be a cool format to implement this.
- najork 2y agoUnit tests to assert a spec are great, but they don't provide any context as to why the spec was defined as it was. This context is critical when making changes to the spec, so unit tests on their own aren't sufficient in my experience.
- projectileboy 2y agoThey express behaviors, yes, but they do not express intent, or any broader context. I’ve been on more than one project where well-meaning developers forbid comments and additional documentation because they felt that the only documentation should be the tests, and the result was always a complete lack of understanding as to why anything worked the way it did.
- HelloNurse 2y agoMeaningful test names are unimportant compared to sensible test organization and understandable test content: meaningful test fixtures, helper functions etc. to reduce repetitions; no dependencies on test execution order and other capital sins; straightforward and concise patterns like invoking the ordinary validation of object results instead of inspecting them ad-hoc or tables of simple inputs and expected outputs. Simple tests don't really need descriptive names because what they test should be obvious, while special tests need more than a good name to be documented enough: comments (e.g. why can crazy cases actually occur), links, extra effort to write them clearly, etc.
- commandersaki 2y agoI like examples in documentation and unit tests to be separated like how Go does it.
- tdiff 2y agoI doubt anyone who have ever seen complex test fixtures testing complex cases would agree it is better than free-form explanation of intended behaviour. Human language is just much more dense in terms of amount of conveyed information.
- nonameiguess 2y agoThis at least can work reasonably well if you're talking purely about library documentation meant to be consumed by other developers. It does nothing at all to provide documentation to software users who don't read code.