5 ms·
A subset of this idea is a hill I am willing to die on: the documentation for a codebase should live in the same repository as the codebase itself. I'm talking
by simonw 2y ago
A subset of this idea is a hill I am willing to die on: the documentation for a codebase should live in the same repository as the codebase itself.
I'm talking about API documentation here - for both code-level APIs (how to use these functions and classes) as well as HTTP/JSON/GRPC/etc APIs that the codebase exposes to others.
If you keep the documentation in the same repo as the code you get so many benefits for free:
1. Automatic revision control. If you need to see documentation for a previous version it's right there in the repo history, visible under the release tag.
2. Documentation as part of code review: if a PR updates code but forgets to update the accompanying documentation you can catch that at review time.
3. You can run documentation unit tests - automated tests that check that the documentation at least mentions specific pieces of the code (discovered via introspection). I wrote about that a few years ago and it's been working great for me: https://simonwillison.net/2018/Jul/28/documentation-unit-tests/ https://simonwillison.net/2018/Jul/28/documentation-unit-tes...
4. Most important: your documentation can earn trust. Most documentation is out of date and everyone knows that, which means people default to not trusting documentation. If anyone who looks at the commit log can see that the documentation is being actively maintained alongside the code it documents they are far more likely to learn to trust it.
The exception to this rule for me is user-facing documentation describing how end users should use the features provided by the software. I'd ideally love to keep this in the repo too, but there are rational reasons not to - it might be maintained by the customer support team who may want to work in more of a CMS environment, for example.
- cedws 2y agoTrue. Confluence or whatever corp shitware is where technical documentation goes to die.
- hitchstory 2y agoIt works for user facing documentation too. There are actually pretty good reasons for this - e.g. you can use the test to autogenerate up to date screenshots with playwright to put in the documentation. I'm pretty convinced that there should be a single source of truth for specifications, tests and documentation but I think the industry will take a while to catch up to this idea. I built a testing library centered around this (same as my username) but it's hard to get people to stop writing unit tests :)
- simonw 2y agoI actually built my own Playwright screenshotting software with this idea in mind too: https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ - I wrote about using that for my project documentation here: https://simonwillison.net/2022/Oct/14/automating-screenshots/ https://simonwillison.net/2022/Oct/14/automating-screenshots... Really it comes down to the team you are working with. If you have user-facing documentation authors who are happy with Markdown and Git you can probably get this to work.
- hitchstory 2y agoThat's very cool. I think screenshotting needs to be integrated into the tests though - if a scenario involves a wizard or something, the latter screenshots will be dependent upon the actions in earlier steps.
- simonw 2y agoThe thing I really want is automated short video demos, but I've not found a good path to those yet.
- hitchstory 2y agoThe skeleton test-generating-docs example I built to exhibit the framework actually does this. Here's an example: https://github.com/hitchdev/hitchstory/blob/master/examples/website/hitch/docs/correct-my-spelling.md https://github.com/hitchdev/hitchstory/blob/master/examples/... It records a video while running the test and then at the end runs it through FFMPEG to make a smaller, slowed down GIF that can be embedded in the autogenerated docs. It's quite rudimentary though. I've been meaning to try making something more sophisticated and even potentially do additional automated video editing to inject text from the steps or something.
- xorcist 2y agoLove your blog, but in this case I want to take a more nuanced, if not opposite, stance: There are many things closely related to code, that shouldn't necessarily live in the same repository. First, we need a common understanding of what should live together in a repository. This is much like the discussion about mono vs. multi-repo. A good rule of thumb is that if it is branched together, it lives together. Effective documentation is not only a strict API reference, and not something that can be generated from docstrings alone. It offers a high level overview to understand the problem being solved, the architecture of the software, and a general roadmap of how it is developed. Effective documentation should cover both backwards and forwards revisions and how those migrations should be handled. But this is also true on a reference level. Reading the documentation of a specific function I want to know if something relevant happens to this function in the next revision. There is nothing worse than checking out documentation for current production revision 34.5 and follow best practice there only to discover I should have checked out revision 34.6 instead because best practice changes there. Specific revisions should be documented, but documentation should not be limited to a specific revision. There is a scale of how closely other artifacts follow code revisions: Tests is mostly branched with code, and should probably live together. Documentation can sometimes be branched with code, some should and some shouldn't live together with code. Deployment code and configuration management must be able to deploy old and new code from the same code base, and is even less likely to benefit from living with it. Then there's application state and test data which is something else entirely.
- simonw 2y agoIf the deployment code needs to be able to ship different versions, I would keep that deployment code in a separate repository - with its documentation bundled there. The other form of documentation that I am passionate about is documentation that lives in issues, and then linked to from commit messages. The great thing about issues and issue comments is that they have a clear timestamp attached to them, and there is no expectation that they will be kept up-to-date in the future. This makes them the ideal place to keep documentation about how the code evolved overtime, and the design decisions that were made along the way.
- 2y ago
- godelski 2y ago> a hill I am willing to die on: the documentation for a codebase should live in the same repository as the codebase itself. I'm a big fan of this and treating documentation like a first class citizen. There's also another benefit I think should be explicitly mentioned. It makes debugging, onboarding, and solving things much faster. We all know and have experienced the joke where you question who wrote this pile of garbage to find out that it was you all along. But at the core of this joke is the fact that we can't even remember what we ourselves did. So while things make sense at the time and might even seem obvious, that does not mean it'll continue to make sense nor that it'll be obvious to others. Especially to people who are onboarding into a new codebase. Yes, documenting while you code takes "longer." But it only takes longer in the short run. It is much faster in the long run. The question you have to ask is if you're doing a sprint or a marathon. But then again there's very ill advised and self-contradictory advice on well known sites[0] and some companies perform back to back sprints. But I don't think people realize we're the ones creating our own messes. As anyone with anxiety will tell you, when you are rushing around it becomes easy to overlook small mistakes that will compound and only accumulate to make your anxiety worse than it was had you just slowed down in the first place. Creating a negative feedback loop where you only get more stressed to end up creating more problems than you solve. There's times to move fast and break things, but if you don't also dedicate time to clean up your house will be filled with garbage and inhabited by a Lovecraftian entities made of spaghetti and duct tape. [0] https://www.codecademy.com/resources/blog/what-is-a-sprint/ https://www.codecademy.com/resources/blog/what-is-a-sprint/
- strken 2y agoI would be very happy if as much developer documentation as possible was actually executed as part of the code. For example, a diagram of how different services interact can go out of date. It would be better if there was a config file describing which services can be called, and this config file was used to generate firewall rules (for the case where dependencies on services are missing) and alert rules (for the case where unnecessary dependencies are never removed). Another example might be OpenAPI docs that you use to validate requests and responses. I think that when you enforce a common source of truth behind both your docs and the functionality of your system, those docs can never become outdated. If you just shove docs into git without using them for anything they can easily rot away.
- clumsysmurf 2y agoI have often wondered why Android's javadoc is so awful ... and thought, maybe precisely because its its embedded in such a large codebase it doesn't get updated for risk vs perceived benefit reasons (becuase of proximity of javadoc to code). Of course, it could be cultural or other things ... Perhaps the tooling sees changed sources, false positives for code changes, and there is a desire to eliminate this to help downstream consumers etc?
- klysm 2y agoI’ll happily die on that hill with you
- gofreddygo 2y ago> the documentation for a codebase should live in the same repository as the codebase itself This! 100%. Emphasis - codebase documentation. Not user guides. After doing this a couple times, it's a no brainer. The benefits are significant, the effort minimal. Just add a docs dir at the project root and go to town. The docs dir has some very interesting stuff - how to run parts of the api locally, tricks to make auth bearable for local development, commands that get new team members going at hyperspeed, what parts talk to what parts, which files are important for what flows, why some refactoring was attempted but abandoned, high level limitations and benchmarks, history on how some monstrosity came to be with some jokes sprinkled about. everything just one cmd+shift+f away.
- tivert 2y ago5. The documentation won't get lost in a botched wiki migration or something like that. The documentation in the repo should not be restricted to relatively low-level stuff about APIs, it should also include design documents and cover the higher level concepts the developers use to make sense of the app and its APIs. I can't tell you how many times I've seen these concepts lost after the original developers move on, and then get violated in ways that make the app much harder to comprehend.
- ranger_danger 2y agoThe "documentation" for Lemmy consists merely of an auto-generated JavaScript library API dump with no real explanation for what most of the endpoints do (and are often named ambiguously) or how the general flow of things is supposed to work, or even how to do common things like find a user's comments or posts (would you have guessed they're both under "/user"? Because they sure don't tell you that). Especially if you don't know Javascript you're going to have a bad time trying to use that API. And the devs defend it if you tell them this, claiming "it defines everything perfectly, it's so easy." One time my company purchased a $5k commercial license for x264 and were met with "the code is the documentation." That set us back literal weeks.