8 ms·
What docs-as-code means
- cheschire 2y agoThis feels like a job ripe for startup disruption. LLM documentation generators tend to benefit from context, and nothing provides better context than a mostly functional code base. The best part is the code doesn’t even need to compile for the LLM to build the context needed.
- remoquete 2y agoAuthor here. No, I don't think it does. https://passo.uno/ai-anxiety-tech-writer-howto/ https://passo.uno/ai-anxiety-tech-writer-howto/
- chaffroomba 2y agoProblem I'm having as a developer with LLM documentation is their reliability, or rather lack of it. Every time there is an assertation I end up having to double-confirm it because they tend to be wrong as often as they're right. Reading imaginary hallucinated documentation is just about as useful as zero documentation. While I could keep doing this for the rest of my life, my employer doesen't really appreciate the extra expense. A technical writer is much, much cheaper than the dozens of developers trying to confirm the docs.
- exe34 2y ago> Reading imaginary hallucinated documentation is just about as useful as zero documentation. no, it's worse. it's closer to reading outdated documentation that outright lies and gives examples that don't work, and will cause you to waste hours/days learning things that aren't relevant to the api anymore.
- joshuanapoli 2y agoI think we will go the other way: A person will write the documentation and the AI will build the system that delivers the described product.
- chaffroomba 2y agoIsn't that what sw development today is? A description of a system and what we want it to do. With the advance of compilers, libraries, frameworks, linters, autocomplete systems and so on, we're already very close to describing the minimum amount of information the system needs in order to produce the correct result. To my knowledge actually physically writing the software has not been a bottleneck in a very, very long time.
- joshuanapoli 2y agoRight now, it takes skill and labor to move descriptions between representations for business goals, engineering (where we have frameworks and linters, etc.), and external/customer facing documentation. The customer is faced with an output from the design process. I think that we can turn that around now. Let customers edit part of the documentation, and let the AI adapt the system to their need.
- philipwhiuk 2y agohttps://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/ https://www.commitstrip.com/en/2016/08/25/a-very-comprehensi...
- euroderf 2y agoI would think that Requirements Engineering would be the ideal prep for AI code generation. A natural language counterpart to (e.g.) TLA+. Is this too damned obvious? Or am I missing something?
- smokel 2y agoThe main problem is that documentation should be written on why code is written the way it is, and why it exists in the first place. This context is typically not available in the code itself. In the best case, this is encoded in requirements and design documentation, but more often than not, the information remains only in the heads of customers, architects, and developers. The code itself merely describes how something is solved. Summarizing that in documentation can be useful at times, but it is not the full story. Especially for code that lives on for some time, the original design philosophy is often lost, and it is forced in horrible directions.
- spencerchubb 2y agoThis article mostly seems to be about documentation for users of the code, not documentation within the code.
- janice1999 2y agoCode doesn't capture architectural decision making, one of the most important being why other solutions were not used. LLMs would need to be nearly omniscient (understanding underlying hardware, customer needs, budgets even) to derive the reasoning behind those decisions afterwards from code alone.
- from-nibly 2y agoA source code repository is actually a terrible place to get that context. All of the things like decisions and how this relates to the customer are completely missing.
- Etheryte 2y agoThis completely misses what valuable code documentation is for. Anyone can read the code and figure out what it does, what as a documentation can be convenient, but it's not really all that useful, especially if it rots over time. Even if it's painful at times, the what is all there in front of you in the code. Valuable documentation explains the why, why was one approach chosen over another, why do we need to do this to begin with, why are the edge cases the way they are. This information is not present in the code and no tool can ever extract it since it isn't there.
- keybored 2y ago> LLM documentation generators I can’t wait for the LLM digression-generators to quit.
- throwaway984393 2y ago[dead]
- sshine 2y agoThe problem with "documentation" in commercial code is that nobody reads it. Any "documentation" that isn't embedded in code will never be opened again. Any "documentation" in the form of doc-strings and comments will, over time, lie. This is because competent programmers cannot agree on whether to even comment. When any large percentage of competent programmers do not comment, it is better to not rely on comments. Here's another pitch: Code-as-Docs
- llm_trw 2y agoHere's another: literate programming. If it's good enough for Knuth it's good enough for me.
- oersted 2y agoI tried to do it seriously a number of times. Perhaps I am doing it wrong, but my productivity drops like a stone. Writing down your thoughts as you go and maintaining them takes serious time. The romantic notion of writing (and reading) code like a book is appealing, but writing books is hard and arduous, it cannot be underestimated as a craft on its own right, and there is coding to be done. There is also the question of structuring the literate code. Telling a story of how you are building it or explaining how it works has a very different flow and order to how code is usually structured for good maintainability. Please correct me if I’m wrong, because I would love to dive into it, but I don’t think there has ever been any major piece of software developed following literate programming (at least as Knuth envisioned it). I also don’t think there is any significant book that contains a sizeable working program embedded in it throughout, that can be compiled and executed as-is. In practice Knuth was most concerned with embedding short code snippets in his papers and books. Having the whole thing be an actual compilable program was secondary, and it was mostly short academic proof-of-concept prototypes and algorithms. Don’t get me wrong, I love the concept, that’s why I have given it multiple serious tries over the years, likely I will again, and why I think I have some insight of what happens when you use it for “real-world” work.
- llm_trw 2y ago
- th0masfrancis 2y agoLove the theme of your blog
- remoquete 2y agoThanks! It's a tweaked version of https://github.com/mrmierzejewski/hugo-theme-console https://github.com/mrmierzejewski/hugo-theme-console
- codelion 2y agoWith LLMs it is now quite easy to generate docs as needed. In fact we built a service to do just that - https://docs.codes/ https://docs.codes/ Here is an example of how it is very useful especially for newer libraries.
- remoquete 2y agoYou need good source material, including docs, to have LLMs generate docs that are accurate, reliable, and safe. LLMs have interesting applications in areas like SDK and API docs, for sure, but can't replace an entire function.
- codelion 2y agoCorrect but this is a good starting point for code that is written after the cut off of language models training data as you cannot otherwise debate accurate code form then for the newer versions of the library.
- codelion 2y agoJust realised that I missed adding the link to the example - https://github.com/unclecode/crawl4ai/issues/126 https://github.com/unclecode/crawl4ai/issues/126
- jerrac 2y agoSo, I feel like "Docs-as-Code" has some context I'm missing, so I'm going to comment on docs in general. I think there multiple kinds of docs for software. * Comments explaining a specific section of code. * API docs describing functions/classes/etc. * Docs on how to use a library/class/etc. Usually including simple, isolated, examples. * Tutorials on how to create simplified applications using the developed tools. * Docs on how to deploy, configure, and maintain an application. * Docs on how to use an application. * Docs on how to troubleshoot an application. * Docs on how to integrate applications. * And likely others I'm missing. Personally, I've been seriously frustrated by how bad most of the open source (haven't done much with proprietary code) documentation is. Case in point is Drupal and Symfony. Trying to use api.drupal.org is not fun, and Symfony's docs always cover the basics, and then there's nothing on pulling everything together into something complicated. So you try to dig into the actual code, and end up finding multiple layers of uncommented abstractions. Yes, I can eventually figure out what is going on if I put the effort in, but that's a lot of time that could be save by a few lines of comments. I usually end up asking JetBrains AI about what I need, then use what it says after I fix the errors it makes... It's also very good at summarizing everything I'd find if I used a normal search. But that all only works if others have already asked and answered my questions. Some things I've been trying to do to improve my own code's documentation: * Unless the line is super obvious, even if I think it is obvious, I try to leave a comment. Yes, it seems pointless, but I have gone back to old code I remember being obvious without said comment enough times that I think it is worth it. * Avoiding "elegance" in favor of "explicitness". For example, I use full `if` statements instead of ternary operators even when ternary operators would look better. For whatever reason the syntax of ternary operators has never sunk in for me, and the explicitness of `if` is much easier to parse. I also use very descriptive function and variable names. Basically, if I have to think about what something means, I try to change it so I don't have to. * Split out functions into smaller functions as much as I can't. This means I can use descriptive function names. And I'm pretty sure it's just good practice. I also have been trying to figure out ways to keep higher level docs closer to my code. I have some ideas, but haven't tried them yet. Has anyone ever written something that detects changes to a method/function, and then when you save your file it pops up asking if related docs need updating? Maybe add comments to the method pointing to where related docs live, and then your IDE/tool uses that to know what docs need updating?
- NiklasBegley 2y agoI think this is a really good point in the post: > If you don’t review, check, and merge docs the same way your org reviews, checks, and merges code, you’re not doing docs-as-code — you’re doing docs-as-bore. While some WYSIWYG cloud-based docs platforms make it easier to make changes, that's not necessarily what you want. Docs are a critical component of how your users perceive your product - you want to have checks that it meets certain quality and accuracy standards. Just like your code. And if you're an engineering lead company, you probably want your docs updates to be coordinated with your product releases. Git is just the logical place to put your docs in that case. I've even created a company specifically to help with this workflow: https://www.doctave.com https://www.doctave.com Also, lots of comments here seem to be thinking of docstrings and other in-code documentation. I think that's really a different category that has a different set of goals and issues. This post is specifically about customer-facing documentation.
- robertlagrant 2y ago> And if you're an engineering lead company, Nit: engineering-led
- zelphirkalt 2y agoA small change in writing, a big change in meaning.
- NiklasBegley 2y agoOops. Good catch.
- tikkabhuna 2y agoYour product looks really interesting. I'm a big advocate for docs-as-code at my company. We use Confluence generally, but our team uses GitLab and Gatsby for our documentation/blogs as we really value the Merge Request workflow. Our biggest challenge is local development. A WYSIWYG is just so useful in that regard. Is that what Doctave Studio is trying to solve?
- 2y ago
- euroderf 2y agoHaving worked both as a developer and as a technical communicator (for software), I'm thinking that low friction for developers is paramount, and that therefore the way to get developers to put some effort into it is to have documents both (a) written in Markdown (or adoc or rST or typst) and (b) co-located with code under version control. Change the code, change the docs, no screwing around, BUT a quick & simple brain dump can suffice because of [see next paragraph]. To whit: I have yet to hear of a documentation system that provides fully bidirectional updates between such documents-as-code and edits made further downstream along the documentation production pipeline. That is to say, when a TC person or a reviewer makes edits to content, these changes should propagate back to the docs-as-code material. Then everyone benefits, including senior devs whose scarce time is optimised by having TC people expand and polish their hasty scribblings, and junior devs who have well-maintained documentation at-hand in-place. My 0.02€, YMMV. Maybe too niche.
- smokel 2y agoDocumentation near code is a good idea, but unfortunately it covers only part of the problem. There is a big portion of documentation that should be available to "other persons", such as architects or project managers, who may not want to visit the codebase. Another challenge is that for these people, diagrams are typically more useful than text. This still requires some manual effort which is difficult to achieve with Markdown.
- euroderf 2y agoAgreed on all points. Regarding para 2, I am assuming that there is something like a CMS in place that can pluck docfiles from version control and massage them and insert them into an outline/ToC. (And then propagate changes back to version control.) Regarding para 3, there's now many GUI tools for working with Mermaid et al. But are any of them properly integrated into documentation systems ?
- robertlagrant 2y agoArchitects should definitely be visiting the codebase. Why would project managers be updating docs? I'd have thought stuff like docs translations done by translators would be a better candidate for non-code editing.
- OhNoNotAgain_99 2y ago[dead]
- deleted 2y ago[deleted]
- ChrisMarshallNY 2y agoI was thinking this might have been a treatise on prompt engineering, or something like Rational Rose. I was actually pleasantly disappointed. I am glad to see documentation treated as a "first class" engineering Discipline. I would say the same goes to creative authors, such as graphic designers, or 3D modelers. I'm fairly big on good documentation[0], myself, and feel its absence, whenever I look at most codebases, these days. > Documentation is vital software infrastructure Definitely agree. I am feeling the decline in Apple's documentation (I'm an old-time Apple app developer, and remember Inside Macintosh). It's gotten absolutely awful, these days. I'm just starting on a SwiftUI app (I think I may have finally found a good application for it), and, boy howdy, is that documentation ... less than ideal. Fortunately, someone else thinks similarly, and did something about it[1]. I hate to rely on these types of things, because they often lose sync with the subject, but it is totally necessary. [0] https://littlegreenviper.com/leaving-a-legacy/ https://littlegreenviper.com/leaving-a-legacy/ [1] https://swiftui-lab.com https://swiftui-lab.com
- misja111 2y agoIn all my 26 years of working as a software dev, in the ~20 companies that I've worked or consulted for in various countries, never once did I see code documentation work out well. It wasn't that the will was not there: in almost every single of the companies the devs and management agreed that good documentation was important. And every now and then some heroic effort would be made to finally clean up Confluence or whatever system was used at the time. And for a little while that would work, then slowly documentation would become neglected here and there, and in the end it would become so bad that nobody would use it anymore. Except perhaps to showcase your commitment to best dev practices to your manager when it was time for your performance review. It doesn't matter if documentation has to be added in the code, or to some external system or both. Sooner or later it will be neglected, become out of sync and eventually become worse than no documentation at all. With the exception of external libraries or API's, software simply changes too fast. You'd need an army of technical writers to keep all documentation in sync. And unlike what the author writes, I have actually been working for a couple of companies that gave up and simply used the code as documentation, and surprisingly this worked out pretty well.
- remoquete 2y agoSad to read your n=20 was like that. There are also cases where documentation led companies to growth—one example is Stripe. Poking the finger at documentation is easy because it's visible and readily available. Would you say the bit rot phenomenon you described never happened to code, or processes, or UIs? Docs reflect organizations.
- solatic 2y agoDoing docs-as-code by putting documentation in the codebase is somewhat of an oversimplification. Usually, you want to use feature flags to decouple changes in Product from Engineering-driven deployments, so that new feature work can be rolled out slowly and its impact on KPIs measured, and so that engineering can de-risk the delivery of large changes by putting those changes behind feature flags with 0% rollout. The problem is, documentation is much closer to Product than Engineering, meaning, doing docs-as-code correctly means putting your documentation behind the same feature flags. So you should have both a public static site that shows the documentation for features whose flags have 100% rollout, plus your application itself should also show the documentation, so that you can show the documentation to users for feature flags enabled for those users. There aren't really any frameworks that have this docs-coupled-to-feature-flags pattern available off-the-shelf and most companies don't consider it high-enough priority to build in-house.
- ElevenLathe 2y agoThis is an interesting idea. Have you experienced any in-house versions of such a product? Anything that stuck out as good/annoying?
- solatic 2y agoNo, the closest is staging documentation changes until the features reach 100% rollout, then manually pressing a button to try to launch the docs at the same time. But when users don't find docs for new features (that are rolled out only to them), they naturally revert back to trying to talk directly to Engineering, which doesn't scale. An example of this mismatch is when QA isn't testing documentation changes: best-practice QA discovery-testing is for QA to have a production tenant, where the feature flag is enabled only for the QA tenant, but if QA doesn't have documentation for how the feature is supposed to work, then they end up having completely separate infrastructure that nobody else pays attention to, or else QA is talking directly to Product/Engineering, which somewhat defeats the purpose since that's something that customers can't do.
- florianmartens 2y agoThis article is awesome and resonates with my experience! I used to work for one of the big cloud vendors. We frequently had doc writers in our sync calls. Quite frankly, they never had a chance! Much of our documentation, our errors, status codes etc. was burried in the code and we (developers) were simply to busy to work effectively with technical writers. Don't get me wrong here. Where I say "busy" I don't mean their work wasn't important (it was!) but we had our own projects and jumping on a call to improve errors messages is not too exciting for most of us - and it's not the stuff that drives promotions. As a develper it's simply to easy to gatekeep code. Now follows a shameless plug: I'd be so excited to hear your opinion about a tool I'm building. It's called https://api-fiddle.com https://api-fiddle.com. Conceptually, it should help developers work API-first instead of code-first. It's great for devs because it makes the work faster and safer. BUT, an added benefit is that API docs are no longer burried in code and technical writers have a nice interface to contribute to them (and hold devs accountable!).