5 ms·

Seems to me documentation can also be superstition if it's out of date. I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy

• iterateoften 2y ago
  Maybe LLMs make docs irrelevant if they are just generating static pages or doc strings. Why spend the time generating something static when everyone has their own LLMs they can just ask directly? Anything you can generate as docs someone can generate themselves anyways.
  • danihh 2y ago
    We actually ran this train of thought with our startup right now. The problem? Not everybody knows what questions to ask. For non-power users it’s usually better to present digestible information and create docs that answer questions they didn’t even know they had.
  • blharr 2y ago
    I mean, LLMs can be regenerated as needed. With every major release, you might regenerate the documentation. Then, you don't need to have everyone taking time to ask the same questions, reducing the amount of times you need to call the LLM. And thereby you can use a more expensive LLM, like o1 to generate the documentation where you wouldn't want to use it normally
  • pona-a 2y ago
    The thing is, good documentation captures the difference between what you expect and what something is. LLMs, as they exist now, are mass-producers of cliché. You can sense it even in their prose or those inverted puzzle evaluations [0] , they are simply statistical models picking the most likely options without enough depth to subvert the average. When you ask them to describe a piece of code, you'll only get the rephrasing of what the code already says. - [0] https://github.com/cpldcpu/MisguidedAttention https://github.com/cpldcpu/MisguidedAttention
• AshamedCaptain 2y ago
  Why would you bring ML up? The entire example problem brought up by TFA is because the ink-to-shape transform likely uses some ML behind the scenes (plain interpolation likely) , and as consequence, it is an _indocumentable_ blackbox even if you wanted to document it. Superstition is the only way you can make it work. In fact, "superstition" very neatly defines most prompt engineering for LLMs.
  • VectorLock 2y ago
    >In fact, "superstition" very neatly defines most prompt engineering for LLMs. Its weirdly fascinating how this is developing. Like some people boldly claiming that if you offer to tip your LLM it does better. And if you say things like "Think hard and check your work" it gives more accurate results.
• ChrisMarshallNY 2y ago
  I tend to use in-code documentation[0]. Works pretty well. Could still [theoretically] go out of date, but that hasn't happened, so far. The one thing that does happen, though, is copy/pasta errors. I'll copy a bunch of code, including the headerdocs, and paste, but forget to adjust the docs. [0] https://littlegreenviper.com/leaving-a-legacy/ https://littlegreenviper.com/leaving-a-legacy/
  • pasc1878 2y ago
    But it covers only how to use a function if you know what the system is meant to do or know which function to call in the first place. You need some overalls documentation to explain the concepts and how to put things together. When the user understands that you can use in-code documentation but even then only if you need to use one function at a time - there needs an explanation of how to put things together.
    • ChrisMarshallNY 2y ago
      That's what I use the README for. With Apple's docc, you can drop READMEs throughout the system, and they show up in Xcode's doc browser (and can be referenced from the code).
    • eddd-ddde 2y ago
      Integration tests? They show how the system works as a whole, and they don't go out of date.
      • ChrisMarshallNY 2y ago
        They are, indeed, useful. I usually create full-fat test harnesses[0] for my stuff, as that gives a totally realistic implementation example. But I think having a well-written README is more likely to be useful. API/implementation documentation (whether docs or tests), are much more likely to be read, than end-user docs. [0] https://littlegreenviper.com/testing-harness-vs-unit/ https://littlegreenviper.com/testing-harness-vs-unit/
• martin-t 2y ago
  I inherited 100k LoC "documented" by an LLM. LLMs can put into words what a programmer can infer from struct definitions, function signatures, names and even taking a cursory look at a function body. LLMs cannot retroactively find out the _why_ of design choices. If you don't document the reasons behind decisions while making them, they'll be lost, which is exactly what leads to superstition. As a result, changing code is error prone because you never know when code is written in a particular way for a good (and still valid) reason or just because.
  • ValentinA23 2y ago
    Related: Show HN: Auto Wiki – Turn your codebase into a Wiki https://news.ycombinator.com/item?id=38915999 https://news.ycombinator.com/item?id=38915999
  • VectorLock 2y ago
    Someone at work recently introduced a thing that would create git commit messages from code, completely negating the necessity of the programmer to provide accurate not-contained-in-code context. If they weren't across an ocean I would have shook them by the shoulders until they understood.
• onemoresoop 2y ago
  > I wonder if LLMs can generate docs at any reasonable level of usefulness and accurate A lot of times documentation isn’t read by anyone for stretches of time till it’s needed. I imagine documentation by LLMs will make the situation worse if nobody verifies its corectness. I don’t believe in magic solution, good documentation that is not out of date requires work and if a company doesn’t allocate resources no magic will fill in the gaps.