2 ms·

Even without introducing LLMs into the equation, I've been brought on as the technical writer for many projects where the team says "oh, we already have a readm

Even without introducing LLMs into the equation, I've been brought on as the technical writer for many projects where the team says "oh, we already have a readme, you just need to clean it up" and then all of the readme definitions for parameters or settings or whatever are like: brickLock: The lock of the brick. brickDrink: The drink of the brick. brickWink: The wink of the brick. ...which is to say, definitions that just restate whatever's evident from the code or variable names themselves, and that make sense if you're already familiar with the thing being defined, but don't actually explain their purpose or provide context for how to use them (in other words, the main reasons to have documentation). My role as a writer is then to (1) extract net-new information out of the team, (2) figure out how all of that new info fits together, (3) figure out the implications of that info for readers/users, and then (4) assemble it in an attractive manner. An autogenerated code wiki (or a lazy human) can presumably do the fourth step, but it can't do the first three steps preceding it, and without those preceding steps you're just rearranging known data. There are times where that can be helpful, but it's more often just gloss

Neywiny 11mo ago
This is what I wanted to focus on so thanks for starting the convo. This all feels like 100% coverage = perfectly tested, no bugs possible. Nooooo, there needs to be more than that. I lately had a really good readme for a project in a heavy development phase. Basically everything I'd done, every command, every concept, got documented. That's worry about cleanup later. I did not put in every line of code, I put concepts. So when a new person got brought on and asked stuff like "well but how do I change the config?" Bam, it's in the readme. Over and over, every task I had to do, they had to at least consider or understand, so it's in the readme. Of course I did start with a quick-start "how do I use this repo" and only later did "how do I develop this repo" but still, it was all useful because it's what I needed. It doesn't seem impossible for an LLM to go "hmmm, the way this repo passes configurations around isn't standard. I should focus more on that." But that's a level of understanding I don't think they currently have
- RealityVoid 11mo ago
  > But that's a level of understanding I don't think they currently have I think they do, at least in some of the cases, especially if it's something well represented in the dataset. I've been surprised sometimes by the insights it provides, and other times it's completely useless. That's one of the problems, it's unreliable, so you have to treat all info it gives you with doubt. But, anyways, at times it makes very surprising and seeming intelligent observations. It's worth at least considering it and thinking it through.
  - Neywiny 11mo ago
    I guess I should try it before dismissing it, but I would be curious to see if it can accurately detect which things we've found workarounds for that need special attention and whatnot.
    - RealityVoid 11mo ago
      As mentioned, it's a throw of the die. It can find very obscure things that you forgot about and give you good ideas. It might come with utterly stupid ideas. You clearly need to drive these things else they will drive you and you won't like where it will take you.
rkomorn 11mo ago
Sorry for the tangent but is there a story behind the choice of "brick lock/drink/wink" for your example? It's so odd and random it seems like there must be more to it.
- cryzinger 11mo ago
  No story, just a truly random choice lol. I was trying to come up with something completely meaningless just to show how unhelpful those descriptions are, but believe me that I've encountered many real examples that were equally inscrutable :P
moomin 11mo ago
There’s a tool that used to be popular in the .NET community called GhostDoc, that did pretty much exactly what you’re describing: rewording the blindingly obvious. I loathed it. But in terms of filling a very specific and all-too-common niche of “My manager’s insisting we do this thing, but has allocated no time to it, and will never spend more than five minutes verifying that it is done” it was excellent. I feel like Google is just creating the next generation of that technology and it will be very effective at solving the same problem.

Neywiny 11mo ago

This is what I wanted to focus on so thanks for starting the convo. This all feels like 100% coverage = perfectly tested, no bugs possible. Nooooo, there needs to be more than that. I lately had a really good readme for a project in a heavy development phase. Basically everything I'd done, every command, every concept, got documented. That's worry about cleanup later. I did not put in every line of code, I put concepts. So when a new person got brought on and asked stuff like "well but how do I change the config?" Bam, it's in the readme. Over and over, every task I had to do, they had to at least consider or understand, so it's in the readme. Of course I did start with a quick-start "how do I use this repo" and only later did "how do I develop this repo" but still, it was all useful because it's what I needed. It doesn't seem impossible for an LLM to go "hmmm, the way this repo passes configurations around isn't standard. I should focus more on that." But that's a level of understanding I don't think they currently have

RealityVoid 11mo ago
> But that's a level of understanding I don't think they currently have I think they do, at least in some of the cases, especially if it's something well represented in the dataset. I've been surprised sometimes by the insights it provides, and other times it's completely useless. That's one of the problems, it's unreliable, so you have to treat all info it gives you with doubt. But, anyways, at times it makes very surprising and seeming intelligent observations. It's worth at least considering it and thinking it through.
- Neywiny 11mo ago
  I guess I should try it before dismissing it, but I would be curious to see if it can accurately detect which things we've found workarounds for that need special attention and whatnot.
  - RealityVoid 11mo ago
    As mentioned, it's a throw of the die. It can find very obscure things that you forgot about and give you good ideas. It might come with utterly stupid ideas. You clearly need to drive these things else they will drive you and you won't like where it will take you.

rkomorn 11mo ago

Sorry for the tangent but is there a story behind the choice of "brick lock/drink/wink" for your example? It's so odd and random it seems like there must be more to it.

cryzinger 11mo ago
No story, just a truly random choice lol. I was trying to come up with something completely meaningless just to show how unhelpful those descriptions are, but believe me that I've encountered many real examples that were equally inscrutable :P

moomin 11mo ago

There’s a tool that used to be popular in the .NET community called GhostDoc, that did pretty much exactly what you’re describing: rewording the blindingly obvious. I loathed it. But in terms of filling a very specific and all-too-common niche of “My manager’s insisting we do this thing, but has allocated no time to it, and will never spend more than five minutes verifying that it is done” it was excellent. I feel like Google is just creating the next generation of that technology and it will be very effective at solving the same problem.