5 ms·
I have this problem even in codebases. Claude will work on something, and add detailed comments in which it extrapolates the from the design and confidently sta
by synalx 1mo ago
I have this problem even in codebases. Claude will work on something, and add detailed comments in which it extrapolates the from the design and confidently states intentions and decisions which aren't actually grounded in reality. Then, later sessions suffer when it reads back those hallucinations and treats them as canonical.
- dan_gggggg 1mo ago[dead]
- mrtesthah 1mo agoProbably good to establish canonical spec documents up front that it can use and maintain as an independent reference?
- lupire 1mo agoDoesn't need to be "up front", but you do need to curate the context. AI can't read your mind, and it will read what you (or anyone else) wrote.
- pydry 1mo agoIt can't but it will try and getting it to stop is often infeasible.
- Sharlin 1mo agoHumans can’t read minds either, but most humans aren’t so dumb as to forget what is a normative spec and what is just their own current, evolving model of the problem space. (Though some humans certainly are…)
- rendaw 1mo agoThat doesn't solve the problem of it hallucinating more design details and putting them in comments. The solution to too much authoritative documentation isn't more authoritative documentation.
- pc86 1mo agoComments exist for humans to understand the why of the code, including perhaps external dependencies/assumptions/requirements. LLMs inserting code comments makes zero sense by definition.
- pavlov 1mo agoThat’s not true because LLMs clearly benefit from having multiple statements representing aspects of the same thing. That’s why they’re always doing the “it’s not X, it’s Y” kind of patterns. It’s reinforcement.
- selcuka 1mo ago> LLMs inserting code comments makes zero sense by definition. Not completely, because anyone who reviews the code in the future does not have access to your original prompt, so theoretically leaving a comment that explains the "why" portion of the prompt would be useful. LLMs rarely do that, though.
- hombre_fatal 1mo agoLLMs benefit from knowing the why, the intent, and the reason/justification for code to exist, too. Catching code desyncing from it is a valuable place to reconsider assumptions and maybe even invariants.
- throwaway613746 1mo agoAnthropic hopes you just eventually give up and become completely dependent on Claude. Toss in 10 (or more) other developers with way less discipline and you will become dissillusioned with Claude's overengineered, incomprehensible, technosalad and your demise is pretty much guaranteed.
- recursive 1mo agoRetirement feels farther away than ever.
- bsjdhdudj 1mo agowhy would you not delete the comments? why are they saved or checked in? don't you read the code? and if you don't: ISN'T THIS WHY YOU NEED TO? I feel like I'm losing my mind, what the fuck is wrong with all of you?
- dan_gggggg 1mo ago> what the fuck is wrong with all of you? They spent 3 years pretending that their magic robot made them 10x harder/better/faster/stronger than the lazy peons rubbing shoulders with them and now the bill has come due.
- deleted 1mo ago[deleted]
- post-it 1mo agoI've been whittling my way towards having Claude write zero comments in code. They're pretty much never helpful.
- sdeframond 1mo agoHow do you do it?
- nottorp 1mo agoI just delete them most of the time. But I keep my LLMs on a short leash. No 40000 line PRs.
- ch4s3 1mo agoYou could probably add linting rules to your tool of choice and tell claude that your lint tool has to pass.
- bartread 1mo agoDelete the comments: I’ve gone on a tear with these recently because they’re nothing but trouble. The absolute least worst outcome is they chew up your token budget. But what tends to happen, and this is much more serious, is they poison future work and make further modification of the codebase more burdensome and error prone.
- LoganDark 1mo agoAbsolutely. IMO, comments can explain historical reasoning for the code, but refactorings can benefit heavily from rethinking ideas from scratch, as opposed to trying to follow the same reasons. LLMs tend to be misguided by comments, probably by treating them as instructions. Ergo, get rid of them.
- acedTrex 1mo agoI completely block all LLM comments via pi extension, it makes using them significantly more enjoyable. If the LLM wants to add a comment it must ASK me explicitly to do so. Seeing LLM comments in other peoples code is very upsetting because theres just so much meaningless noise.
- copperx 1mo agoWhat's the extension, please?
- gessha 1mo agoAuthor probably cobbler it together themselves.
- furst-blumier 1mo agoWhich extension is it?
- acedTrex 1mo agoIts a personal extension i wrote myself, maybe I should put it up somewhere.. Its so simple though i dont know if its worth it vs just "go build it yourself"
- rco8786 1mo agoWe have this in our harness markdown: ``` ## Comments Use comments extremely sparingly. Most comments should be at the request of the user. When something warrants a comment, keep it to one or two lines: what the code does and why it's necessary. No background narrative, no replaying the investigation or failure mode, nothing a test name or the commit message already says. Applies to specs too. If a comment needs a paragraph, make the code clearer instead. ``` The comments Claude was leaving got absolutely out of control. Just lines and lines of LLM drivel that was barely intelligible and not remotely relevant to what a code comment should be used for.
- preommr 1mo agoI've had codex delete useful (albeit not directly relevant or perhaps messy wip notes) comments, even though I explicitly have it in my agents.md not to delete comments, and ask for permission if it thinks it should. It deleted the comments, and when I asked why it did that even though I expressedly asked it not to, it responded that me prompting it in the first place explicit permission. I have no idea if that's the actual reason or just some post-hoc explanation. But I genuinely don't think it's possible to just have these things be completely, 100%, indpenedent and also solve deep problems that need to also be understood by people in a people-based organization context.
- recursive 1mo agoThere might not be an "actual" reason that's expressible in human-comprehensible language.
- wafflemaker 1mo agoMoreover, even if there was a reason, it was there "in the model" at the moment of generation. The new model can only guess/hallucinate when ordered to give reasons. It's like having an actor play a role while wearing a hat. Then the next actor comes, puts on the same hat, and we're asking them to explain why "they" did something while there was someone else playing the role.
- debugnik 1mo ago> and when I asked why it did that even though I expressedly asked it not to Just be clear, it can't know, and by asking you're just making it roleplay as someone excusing themselves. It's very unlikely that the choice to remove the comment was driven by an internal monologue based on learned criteria that it can refer to. The sampler most likely picked tokens to remove it while writing the patch, because that's what the statistics modelled, and that's it.
- coder-pm 1mo agoMy approach is to keep the comments only with a references. My LLM based projects always have the decision log where are my decisions while working on the features are stored with the date. I found this useful to actually trace why something is in the codebase. It's much easier for both me and the LLM to navigate through the big projects where I spent months and dozens of full night sessions executing my plans with --dangerously-skip-permissions. In the morning I was answering all the model questions and iterating like that. Honestly, try that.
- Utilera 1mo agoThis is almost worse in code because the hallucination starts looking like documentation
- andai 1mo agoA computer cannot be held accountable. Therefore, a computer must never make a decision. -IBM, 1979 (paraphrased)
- gmueckl 1mo agoEven worse to me is the tendency to not properly rewrite when prompted to change something in a text. It will always add to the document instead of replacing any obsolete information. The lengths it goes to to turn every document into a (useles) decision log is astounding and annoying.
- t-writescode 1mo agoSo, I admit that I'm finally getting into this "let Claude help you write code" stuff, and I'm enjoying it a bit for the stuff I, honestly, just don't want to write; but ... I still read it. I still review it. I still look at the code like I'm peer-reviewing it and go, "Uh ... I don't like this area at all. That's going to be really hard to debug at 3am" and so on. I guess my thought / question is: this is your code. Why are you submitting the code to the git repo like that, with all the wrong text? I get making mistakes here or there; but ... everywhere? Enough that it's causing huge problems you can't quickly correct? And ... aren't you lucky you still *have* the context in your mind? What if you hadn't caught it, or someone else had done that and you inherited that code with those wrong comments that you let through the PR and into the main branch?
- bojan 1mo ago> Why are you submitting the code to the git repo like that, with all the wrong text? According to my enterprise architect I shouldn't be reviewing code, it's a waste of time in this new reality. I'm still doing it because it's going to be me answering that 3 AM call. But I don't know for how long I'll be allowed to swim upstream like that.
- qsera 1mo agoI have found manual reviewing of LLM generated code to be an uphill battle. LLMs does not believe in abstraction. So complexity spills everywhere. Some details in the lowest level might be handled separately in more than one place, at the higher levels. In a short while it is a copiously documented unreadable mess (both code and documentation). LLM written PR descriptions and comments are a sight to behold. I am not sure how stuff can be written so cryptically. It seems that LLMs just make up what ever terminology so that it can cram as much details into a single sentence as possible! I generally just paste it to chatgpt and ask it to decrypt it.
- sfn42 1mo agoYeah, that happens when you vibe code and just let the LLM be in control of everything. If you take responsibility for the architecture and instruct it to do things properly it will do them properly. You can tell it exactly how to do it or you can ask it to handle it in a way that avoids duplication, you can tell it to design a reusable abstraction for this usecase etc.
- p0w3n3d 1mo agoIn numerical methods this is called error propagation. And since the neural networks are basically numerical approximations of a certain function, this shows apparent similarity of the propagation on a macro scale
- abjhn 1mo ago[dead]