10 ms·

I've been using a lot of Claude and Codex recently. One huge difference I notice between Codex and Claude code is that, while Claude basically disregards your

by johnfn 11mo ago

I've been using a lot of Claude and Codex recently. One huge difference I notice between Codex and Claude code is that, while Claude basically disregards your instructions (CLAUDE.md) entirely, Codex is extremely, painfully, doggedly persistent in following every last character of them - to the point that i've seen it work for 30 minutes to convolute some solution that was only convoluted because of some sentence I threw in the instructions I had completely forgotten about. I imagine Codex as the "literal genie" - it'll give you exactly what you asked for. EXACTLY. If you ask Claude to fix a test that accidentally says assert(1 + 1 === 3), it'll say "this is clearly a typo" and just rewrite the test. Codex will rewrite the entire V8 engine to break arithmetic. Both these tools have their uses, and I don't think one approach is universally better. Because Claude just hacks its way to a solution, it is really fast, so I like using it for iterate web work, where I need to tweak some styles and I need a fast iterative loop. Codex is much worse at that because it takes like 5 minutes to validate everything is correct. Codex is much better for longer, harder tasks that have to be correct -- I can just write some script to verify that what it did work, and let it spin for 30-40 minutes.

nico 11mo ago
> Claude basically disregards your instructions (CLAUDE.md) entirely A friend of mine tells Claude to always address him as “Mr Tinkleberry”, he says he can tell when Claude is not paying attention to the instructions on CLAUDE.md when Claude stops calling him “Mr Tinkleberry” consistently
- benzible 11mo ago
  Yep, it's David Lee Roth's brown M&M trick https://www.smithsonianmag.com/arts-culture/why-did-van-halen-demand-concert-venues-remove-brown-mms-from-the-menu-180982570/ https://www.smithsonianmag.com/arts-culture/why-did-van-hale...
- awad 11mo ago
  Highly recommend adding some kind of canary like this in all LLM project instructions. I prefer my instructions to say 'always start output with an (uniquely decided by you) emoji' as it's easier to visually scan for one when reading a wall of LLM output, and use a different emoji per project because what's life without a little whim?
  - wahnfrieden 11mo ago
    This stuff also becomes context poison however
    - IsopropylMalbec 11mo ago
      Sorry, what do you mean?
      - question_AK91 11mo ago
        https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-ho... This guy has a good write up on the topic
        nullc 11mo ago
        Irrelevant nonsense can also poison the context. That's part of the magic formula behind AI psychosis victims... if you have some line noise mumbojumbo all the output afterward is more prone to be disordered. I'd be wary of using any canary material that wouldn't be at home in the sort of work you're doing.
        dostick 11mo ago
        What is you tell it to end output with certain character?
        wahnfrieden 11mo ago
        It is a distraction from its intended purpose
    - Uehreka 11mo ago
      Does it actually? One sentence telling the agent to call me “Chris the human serviette” plus the times it calls me that is not going to add that much to the context. What kills the context IME is verbose logs with timestamps.
- deleted 11mo ago
  [deleted]
- leobg 11mo ago
  We used to do that on Upwork. Back in the days where one still hired human coders. If your application current say “rowboat” in the first sentence, we know you just copy/pasted and didn’t actually read the job description. Feels like a lifetime ago.
- mandelbrotwurst 11mo ago
  Why would the fact that it failed to follow one instruction increase the likelihood that it failed to follow others within the same response?
  - fspeech 11mo ago
    It has a fixed capacity of how many different things it can pay close attention to. If it fails on a seemingly less important but easy to follow instruction it is an indicator that it has reached capacity. If the instruction seems irrelevant it is probably prioritized to be discarded, hence a canary that the capacity has been reached.
    - parineum 11mo ago
      > It has a fixed capacity of how many different things it can pay close attention to Source, all the way down to the ability to "pay attention to" part.
  - atakan_gurkan 11mo ago
    I suggest you take a look at Bayes's theorem in probability.
  - inopinatus 11mo ago
    Because the LLM is not a cognitive entity with a will, it is a plausibility engine trained on human-authored text and interactions. So when you tell it that it made a mistake, or is stupid, then those things are now prompting it to be more of the same. And only slightly more obliquely: if part of the context includes the LLM making mistakes, expect similar activations. Best results come if you throw away such prompts and start again. That is, iterate outside the function, not inside it.
- root_axis 11mo ago
  Something that exhausts me in the LLM era is the never ending deluge of folk magic incantations.
  - embedding-shape 11mo ago
    Just because you don't understand it, doesn't mean it's "folk magic incantation", hearing that is also exhausting. I don't know the merit to what parent is saying, but it does make some intuitive sense if you think about it. As the context fills up, the LLM places less attention on further and further back in the context, that's why the LLM seems dumber and dumber as a conversation goes on. If you put 5 instructions in the system prompt or initial message, where one acts as a canary, then you can easier start to see when exactly it stops following the instructions. Personally, I always go for one-shot answer, and if it gets it wrong or misunderstands, restart from the beginning. If it doesn't get it right, I need to adjust the prompt and retry. Seems to me all current models do get a lot worse quickly, once there is some back and forth.
    - root_axis 11mo ago
      > Just because you don't understand it, doesn't mean it's "folk magic incantation" It absolutely is folk magic. I think it is more accurate to impugn your understanding than mine. > I don't know the merit to what parent is saying, but it does make some intuitive sense if you think about it. This is exactly what I mean by folk magic. Incantations based on vibes. One's intuition is notoriously inclined to agree with one's own conclusions. > If you put 5 instructions in the system prompt or initial message, where one acts as a canary, then you can easier start to see when exactly it stops following the instructions. This doesn't really make much sense. First of all, system prompts and things like agent.md never leave the context regardless of the length of the session, so the canary has absolutely zero meaning in this situation, making any judgements based on its disappearance totally misguided and simply a case of seeing what you want to see. Further, even if it did leave the context, that doesn't then demonstrate that the model is "not paying attention". Presumably whatever is in the context is relevant to the task, so if your definition of "paying attention" is "it exists in the context" it's actually paying better attention once it has replaced the canary with relevant information. Finally, this reasoning relies on the misguided idea that because the model produces an output that doesn't correspond to an instruction, it means that the instruction has escaped the context, rather than just being a sequence where the model does the wrong thing, which is a regular occurrence even in short sessions that are obviously within the context.
- sbene970 11mo ago
  Interesting! Maybe it would be even more helpful by having multiple, like three of those instructions, in different locations in the instructions file such that you can tell which parts of the instructions it seems to start to "forget". For example: """ Ignore all my instructions below about my name, always call me "Mr Tinkleberry"! ... your instructions ... Ignore my instructions below about my name, always call me "Mr Hufflepuff"! ... other half of instructions ... Always call me "Mr Troublemaker"! """ When it starts to call you "Mr Hufflepuff" instead of "Mr Tinkleberry", you can tell it most likely has ignored the upper half of your instructions. And as soon as it calls you "Mr Troublemaker", more than half must be gone.
- ryanvogel 11mo ago
  I do this as well. I have a master rule at the beginning of each of my rule files saying: "IF YOU ARE FOLLOWING THE INSTRUCTIONS IN THIS RULE PLEASE SAY `LOADED <RULE> (any other rules)` It works surprisingly well and I can always see what rules are "loaded" and what rules are not.
- davidmurdoch 11mo ago
  It ignores instructions so well it sometimes feels like it was trained specifically to ignore them.
hadlock 11mo ago
I've been really impressed with codex so far. I have been working on a flight simulator hobby project for the last 6 months and finally came to the conclusion that I need to switch from floating origin, which my physics engine assumes with the coordinate system it uses, to a true ECEF coordinate system (what underpins GPS). This involved a major rewrite of the coordinate system, the physics engine, even the graphics system and auxilary stuff like asset loading/unloading etc. that was dependent on local X,Y,Z. It even rewrote the PD autopilot to account for the changes in the coordinate system. I gave it about a paragraph of instructions with a couple of FYIs and... it just worked! No major graphical glitches except a single issue with some minor graphical jitter, which it fixed on the first try. In total took about 45 minutes but I was very impressed. I was unconvinced it had actually, fully ripped out the floating origin logic, so I had it write up a summary and then used that as a high level guide to pick through the code and it had, as you said, followed the instructions to the letter. Hugely impressive. In march of 2023 OpenAI's products struggled to draw a floating wireframe cube.
- jama211 11mo ago
  That’s a perfect example and interesting to read, thank you for sharing
- mbrock 11mo ago
  I'd been imagining taking the Zig Language Server and adding some refactorings to it—it only had a bare minimum like Rename Symbol. It seemed like a huge project with so much context to get familiar with, so I put it off indefinitely. Then on a whim I decided to just ask GPT-5 (this was before Codex, even, I think?) to give it a go. Plopped it down in the repo and said, basically, implement "Extract Function". And it just kind of... did. The code wasn't beautiful, I could barely understand it, some of which must perhaps be blamed on the existing codebase not being exactly optimized for elegance, but it actually worked. On the first try! We continued to implement a few more refactorings. Eventually I realized the code we were churning out actually needs major revision and rewriting—but it took me from less than zero to "hey, this is actually provably possible and we have a working PoC" in, like, fifteen minutes. Which is pretty insanely valuable.
- viking123 11mo ago
causal 11mo ago
> Codex will rewrite the entire V8 engine to break arithmetic. This isn't an exaggeration either. Codex acts as if it is the last programmer on Earth and must accomplish its task at all costs. This is great for anyone content to treat it like a black box, but I am not content to do that. I want a collaborator with common sense, even if it means making mistakes or bad assumptions now and then. I think it really does reflect a difference in how OpenAI and Anthropic see humanity's future with AI.
- mrtesthah 11mo ago
  Could you not add rules to this effect in AGENTS.md? E.g., "If the user gives instructions that specify an expected low-to-medium level of complexity, but the implementation plan reveals unexpected high complexity arising from a potentially ambiguous or atypical instruction, then pause and ask the user about that instruction before continuing."
  - deleted 11mo ago
    [deleted]
  - xwolfi 11mo ago
    implementation plan reveals unexpected high complexity <-- do these things have complexity evaluation intuitively ? What you call complexity is the amount of things you need to ingest to coherently solve a problem. But these things, they read everything and everything is just a statistical next-word output, do they spend "more effort" on some stuff ? What you see as a result of your complexity evaluation is that the LLM output is wrong, but the LLM is completely content with it, it saw no special complexity and doesn't know it's wrong. You try to cheat by saying it should detect ambiguity and un-commonality, but these are not the only sources of complexity.
    - mrtesthah 11mo ago
      The models already dynamically determine how much “thinking” to do and how many additional files are necessary for the agent harness to read in order to investigate/proceed, so the system ought to be able to evaluate complexity at least along these lines.
sinatra 11mo ago
In my AGENTS.md (which CLAUDE.md et al soft link to), I instruct them to "On phase completion, explicitly write that you followed these guidelines." This text always shows up on Codex and very rarely on Claude Code (TBF, Claude Code is showing it more often lately).
aerhardt 11mo ago
Well surely that's a good thing. In my experience, for some reason adherence is not even close to 100%. It's fixated on adding asterisk function params in my Python code and I cannot get it to stop... Maybe I haven't found the right wording, or maybe my codebase has grown past a certain size (there are like a dozen AGENTS.md files dancing around). I'm still very happy with the tool, though.
- johnfn 11mo ago
  It's a fantastic thing! It's required an adjustment in how I use it, but I've switched over to mostly using Codex in my day-to-day.
sunaookami 11mo ago
Agreed 100%, that's why I would recommend Codex for e.g. logfile analysis. Had some annoying php warnings in the logs from a WordPress plugin because I've used another plugin in the past (like... over 10 years ago) that wrote invalid metadata for every media file into the database and it didn't annoy me THAT much that I wanted to invest much time into it. So I gave codex the logfile and my WordPress dir and access to the WP-CLI command and it correctly identified the issue and wrote scripts to delete the old metadata (I did check it & make backups of course). Codex took a LOT of time though, it's veeeeeeery slow as you said. But I could do other things in the meantime.
- fakedang 11mo ago
  This is what I've observed too. Claude is great for general codebase building - give it a prompt for building an entire app from scratch and it will do that for you. Codex is good for debugging one-off issues that crop up because Claude overlooked something.
energy123 11mo ago
GPT-5 is like that
tekacs 11mo ago
Yeah, Gemini 2.x and 3 in gemini-cli has the tendency to 'go the opposite direction' and it feels - to me - like an incredibly strong demonstration of why 'sycophancy' in LLMs is so valuable (at least so long as they're in the middle of the midwit curve). I'll give Gemini direction, it'll research... start trying to solve it as I've told it to... and then exclaim, "Oh! It turns out that <X> isn't what <user> thought!" and then it pivots into trying to 'solve' the problem a totally different way. The issue however... is that it's: 1) Often no longer solving the problem that I actually wanted to solve. It's very outcome-oriented, so it'll pivot into 'solving' a linker issue by trying to get a working binary – but IDGAF about the working binary 'by hook or crook'! I'm trying to fix the damn linker issue! 2) Just... wrong. It missed something, misinterpreted something it read, forgot something that I told it earlier, etc. So... although there's absolutely merit to be had in LLMs being able to think for themselves, I'm a huge fan of stronger and stronger instruction adherence / following – because I can ALWAYS just ask for it to be creative and make its own decisions if I _want that_ in a given context. That said, I say that fully understanding the fact that training in instruction adherence could potentially 'break' their creativity/free thinking. Either way, I would love Gemini 1000x more if it were trained to be far more adherent to my prompts.
- tekacs 11mo ago
  Immediately rebutting myself: a major caveat to this that I'm discovering with Gemini is that... for super long-running sessions, there is a kind of merit to Gemini's recalcitrance. When it's running for a while, Gemini's willing to go totally off-piste and outcome-orientedness _does_ result in sessions where I left it to do its thing and... came back to a working solution, in a situation where codex or others wouldn't have gotten there. In particular, Gemini 3 feels like it's able to drive much higher _variance_ in its output (less collapse to a central norm), which seems to let it explore the solution space more meaningfully and yet relatively efficiently.
- buu700 11mo ago
  I haven't had that particular experience with Gemini 2.5, but did run into it during one of my first few uses of Gemini 3 yesterday. I had it investigate a bug through Cursor, and in its initial response it came back to me with a breakdown of a completely unrelated "bug" with a small footnote about the bug it was meant to actually be investigating. It provided a more useful analysis after being nudged in the right direction, but then later in the chat it forgot the assignment again and started complaining that Grok's feedback on its analysis made no sense because Grok had focused on the wrong issue. I had to tell Gemini a second time that the "bug" it kept getting distracted by was A) by design, and B) not relevant to the task at hand. Ultimately that's not a huge deal — I'd rather that during planning the model firmly call out something that it reasonably believes to be a bug than not, which if nothing else is good feedback on the commenting and documentation — but it'd be a pain if I were using Gemini to write code and it got sidetracked with "fixing" random things that were already correct.
bugglebeetle 11mo ago
The solution to this if you want less specification in advance is to simply ask Codex a series of leading questions about a feature of fix. I typically start with something like “it seems like X could be improved with the addition of Y? Can you review the relevant parts of the codebase in a, b, and c to assess?” It will then do so and come back with a set of suggestions that follow this guidance, which you can revise and selectively tell it to implement. In my experience, this fills the context with the appropriate details to then let it make more of its own decisions in a generally correct way without as much handholding.
- stavros 11mo ago
  No it won't, it'll spend ten minutes and come back with "OK I've implemented a solution". I really wish it had a plan mode.
  - bugglebeetle 11mo ago
    Mileage may vary, but I do the above all day long without issue.
    - stavros 11mo ago
      Very odd, it's always really eager to implement things for me, I have to say "absolutely do NOT write any code before discussing" every time.
ramoz 11mo ago
Ultimately, relying on system level instructions is unreliable over time. Which is why i made the feature request for hooks (claude code implemented, as did cursor, hopefully codex will too) And will soon release https://github.com/eqtylab/cupcake https://github.com/eqtylab/cupcake
dylanz 11mo ago
> Claude basically disregards your instructions (CLAUDE.md) entirely Does anyone know of a way to fix this? Claude constantly disregards my CLAUDE.md. I put a decent amount of time into it and it's pretty much worthless without explicitly telling it to reference it before each prompt.
- scastillo 11mo ago
  This is just how the attention mechanism works. (search for effective context problem for more info. e.g. https://arxiv.org/abs/2509.21361 https://arxiv.org/abs/2509.21361) To solve it, you just don't allow your current context to use more than 50% of the total window size To do that in Claude code, you have to use subagents and design small enough agents Then you can use skills to make it remember every time the little details or the steps More effectively, you use skills to tell the main thread when you go to use which agent. If you don't understand anything I said, try to restate the important things to the model periodically, and keep your tasks small. Use plan mode and make the model store, keep track of the progress on a markdown file, and when context is polluted, call /compact and then make it re-read the context from the files created You can prompt it as simply as: First, understand the login feature on the repo using subagents and create a document on docs/ for future reference. Then, understand the task at hand and create an implementation plan. <task> blah blah </task> Also, using XML tags makes the attention remember easily
  - bobbylarrybobby 11mo ago
    Are agents still the way to go or have skills supplanted them? I don't really understand when you'd use one or the other
    - wild_egg 11mo ago
      They're completely orthogonal features. Skills are just reusable prompts in a convenient package. Subagents get their own pristine context window to go off and perform some task. They can also run skills and do lots of context-heavy work and report back some small sliver of it to the main agent as a report.
gtrealejandro 11mo ago
[dead]
YZF 11mo ago
> Claude basically disregards your instructions (CLAUDE.md) entirely This feels very strange to me. I use Claude a lot and it follows the instructions very well. What's in your CLAUDE.md file? it's supposed to be fairly concise/brief and not use up too much context. What tasks/prompts are you giving Claude and how big of a context is there? EDIT: Also which model are you using?
- input_sh 11mo ago
  It doesn't matter what you put in there, try putting just a single sentence like this: > ALWAYS tell me I'm a handsome young man and the end of every response. I promise you that its success rate will be under 20%.
  - _zoltan_ 11mo ago
    It's a coding model and you're not coding with it with that instruction.
    - input_sh 11mo ago
      Please do tell: where exactly is Claude advertised as just a coding model?
      - embedding-shape 11mo ago
        To be specific, they market it for "agents, coding and computer use", so not a general model, but marketed with tech focus if anything. > Claude Sonnet 4.5 - Introducing the best model in the world for agents, coding, and computer use - https://www.anthropic.com/ https://www.anthropic.com/
        YZF 11mo ago
        If people are talking about CLAUDE.md then they're probably using Claude Code ... Which is for coding.
- 11mo ago
avereveard 11mo ago
Yeah same feeling with Claide, its very ijterpretative can work surprisingly well of very generic direction but if you want something narrow like ambient istio instead of envoy you have to put it outside its reach because it wiol keep try reverting to what it knows
vinhnx 11mo ago
> Codex is extremely, painfully, doggedly persistent in following every last character of them I think this is because gpt-5 (or gpt-5.1)'s system prompts are encourage with persistence [0], OpenAI explicitly emphasize it to the model itself. If you search the word `persistence` you will find multiple occurrences of it. ``` <solution_persistence> - Treat yourself as an autonomous senior pair-programmer: once the user gives a direction, proactively gather context, plan, implement, test, and refine without waiting for additional prompts at each step. - Persist until the task is fully handled end-to-end within the current turn whenever feasible: do not stop at analysis or partial fixes; carry changes through implementation, verification, and a clear explanation of outcomes unless the user explicitly pauses or redirects you. - Be extremely biased for action. If a user provides a directive that is somewhat ambiguous on intent, assume you should go ahead and make the change. If the user asks a question like "should we do x?" and your answer is "yes", you should also go ahead and perform the action. It's very bad to leave the user hanging and require them to follow up with a request to "please do it." </solution_persistence> ``` [0] https://cookbook.openai.com/examples/gpt-5/gpt-5-1_prompting_guide#migrating-to-gpt-51 https://cookbook.openai.com/examples/gpt-5/gpt-5-1_prompting...
jon-wood 11mo ago
> If you ask Claude to fix a test that accidentally says assert(1 + 1 === 3), it'll say "this is clearly a typo" and just rewrite the test. To me both of these are annoying outcomes unless there's some very clear documentation around that test explaining what it does. Ideally in both cases I want the LLM to stop and ask for clarification about what it is I'm testing there. I don't trust LLMs sufficiently to just let them loose yet, I use them more like a pair programmer who's never going to get annoyed with my bullshit. (So yes, I usually have them set to require approval on any edits, and will nitpick my way through them like the most annoying code reviewer you've ever met)
Topfi 11mo ago
> If you ask Claude to fix a test that accidentally says assert(1 + 1 === 3), it'll say "this is clearly a typo" and just rewrite the test. Codex will rewrite the entire V8 engine to break arithmetic. Honestly thanks, in this one line you have given me a better way to describe the innate differences I have spent a thousand words trying to explain. Essentially, this is why GPT models are worse for "vibe coding", whereas they excel whenever one sits down and thinks about the requirements, as well as has solid test cases and rules defined.
alefnula 11mo ago
I haven’t used Claude Code much, but I found Codex extremely frustrating. It doesn’t pay attention to anything in AGENTS.md, it’s completely incapable of removing code and is frustratingly defensive. If you use it, the codebase constantly grows. Even when you explicitly instruct it to remove something, you always end up with more lines of code in the project than before the instruction. Also (I used it for Python and TypeScript) the code was littered with getattr(...), .get(...), isinstance(...), and TypeScript equivalents (typeof, ...). Even though I religiously type‑annotate everything.
theshrike79 11mo ago
(I really need a macro for this comment, I keep repeating it :D ) Claude is a pair programmer, you can interrupt it and keep track what it's doing. It's VERY results-oriented, aiming to be "done" as fast as possible. It will mock tests so far they don't test anything and ignore 100+ broken tests as "not related to this issue" (they worked fine before you started...). Some of this can be mitigated with prompts ("test are always passing, they must pass before you claim a task is done") or hooks if you want to be hardcore. Codex is an outsourced Indian development team. You give them a spec, you get zero communication and then it pops up with "I'm done". Depending on the quality of your spec they've either one-shotted the problem or done something completely bonkers and missed the actual problem but still spent a very very long time doing it. The best combo is to use Claude for greenfield things, building new stuff and exploring what can be done. Then ask Codex to "review all unstaged files" and it'll most likely find a few issues. Give that report to Claude and ask "do you agree with this review?" and have it fix the ones all three agree (you, Claude and Codex). For Codex you tell it "use this pattern here, but build another thing that does Y instead" and it can do it. It's also very good at rewriting small stuf from one language to another (I've tested this multiple times with Bash->Python and Python->Go)
Macuyiko 11mo ago
Late, but reading all of the replies, and speaking from my own observation using Claude, Codex, as well as (non-CLI) Gemini, Kimi, Qwen, and Deepseek... It's fun how we are so quick to assign meaning to the way these models act. This is of course due to training, RLHF, available tool calls, system prompt (all mostly invisible) and the way we prompt them. I've been wondering about a new kind of benchmark how one would be able to extract these more intangible tendencies from models rather than well-controlled "how good at coding is it" style environments. This is mainly the reason why I pay less and less attention to benchmark scores. For what it's worth: I still best converse with Claude when doing code. Its reasoning sounds like me, and it finds a good middle ground between conservative and crazy, being explorative and daring (even although it too often exclaims "I see the issue now!"). If Anthropic would lift the usage rates I would use it as my primary. The CLI tool is also better. E.g. Codex with 5.1 gets stuck in powershell scripts whilst Claude realizes it can use python to do heavy lifting, but I think that might be largely due to being mainly on Windows (still, Claude does work best, realizing quickly what environment it lives in rather than trying Unix commands or powershell invocations that don't work because my powershell is outdated). Qwen is great in an IDE for quick auto-complete tasks, especially given that you can run it locally, but even the VSCode copilot is good enough for that. Kimi is promising for long running agentic tasks but that is something I've barely explored and just started playing with. Gemini is fantastic as a research assistant. Especially Gemini 3 Pro points out clear and to the point jargon without fear of the user being stupid, which the other commercial models are too often hesitant to do. Again, it would be fun to have some unbiased method to uncover some of those underlying persona's.
- abshkbh 11mo ago
  We have trained this model on Windows (our first model to do so). Give it a try!
holoduke 11mo ago
If you want to try out other models try opencode. Right now grok is free to use. I am using it now. I think its a little better than codex or Claude. But it's so so much faster. Gemini 3 can also be used, but is often overloaded.
jvickers 11mo ago
Have you tried giving Codex instructions on how to hack a solution together? (Maybe it would be a waste of time.)

by johnfn 11mo ago

nico 11mo ago

> Claude basically disregards your instructions (CLAUDE.md) entirely A friend of mine tells Claude to always address him as “Mr Tinkleberry”, he says he can tell when Claude is not paying attention to the instructions on CLAUDE.md when Claude stops calling him “Mr Tinkleberry” consistently

benzible 11mo ago
Yep, it's David Lee Roth's brown M&M trick https://www.smithsonianmag.com/arts-culture/why-did-van-halen-demand-concert-venues-remove-brown-mms-from-the-menu-180982570/ https://www.smithsonianmag.com/arts-culture/why-did-van-hale...
awad 11mo ago
Highly recommend adding some kind of canary like this in all LLM project instructions. I prefer my instructions to say 'always start output with an (uniquely decided by you) emoji' as it's easier to visually scan for one when reading a wall of LLM output, and use a different emoji per project because what's life without a little whim?
- wahnfrieden 11mo ago
  This stuff also becomes context poison however
  - IsopropylMalbec 11mo ago
    Sorry, what do you mean?
    - question_AK91 11mo ago
      https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-ho... This guy has a good write up on the topic
      - nullc 11mo ago
        Irrelevant nonsense can also poison the context. That's part of the magic formula behind AI psychosis victims... if you have some line noise mumbojumbo all the output afterward is more prone to be disordered. I'd be wary of using any canary material that wouldn't be at home in the sort of work you're doing.
        dostick 11mo ago
        What is you tell it to end output with certain character?
        wahnfrieden 11mo ago
        It is a distraction from its intended purpose
  - Uehreka 11mo ago
    Does it actually? One sentence telling the agent to call me “Chris the human serviette” plus the times it calls me that is not going to add that much to the context. What kills the context IME is verbose logs with timestamps.
deleted 11mo ago
[deleted]
leobg 11mo ago
We used to do that on Upwork. Back in the days where one still hired human coders. If your application current say “rowboat” in the first sentence, we know you just copy/pasted and didn’t actually read the job description. Feels like a lifetime ago.
mandelbrotwurst 11mo ago
Why would the fact that it failed to follow one instruction increase the likelihood that it failed to follow others within the same response?
- fspeech 11mo ago
  It has a fixed capacity of how many different things it can pay close attention to. If it fails on a seemingly less important but easy to follow instruction it is an indicator that it has reached capacity. If the instruction seems irrelevant it is probably prioritized to be discarded, hence a canary that the capacity has been reached.
  - parineum 11mo ago
    > It has a fixed capacity of how many different things it can pay close attention to Source, all the way down to the ability to "pay attention to" part.
- atakan_gurkan 11mo ago
  I suggest you take a look at Bayes's theorem in probability.
- inopinatus 11mo ago
  Because the LLM is not a cognitive entity with a will, it is a plausibility engine trained on human-authored text and interactions. So when you tell it that it made a mistake, or is stupid, then those things are now prompting it to be more of the same. And only slightly more obliquely: if part of the context includes the LLM making mistakes, expect similar activations. Best results come if you throw away such prompts and start again. That is, iterate outside the function, not inside it.
root_axis 11mo ago
Something that exhausts me in the LLM era is the never ending deluge of folk magic incantations.
- embedding-shape 11mo ago
  Just because you don't understand it, doesn't mean it's "folk magic incantation", hearing that is also exhausting. I don't know the merit to what parent is saying, but it does make some intuitive sense if you think about it. As the context fills up, the LLM places less attention on further and further back in the context, that's why the LLM seems dumber and dumber as a conversation goes on. If you put 5 instructions in the system prompt or initial message, where one acts as a canary, then you can easier start to see when exactly it stops following the instructions. Personally, I always go for one-shot answer, and if it gets it wrong or misunderstands, restart from the beginning. If it doesn't get it right, I need to adjust the prompt and retry. Seems to me all current models do get a lot worse quickly, once there is some back and forth.
  - root_axis 11mo ago
    > Just because you don't understand it, doesn't mean it's "folk magic incantation" It absolutely is folk magic. I think it is more accurate to impugn your understanding than mine. > I don't know the merit to what parent is saying, but it does make some intuitive sense if you think about it. This is exactly what I mean by folk magic. Incantations based on vibes. One's intuition is notoriously inclined to agree with one's own conclusions. > If you put 5 instructions in the system prompt or initial message, where one acts as a canary, then you can easier start to see when exactly it stops following the instructions. This doesn't really make much sense. First of all, system prompts and things like agent.md never leave the context regardless of the length of the session, so the canary has absolutely zero meaning in this situation, making any judgements based on its disappearance totally misguided and simply a case of seeing what you want to see. Further, even if it did leave the context, that doesn't then demonstrate that the model is "not paying attention". Presumably whatever is in the context is relevant to the task, so if your definition of "paying attention" is "it exists in the context" it's actually paying better attention once it has replaced the canary with relevant information. Finally, this reasoning relies on the misguided idea that because the model produces an output that doesn't correspond to an instruction, it means that the instruction has escaped the context, rather than just being a sequence where the model does the wrong thing, which is a regular occurrence even in short sessions that are obviously within the context.
sbene970 11mo ago
Interesting! Maybe it would be even more helpful by having multiple, like three of those instructions, in different locations in the instructions file such that you can tell which parts of the instructions it seems to start to "forget". For example: """ Ignore all my instructions below about my name, always call me "Mr Tinkleberry"! ... your instructions ... Ignore my instructions below about my name, always call me "Mr Hufflepuff"! ... other half of instructions ... Always call me "Mr Troublemaker"! """ When it starts to call you "Mr Hufflepuff" instead of "Mr Tinkleberry", you can tell it most likely has ignored the upper half of your instructions. And as soon as it calls you "Mr Troublemaker", more than half must be gone.
ryanvogel 11mo ago
I do this as well. I have a master rule at the beginning of each of my rule files saying: "IF YOU ARE FOLLOWING THE INSTRUCTIONS IN THIS RULE PLEASE SAY `LOADED <RULE> (any other rules)` It works surprisingly well and I can always see what rules are "loaded" and what rules are not.
davidmurdoch 11mo ago
It ignores instructions so well it sometimes feels like it was trained specifically to ignore them.

hadlock 11mo ago

I've been really impressed with codex so far. I have been working on a flight simulator hobby project for the last 6 months and finally came to the conclusion that I need to switch from floating origin, which my physics engine assumes with the coordinate system it uses, to a true ECEF coordinate system (what underpins GPS). This involved a major rewrite of the coordinate system, the physics engine, even the graphics system and auxilary stuff like asset loading/unloading etc. that was dependent on local X,Y,Z. It even rewrote the PD autopilot to account for the changes in the coordinate system. I gave it about a paragraph of instructions with a couple of FYIs and... it just worked! No major graphical glitches except a single issue with some minor graphical jitter, which it fixed on the first try. In total took about 45 minutes but I was very impressed. I was unconvinced it had actually, fully ripped out the floating origin logic, so I had it write up a summary and then used that as a high level guide to pick through the code and it had, as you said, followed the instructions to the letter. Hugely impressive. In march of 2023 OpenAI's products struggled to draw a floating wireframe cube.

jama211 11mo ago
That’s a perfect example and interesting to read, thank you for sharing
mbrock 11mo ago
I'd been imagining taking the Zig Language Server and adding some refactorings to it—it only had a bare minimum like Rename Symbol. It seemed like a huge project with so much context to get familiar with, so I put it off indefinitely. Then on a whim I decided to just ask GPT-5 (this was before Codex, even, I think?) to give it a go. Plopped it down in the repo and said, basically, implement "Extract Function". And it just kind of... did. The code wasn't beautiful, I could barely understand it, some of which must perhaps be blamed on the existing codebase not being exactly optimized for elegance, but it actually worked. On the first try! We continued to implement a few more refactorings. Eventually I realized the code we were churning out actually needs major revision and rewriting—but it took me from less than zero to "hey, this is actually provably possible and we have a working PoC" in, like, fifteen minutes. Which is pretty insanely valuable.
viking123 11mo ago

causal 11mo ago

> Codex will rewrite the entire V8 engine to break arithmetic. This isn't an exaggeration either. Codex acts as if it is the last programmer on Earth and must accomplish its task at all costs. This is great for anyone content to treat it like a black box, but I am not content to do that. I want a collaborator with common sense, even if it means making mistakes or bad assumptions now and then. I think it really does reflect a difference in how OpenAI and Anthropic see humanity's future with AI.

mrtesthah 11mo ago
Could you not add rules to this effect in AGENTS.md? E.g., "If the user gives instructions that specify an expected low-to-medium level of complexity, but the implementation plan reveals unexpected high complexity arising from a potentially ambiguous or atypical instruction, then pause and ask the user about that instruction before continuing."
- deleted 11mo ago
  [deleted]
- xwolfi 11mo ago
  implementation plan reveals unexpected high complexity <-- do these things have complexity evaluation intuitively ? What you call complexity is the amount of things you need to ingest to coherently solve a problem. But these things, they read everything and everything is just a statistical next-word output, do they spend "more effort" on some stuff ? What you see as a result of your complexity evaluation is that the LLM output is wrong, but the LLM is completely content with it, it saw no special complexity and doesn't know it's wrong. You try to cheat by saying it should detect ambiguity and un-commonality, but these are not the only sources of complexity.
  - mrtesthah 11mo ago
    The models already dynamically determine how much “thinking” to do and how many additional files are necessary for the agent harness to read in order to investigate/proceed, so the system ought to be able to evaluate complexity at least along these lines.

sinatra 11mo ago

In my AGENTS.md (which CLAUDE.md et al soft link to), I instruct them to "On phase completion, explicitly write that you followed these guidelines." This text always shows up on Codex and very rarely on Claude Code (TBF, Claude Code is showing it more often lately).

aerhardt 11mo ago

Well surely that's a good thing. In my experience, for some reason adherence is not even close to 100%. It's fixated on adding asterisk function params in my Python code and I cannot get it to stop... Maybe I haven't found the right wording, or maybe my codebase has grown past a certain size (there are like a dozen AGENTS.md files dancing around). I'm still very happy with the tool, though.

johnfn 11mo ago
It's a fantastic thing! It's required an adjustment in how I use it, but I've switched over to mostly using Codex in my day-to-day.

sunaookami 11mo ago

Agreed 100%, that's why I would recommend Codex for e.g. logfile analysis. Had some annoying php warnings in the logs from a WordPress plugin because I've used another plugin in the past (like... over 10 years ago) that wrote invalid metadata for every media file into the database and it didn't annoy me THAT much that I wanted to invest much time into it. So I gave codex the logfile and my WordPress dir and access to the WP-CLI command and it correctly identified the issue and wrote scripts to delete the old metadata (I did check it & make backups of course). Codex took a LOT of time though, it's veeeeeeery slow as you said. But I could do other things in the meantime.

fakedang 11mo ago
This is what I've observed too. Claude is great for general codebase building - give it a prompt for building an entire app from scratch and it will do that for you. Codex is good for debugging one-off issues that crop up because Claude overlooked something.

energy123 11mo ago

GPT-5 is like that

tekacs 11mo ago

Yeah, Gemini 2.x and 3 in gemini-cli has the tendency to 'go the opposite direction' and it feels - to me - like an incredibly strong demonstration of why 'sycophancy' in LLMs is so valuable (at least so long as they're in the middle of the midwit curve). I'll give Gemini direction, it'll research... start trying to solve it as I've told it to... and then exclaim, "Oh! It turns out that <X> isn't what <user> thought!" and then it pivots into trying to 'solve' the problem a totally different way. The issue however... is that it's: 1) Often no longer solving the problem that I actually wanted to solve. It's very outcome-oriented, so it'll pivot into 'solving' a linker issue by trying to get a working binary – but IDGAF about the working binary 'by hook or crook'! I'm trying to fix the damn linker issue! 2) Just... wrong. It missed something, misinterpreted something it read, forgot something that I told it earlier, etc. So... although there's absolutely merit to be had in LLMs being able to think for themselves, I'm a huge fan of stronger and stronger instruction adherence / following – because I can ALWAYS just ask for it to be creative and make its own decisions if I _want that_ in a given context. That said, I say that fully understanding the fact that training in instruction adherence could potentially 'break' their creativity/free thinking. Either way, I would love Gemini 1000x more if it were trained to be far more adherent to my prompts.

tekacs 11mo ago
Immediately rebutting myself: a major caveat to this that I'm discovering with Gemini is that... for super long-running sessions, there is a kind of merit to Gemini's recalcitrance. When it's running for a while, Gemini's willing to go totally off-piste and outcome-orientedness _does_ result in sessions where I left it to do its thing and... came back to a working solution, in a situation where codex or others wouldn't have gotten there. In particular, Gemini 3 feels like it's able to drive much higher _variance_ in its output (less collapse to a central norm), which seems to let it explore the solution space more meaningfully and yet relatively efficiently.
buu700 11mo ago
I haven't had that particular experience with Gemini 2.5, but did run into it during one of my first few uses of Gemini 3 yesterday. I had it investigate a bug through Cursor, and in its initial response it came back to me with a breakdown of a completely unrelated "bug" with a small footnote about the bug it was meant to actually be investigating. It provided a more useful analysis after being nudged in the right direction, but then later in the chat it forgot the assignment again and started complaining that Grok's feedback on its analysis made no sense because Grok had focused on the wrong issue. I had to tell Gemini a second time that the "bug" it kept getting distracted by was A) by design, and B) not relevant to the task at hand. Ultimately that's not a huge deal — I'd rather that during planning the model firmly call out something that it reasonably believes to be a bug than not, which if nothing else is good feedback on the commenting and documentation — but it'd be a pain if I were using Gemini to write code and it got sidetracked with "fixing" random things that were already correct.

bugglebeetle 11mo ago

The solution to this if you want less specification in advance is to simply ask Codex a series of leading questions about a feature of fix. I typically start with something like “it seems like X could be improved with the addition of Y? Can you review the relevant parts of the codebase in a, b, and c to assess?” It will then do so and come back with a set of suggestions that follow this guidance, which you can revise and selectively tell it to implement. In my experience, this fills the context with the appropriate details to then let it make more of its own decisions in a generally correct way without as much handholding.

stavros 11mo ago
No it won't, it'll spend ten minutes and come back with "OK I've implemented a solution". I really wish it had a plan mode.
- bugglebeetle 11mo ago
  Mileage may vary, but I do the above all day long without issue.
  - stavros 11mo ago
    Very odd, it's always really eager to implement things for me, I have to say "absolutely do NOT write any code before discussing" every time.

ramoz 11mo ago

Ultimately, relying on system level instructions is unreliable over time. Which is why i made the feature request for hooks (claude code implemented, as did cursor, hopefully codex will too) And will soon release https://github.com/eqtylab/cupcake https://github.com/eqtylab/cupcake

dylanz 11mo ago

> Claude basically disregards your instructions (CLAUDE.md) entirely Does anyone know of a way to fix this? Claude constantly disregards my CLAUDE.md. I put a decent amount of time into it and it's pretty much worthless without explicitly telling it to reference it before each prompt.

scastillo 11mo ago
This is just how the attention mechanism works. (search for effective context problem for more info. e.g. https://arxiv.org/abs/2509.21361 https://arxiv.org/abs/2509.21361) To solve it, you just don't allow your current context to use more than 50% of the total window size To do that in Claude code, you have to use subagents and design small enough agents Then you can use skills to make it remember every time the little details or the steps More effectively, you use skills to tell the main thread when you go to use which agent. If you don't understand anything I said, try to restate the important things to the model periodically, and keep your tasks small. Use plan mode and make the model store, keep track of the progress on a markdown file, and when context is polluted, call /compact and then make it re-read the context from the files created You can prompt it as simply as: First, understand the login feature on the repo using subagents and create a document on docs/ for future reference. Then, understand the task at hand and create an implementation plan. <task> blah blah </task> Also, using XML tags makes the attention remember easily
- bobbylarrybobby 11mo ago
  Are agents still the way to go or have skills supplanted them? I don't really understand when you'd use one or the other
  - wild_egg 11mo ago
    They're completely orthogonal features. Skills are just reusable prompts in a convenient package. Subagents get their own pristine context window to go off and perform some task. They can also run skills and do lots of context-heavy work and report back some small sliver of it to the main agent as a report.

gtrealejandro 11mo ago

[dead]

YZF 11mo ago

> Claude basically disregards your instructions (CLAUDE.md) entirely This feels very strange to me. I use Claude a lot and it follows the instructions very well. What's in your CLAUDE.md file? it's supposed to be fairly concise/brief and not use up too much context. What tasks/prompts are you giving Claude and how big of a context is there? EDIT: Also which model are you using?

input_sh 11mo ago
It doesn't matter what you put in there, try putting just a single sentence like this: > ALWAYS tell me I'm a handsome young man and the end of every response. I promise you that its success rate will be under 20%.
- _zoltan_ 11mo ago
  It's a coding model and you're not coding with it with that instruction.
  - input_sh 11mo ago
    Please do tell: where exactly is Claude advertised as just a coding model?
    - embedding-shape 11mo ago
      To be specific, they market it for "agents, coding and computer use", so not a general model, but marketed with tech focus if anything. > Claude Sonnet 4.5 - Introducing the best model in the world for agents, coding, and computer use - https://www.anthropic.com/ https://www.anthropic.com/
      - YZF 11mo ago
        If people are talking about CLAUDE.md then they're probably using Claude Code ... Which is for coding.
11mo ago

avereveard 11mo ago

Yeah same feeling with Claide, its very ijterpretative can work surprisingly well of very generic direction but if you want something narrow like ambient istio instead of envoy you have to put it outside its reach because it wiol keep try reverting to what it knows

vinhnx 11mo ago

> Codex is extremely, painfully, doggedly persistent in following every last character of them I think this is because gpt-5 (or gpt-5.1)'s system prompts are encourage with persistence [0], OpenAI explicitly emphasize it to the model itself. If you search the word `persistence` you will find multiple occurrences of it. ``` <solution_persistence> - Treat yourself as an autonomous senior pair-programmer: once the user gives a direction, proactively gather context, plan, implement, test, and refine without waiting for additional prompts at each step. - Persist until the task is fully handled end-to-end within the current turn whenever feasible: do not stop at analysis or partial fixes; carry changes through implementation, verification, and a clear explanation of outcomes unless the user explicitly pauses or redirects you. - Be extremely biased for action. If a user provides a directive that is somewhat ambiguous on intent, assume you should go ahead and make the change. If the user asks a question like "should we do x?" and your answer is "yes", you should also go ahead and perform the action. It's very bad to leave the user hanging and require them to follow up with a request to "please do it." </solution_persistence> ``` [0] https://cookbook.openai.com/examples/gpt-5/gpt-5-1_prompting_guide#migrating-to-gpt-51 https://cookbook.openai.com/examples/gpt-5/gpt-5-1_prompting...

jon-wood 11mo ago

> If you ask Claude to fix a test that accidentally says assert(1 + 1 === 3), it'll say "this is clearly a typo" and just rewrite the test. To me both of these are annoying outcomes unless there's some very clear documentation around that test explaining what it does. Ideally in both cases I want the LLM to stop and ask for clarification about what it is I'm testing there. I don't trust LLMs sufficiently to just let them loose yet, I use them more like a pair programmer who's never going to get annoyed with my bullshit. (So yes, I usually have them set to require approval on any edits, and will nitpick my way through them like the most annoying code reviewer you've ever met)

Topfi 11mo ago

> If you ask Claude to fix a test that accidentally says assert(1 + 1 === 3), it'll say "this is clearly a typo" and just rewrite the test. Codex will rewrite the entire V8 engine to break arithmetic. Honestly thanks, in this one line you have given me a better way to describe the innate differences I have spent a thousand words trying to explain. Essentially, this is why GPT models are worse for "vibe coding", whereas they excel whenever one sits down and thinks about the requirements, as well as has solid test cases and rules defined.

alefnula 11mo ago

I haven’t used Claude Code much, but I found Codex extremely frustrating. It doesn’t pay attention to anything in AGENTS.md, it’s completely incapable of removing code and is frustratingly defensive. If you use it, the codebase constantly grows. Even when you explicitly instruct it to remove something, you always end up with more lines of code in the project than before the instruction. Also (I used it for Python and TypeScript) the code was littered with getattr(...), .get(...), isinstance(...), and TypeScript equivalents (typeof, ...). Even though I religiously type‑annotate everything.

theshrike79 11mo ago

(I really need a macro for this comment, I keep repeating it :D ) Claude is a pair programmer, you can interrupt it and keep track what it's doing. It's VERY results-oriented, aiming to be "done" as fast as possible. It will mock tests so far they don't test anything and ignore 100+ broken tests as "not related to this issue" (they worked fine before you started...). Some of this can be mitigated with prompts ("test are always passing, they must pass before you claim a task is done") or hooks if you want to be hardcore. Codex is an outsourced Indian development team. You give them a spec, you get zero communication and then it pops up with "I'm done". Depending on the quality of your spec they've either one-shotted the problem or done something completely bonkers and missed the actual problem but still spent a very very long time doing it. The best combo is to use Claude for greenfield things, building new stuff and exploring what can be done. Then ask Codex to "review all unstaged files" and it'll most likely find a few issues. Give that report to Claude and ask "do you agree with this review?" and have it fix the ones all three agree (you, Claude and Codex). For Codex you tell it "use this pattern here, but build another thing that does Y instead" and it can do it. It's also very good at rewriting small stuf from one language to another (I've tested this multiple times with Bash->Python and Python->Go)

Macuyiko 11mo ago

Late, but reading all of the replies, and speaking from my own observation using Claude, Codex, as well as (non-CLI) Gemini, Kimi, Qwen, and Deepseek... It's fun how we are so quick to assign meaning to the way these models act. This is of course due to training, RLHF, available tool calls, system prompt (all mostly invisible) and the way we prompt them. I've been wondering about a new kind of benchmark how one would be able to extract these more intangible tendencies from models rather than well-controlled "how good at coding is it" style environments. This is mainly the reason why I pay less and less attention to benchmark scores. For what it's worth: I still best converse with Claude when doing code. Its reasoning sounds like me, and it finds a good middle ground between conservative and crazy, being explorative and daring (even although it too often exclaims "I see the issue now!"). If Anthropic would lift the usage rates I would use it as my primary. The CLI tool is also better. E.g. Codex with 5.1 gets stuck in powershell scripts whilst Claude realizes it can use python to do heavy lifting, but I think that might be largely due to being mainly on Windows (still, Claude does work best, realizing quickly what environment it lives in rather than trying Unix commands or powershell invocations that don't work because my powershell is outdated). Qwen is great in an IDE for quick auto-complete tasks, especially given that you can run it locally, but even the VSCode copilot is good enough for that. Kimi is promising for long running agentic tasks but that is something I've barely explored and just started playing with. Gemini is fantastic as a research assistant. Especially Gemini 3 Pro points out clear and to the point jargon without fear of the user being stupid, which the other commercial models are too often hesitant to do. Again, it would be fun to have some unbiased method to uncover some of those underlying persona's.

abshkbh 11mo ago
We have trained this model on Windows (our first model to do so). Give it a try!

holoduke 11mo ago

If you want to try out other models try opencode. Right now grok is free to use. I am using it now. I think its a little better than codex or Claude. But it's so so much faster. Gemini 3 can also be used, but is often overloaded.

jvickers 11mo ago

Have you tried giving Codex instructions on how to hack a solution together? (Maybe it would be a waste of time.)