4 ms·

When you submit a /bug we get a way to see the contents of the conversation. We don't see anything else in your codebase.

by bcherny 6mo ago

murkt 6mo ago
Was there a change in Claude Code system prompt at that time that nudges Claude into simplistic thinking? Here is a gist that tries to patch the system prompt to make Claude behave better https://gist.github.com/roman01la/483d1db15043018096ac3babf5688881 https://gist.github.com/roman01la/483d1db15043018096ac3babf5... I haven’t personally tried it yet. I do certainly battle Claude quite a lot with “no I don’t want quick-n-easy wrong solution just because it’s two lines of code, I want best solution in the long run”. If the system prompt indeed prefers laziness in 5:1 ratio, that explains a lot. I will submit /bug in a few next conversations, when it occurs next.
- enigmo 6mo ago
  [dead]
- dev_l1x_be 6mo ago
  Holy sweet LLM, this gist is crazy. Why did they do this to themselves? I am going to try this at home, it might actually fix Claude.
  - murkt 6mo ago
    Remember Sonnet 3.5 and 3.7? They were happy to throw abstraction on top of abstraction on top of abstraction. Still a lot of people have “do not over-engineer, do not design for the future” and similar stuff in their CLAUDE.md files. So I think the system prompt just pushes it way too hard to “simple” direction. At least for some people. I was doing a small change in one of my projects today, and I was quite happy with “keep it stupid and hacky” approach there. And in the other project I am like “NO! WORK A LOT! DO YOUR BEST! BE HAPPY TO WORK HARD!” So it depends.
  - pbowyer 6mo ago
    Let us know if it does, because we all want it to work :)
- Avamander 6mo ago
  That Gist does explain quite a few flaws Claude has. I wonder if MEMORY.md is sufficient to counteract the prompt without patching.
  - liamsfr 6mo ago
    And if memory.md can’t and you need something quick and dirty for flat memory management, I wrote a plugin just for this. https://github.com/NominexHQ/pmm-plugin https://github.com/NominexHQ/pmm-plugin
- withinboredom 6mo ago
  Is there not a setting to change the system prompt itself? I vaguely remember seeing it in the docs.
  - matheusmoreira 6mo ago
    There is!! https://code.claude.com/docs/en/cli-reference#system-prompt-flags https://code.claude.com/docs/en/cli-reference#system-prompt-... --append-system-prompt --append-system-prompt-file --system-prompt --system-prompt-file Can this script be made to work without patching the executable?
    - withinboredom 6mo ago
      Might be worth extracting the system prompt and then patching it. TBH, that's what I was expecting when I saw the gist.
      - matheusmoreira 6mo ago
        This might be more complex than I imagined. It seems Claude Code dynamically customizes the system prompt. They also update the system prompt with every version so outright replacing it will cause us to miss out on updates. Patching is probably the best solution. https://github.com/Piebald-AI/claude-code-system-prompts https://github.com/Piebald-AI/claude-code-system-prompts https://github.com/Piebald-AI/tweakcc https://github.com/Piebald-AI/tweakcc
        withinboredom 6mo ago
        Interesting. So literally triggering any of these changes probably invalidates the cache as well…
- andersa 6mo ago
  I didn't know we could change the base system prompt of Claude Code. Just tried, and indeed it works. This changes everything! Thank you for posting this!
- naasking 6mo ago
  Very interesting. I run Claude Code in VS Code, and unfortunately there doesn't seem to be an equivalent to "cli.js", it's all bundled into the "claude.exe" I've found under the VS code extensions folder (confirmed via hex editor that the prompts are in there). Edit: tried patching with revised strings of equivalent length informed by this gist, now we'll see how it goes!
- matheusmoreira 6mo ago
  I adapted these patches into settings for the tweakcc tool. https://github.com/Piebald-AI/tweakcc https://github.com/Piebald-AI/tweakcc Pushed it to my dotfiles repository: https://github.com/matheusmoreira/.files/tree/master/~/.tweakcc/system-prompts https://github.com/matheusmoreira/.files/tree/master/~/.twea... The tweaks can be applied with npx tweakcc --apply
andoando 6mo ago
Isnt the codebase in the context window?
- frog437 6mo ago
  depending on how large your codebase is, hopefully not. At this point use something like the IX plugin to ingest codebase and track context, rather than from the LLM itself.
  - frog437 6mo ago
    This is crazy.. tokensSaved = naiveTokens - actualTokens - naiveTokens = 19.4M — what ix estimates it would have cost to answer your queries without graph intelligence (i.e., dumping full files/directories into context) - actualTokens = 4.7M — what ix's targeted, graph-aware responses actually used - tokensSaved = 14.7M — the difference
  - andoando 6mo ago
    I mean whatever part of the code that is read by the AI has to be in the content window at some point or another nSprewd throughout your sessions Id think even with a huge codebase, 90% of it is going to be there

When you submit a /bug we get a way to see the contents of the conversation. We don't see anything else in your codebase.

by bcherny 6mo ago

When you submit a /bug we get a way to see the contents of the conversation. We don't see anything else in your codebase.

murkt 6mo ago

Was there a change in Claude Code system prompt at that time that nudges Claude into simplistic thinking? Here is a gist that tries to patch the system prompt to make Claude behave better https://gist.github.com/roman01la/483d1db15043018096ac3babf5688881 https://gist.github.com/roman01la/483d1db15043018096ac3babf5... I haven’t personally tried it yet. I do certainly battle Claude quite a lot with “no I don’t want quick-n-easy wrong solution just because it’s two lines of code, I want best solution in the long run”. If the system prompt indeed prefers laziness in 5:1 ratio, that explains a lot. I will submit /bug in a few next conversations, when it occurs next.

enigmo 6mo ago
[dead]
dev_l1x_be 6mo ago
Holy sweet LLM, this gist is crazy. Why did they do this to themselves? I am going to try this at home, it might actually fix Claude.
- murkt 6mo ago
  Remember Sonnet 3.5 and 3.7? They were happy to throw abstraction on top of abstraction on top of abstraction. Still a lot of people have “do not over-engineer, do not design for the future” and similar stuff in their CLAUDE.md files. So I think the system prompt just pushes it way too hard to “simple” direction. At least for some people. I was doing a small change in one of my projects today, and I was quite happy with “keep it stupid and hacky” approach there. And in the other project I am like “NO! WORK A LOT! DO YOUR BEST! BE HAPPY TO WORK HARD!” So it depends.
- pbowyer 6mo ago
  Let us know if it does, because we all want it to work :)
Avamander 6mo ago
That Gist does explain quite a few flaws Claude has. I wonder if MEMORY.md is sufficient to counteract the prompt without patching.
- liamsfr 6mo ago
  And if memory.md can’t and you need something quick and dirty for flat memory management, I wrote a plugin just for this. https://github.com/NominexHQ/pmm-plugin https://github.com/NominexHQ/pmm-plugin
withinboredom 6mo ago
Is there not a setting to change the system prompt itself? I vaguely remember seeing it in the docs.
- matheusmoreira 6mo ago
  There is!! https://code.claude.com/docs/en/cli-reference#system-prompt-flags https://code.claude.com/docs/en/cli-reference#system-prompt-... --append-system-prompt --append-system-prompt-file --system-prompt --system-prompt-file Can this script be made to work without patching the executable?
  - withinboredom 6mo ago
    Might be worth extracting the system prompt and then patching it. TBH, that's what I was expecting when I saw the gist.
    - matheusmoreira 6mo ago
      This might be more complex than I imagined. It seems Claude Code dynamically customizes the system prompt. They also update the system prompt with every version so outright replacing it will cause us to miss out on updates. Patching is probably the best solution. https://github.com/Piebald-AI/claude-code-system-prompts https://github.com/Piebald-AI/claude-code-system-prompts https://github.com/Piebald-AI/tweakcc https://github.com/Piebald-AI/tweakcc
      - withinboredom 6mo ago
        Interesting. So literally triggering any of these changes probably invalidates the cache as well…
andersa 6mo ago
I didn't know we could change the base system prompt of Claude Code. Just tried, and indeed it works. This changes everything! Thank you for posting this!
naasking 6mo ago
Very interesting. I run Claude Code in VS Code, and unfortunately there doesn't seem to be an equivalent to "cli.js", it's all bundled into the "claude.exe" I've found under the VS code extensions folder (confirmed via hex editor that the prompts are in there). Edit: tried patching with revised strings of equivalent length informed by this gist, now we'll see how it goes!
matheusmoreira 6mo ago
I adapted these patches into settings for the tweakcc tool. https://github.com/Piebald-AI/tweakcc https://github.com/Piebald-AI/tweakcc Pushed it to my dotfiles repository: https://github.com/matheusmoreira/.files/tree/master/~/.tweakcc/system-prompts https://github.com/matheusmoreira/.files/tree/master/~/.twea... The tweaks can be applied with npx tweakcc --apply

andoando 6mo ago

Isnt the codebase in the context window?

frog437 6mo ago
depending on how large your codebase is, hopefully not. At this point use something like the IX plugin to ingest codebase and track context, rather than from the LLM itself.
- frog437 6mo ago
  This is crazy.. tokensSaved = naiveTokens - actualTokens - naiveTokens = 19.4M — what ix estimates it would have cost to answer your queries without graph intelligence (i.e., dumping full files/directories into context) - actualTokens = 4.7M — what ix's targeted, graph-aware responses actually used - tokensSaved = 14.7M — the difference
- andoando 6mo ago
  I mean whatever part of the code that is read by the AI has to be in the content window at some point or another nSprewd throughout your sessions Id think even with a huge codebase, 90% of it is going to be there