8 ms·

Memorizing session transcripts isn't useful

180 points by theahura 4mo ago

bigyabai 4mo ago
Settings > Capabilities > "Generate memory from chat history" Toggle it off and never think about it again.
chopete3 4mo ago
The author says >> We don't really write code by hand anymore. The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself. Damn, I didn't see this coming. Its first the build the intelligent builder. We will figure out what we want to build later. Edit: Before more people take it seriously. This is sarcasm. I don't wish this.
- jmalicki 4mo ago
  > We will figure out what we want to build later. Once the automator automates itself fast enough, we won't have the ability to opine what gets built. The LLM will decide. Just like right now sometimes LLMs delete tests so they pass, they could just delete humanity if humans get in their way.
- otabdeveloper4 4mo ago
  > The software world is very close to building a super intelligent senior software developer. Yeah. Two more weeks, as they say. Just need to iron out some kinks.
  - andai 4mo ago
    It's the error rate. That's what everyone found when they were trying to go Full Auto with OpenClaw in February. You can rely on it like 95% of the time but that means if you keep it running continuously the error rate rapidly approaches 100%. That's getting a little better with each release, and it might actually hit the point where you can more or less trust it indefinitely (on well defined workflows). Or at least it would, if context window permitted...
    - otabdeveloper4 4mo ago
      With an infinite context window, yes. :)
- rvz 4mo ago
  > The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself. Except Claude is more expensive than an actual senior software developer. Otherwise, why are many companies terrified of the usage bill that gets printed on the invoice? The nonsense in "tokenmaxxing" was a complete marketing scam and illusion of cheap tokens which in reality were heavily subsidized. The entire point is detecting bad code before it reaches production. [0] AI generated or not. [0] https://sketch.dev/blog/our-first-outage-from-llm-written-code https://sketch.dev/blog/our-first-outage-from-llm-written-co...
beepbooptheory 4mo ago
There has been this slow transition inside me, as someone who likes to not touch the AI as much as possible, where I've gone from skeptical and argumentative about it all to starting to just feel sad for all the Claude et al heads. Like, this is such a ridiculous house of cards you have to deal with all the time, which isn't even directly concerning the task at hand, presumably. Like you're cooking yourself a meal but its just nuking a burrito and then still somehow needing to wash the dishes for an hour. Not that this isolated article is super damning or anything, but the accumulated set of all these reports has left me only empathetic, I think, of these other devs. Like, I just want to tell them, "it can be ok, it doesn't need to be like this.."
- andai 4mo ago
  I've been having a very nice time with Fable. I cooked up an Anki clone in like half an hour, with tech it's not familiar with. Nothing too ground breaking, but I was very pleased! I think Opus might be on similar level for most of what I'm doing, but I haven't used it much recently, so I can't remember the difference. So I guess I'll find out on the 7th when they pull the plug again! (Free-ish trial of Fable ending.) That being said, I tried using other frontier models to help with a Pong clone the other day and they were introducing new bugs at approximately the same rate as they were fixing it. On Pong!! I found that amusing because I couldn't think of a simpler game, so it didn't inspire confidence. Fable's doing just fine on an online multiplayer game though. I have no idea how that works. (Maybe it would fail Pong too?? I haven't tested that!)
general_reveal 4mo ago
Isn’t this just a form of the bitter lesson? Our attempts to make engineered context and agents will simply be made obsolete with bigger and better models. Those transcripts are probably extremely useful for lesser capable models, and near unnecessary for frontier ones, maybe?
- andai 4mo ago
  Yeah, the question is whether this applies to all of context management. I've been using a custom harness based on https://minimal-agent.com/ https://minimal-agent.com/ (itself based on swe-mini-agent), which is like 50 lines for the core logic. Bash is all you need. For small tasks, I find it's about 8x faster (and uses 8x fewer tokens) than the standard harness for each model. For bigger tasks I haven't tested it much. It seems to work too but I think they're a bit less focused and productive in that case. It could be that those big harnesses' 20k token system prompts are doing something important with regard to steering software development workflows. (e.g. I heard Fable has a custom system prompt in Claude Code which might explain its markedly more proactive behavior.) So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release (since they're fine tuned on mostly non stupid behavior and need less hand holding).
  - sdesol 4mo ago
    > So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release I can't see how it would diminish unless you are literally working on public domain stuff. Unless stuffing context becomes cost effective and will not affect AI reasoning (this will be much harder), I don't see why context engineering is here to stay until we have close to AGI.
  - irthomasthomas 4mo ago
    In think in all cases where I've seen it compared CC performed worse than a minimal harness.
- HarHarVeryFunny 4mo ago
  I don't think so - I think we'll find that to build a brain you need more built-in structure and biases, not less. Bear in mind that brain architecture is learnt too - just over a much longer timescale than an individual lifetime.
wongarsu 4mo ago
I agree with the take not to bother with a sophisticated memory system. Anything worth remembering should be in docs, guides, source comments, commit messages or tickets. You don't need another layer, every conceivable granularity is already covered by existing best practices
- throwup238 4mo ago
  Especially a layer that is largely out of band in a project (i.e. ~/.claude/…). In any project where I’ve needed memory I just add a line to AGENTS.md telling it to use MEMORY.md to save memories or STATUS.md to track progress.
  - andai 4mo ago
    I've been enjoying having a little todo file the agent updates as it goes along, because then I can keep track of progress without scrolling through aeons of "Combobulating..." Also if context runs out you can just do "cat todo.md | agent" and you're off to the races again.
    - mock-possum 4mo ago
      Yep all my projects start with a PLAN.md at the root, and that acts as the ‘save file’ recording our progress over time. My session always ends with updating the plan file with what’s been done, and the next session always begins, as you suggest, with consuming the current state of the plan doc.
- sdesol 4mo ago
  > You don't need another layer I do think we need another layer, but it should be a routing layer. I am finalizing my pi-brains extension for Pi (https://github.com/earendil-works/pi https://github.com/earendil-works/pi) which does this: https://github.com/gitsense/pi-brains https://github.com/gitsense/pi-brains Right now "humans" need to define the routing rules for how to access information, but I will support what I call "knowledge agents" that can monitor conversations to inject context when needed.
dofm 4mo ago
Blog posts like this just blow me away. > I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself. > […] > We don't really write code by hand anymore. Honestly, isn't this just influencer spam? What possible value is there in reading about people who used to have products, but no longer write their own code, complaining about the inscrutable prediction machine they have handed that job and their livelihoods to? Like, if you have complaints about the thing, perhaps you should address them to your supplier directly. None of your readers can help, and nobody's magic folk solution to your problem is better than yours. And there are so many of these sorts of posts. Are we not entirely cooked? (I think I have concluded that if people writing about AI aren't writing about interesting things they have achieved with small, local LLMs — which for clarity I am fully interested in reading - then I'm done reading. This whole blogging-about-cloud-AI genre is just weird and irresponsible now)
- fortyseven 4mo ago
  [flagged]
  - dofm 4mo ago
    I'm not the Blog Police, I'm a very naughty boy. I have opinions people apparently don't like, for no subscriber money.
- LPisGood 4mo ago
  I have to ask: do you still write a lot of code yourself? I and most people I know do not.
  - walt_grata 4mo ago
    I write code by hand every day. I do the main part of the feature implementation myself and leave comments for the code i want the agent to write. I have some skills and a command that sets the stage to get the agent to fill in the rest
  - 4mo ago
semiquaver 4mo ago
Strongly agree here. claude-code’s memory system is occasionally useful but much more often harmful, pulling in obsolete info that muddies the waters about current tasks. I’ve frequently seen Claude’s own memories severely mislead it. My guess is that has something to do with the training process leaving models unable to differentiate between “what’s happening now” and “what happened before”. Perhaps if making inferences from memories was actually part of the training process things would be different but my sense is that as an inference-time-only feature this just gets the models confused.
- pennomi 4mo ago
  Humans make memories constantly, but they also forget things that are no longer relevant. Until Claude can do that, it means the LLM will have an ever-increasing, ever more fragmented context. And LLMs are NOT intelligent enough to survive even mild context poisoning.
- dwaltrip 4mo ago
  I'd add on that the models have a very poor sense of time and the complex changes in world state that occur as time passes. Training with memory is an interesting idea...
- celrod 4mo ago
  When claude goes down a wrong path, I tend to clear context and write a new prompt that helps guide it down the correct path. Whatever thinking or context that led it there has inertia and tends to be sticky, otherwise. Pretty annoying when it brings those up again later from memory...
oefrha 4mo ago
I have this in my global CLAUDE.md after being annoyed by all the random crap memories. > Don't start generating an auto-memory entry before asking me. Ask first, write only if I confirm — no speculative drafting. No more crap after this. Incidentally I don’t recall Opus 4.8 asking me once in the past few weeks. Older models did ask semi-frequently.
aranw 4mo ago
t once had to tell claude 3-4 times to stop assuming the state of a system was the way it kept iterating it was cause it was in it's memory. I repeatably told it to otherwise and it just never updated it's memory and instead kept referencing it's memory about the state of a particular system
- syntheticcdo 4mo ago
  Did you try to delete the memory yourself?
- andai 4mo ago
  In my harness I have all the code auto injected at startup (doing mostly very small codebases). I found that every model will still manually check every file/function, they immediately assume that anything in context is stale. That's sensible because often the user edits stuff while they're running. What it does is save it from having to grep blindly about the codebase. But I think I'd get roughly the same benefit by just dumping the function headers then.
- ErroneousBosh 4mo ago
  Yeah but you also know humans that do that too, right? I know I do.
zahirbmirza 4mo ago
Even with memory off this occurs within a conversation. It is like an annoying friend, who remembers something from a past conversation, that you have grown and developed from, but they still want to hold it against you.
trjordan 4mo ago
It's because it mostly doesn't matter what you are trying to get the code to do. What matters is what the code does. Session logs can absolutely be useful, but not when building further. It's just that that the place they slot in is during validation. You know, that place between the markdown plan and CI passing, where there's 800 new lines of code and it all seems sort of fine when you click around? Session logs can show you what sort of manual validation happened. CI will run the tests you had, and the code will show you what new unit tests were added, but session logs can show you that the agent drove the app with Playwright, or that the agent read and considered the prod config as well as the dev config. Nothing bulletproof, but not every piece of validation work merits a test in the repo that lives forever. We've gotten a lot of mileage out of re-analyzing the sessions, figuring out where the agent made decisions without asking, and forcing the agent to consider validation for those decisions. That's the sort of thing that's hard to dictate up front but easy to highlight with the session logs.
Fabricio20 4mo ago
I specifically disabled claude memory in a project because it kept writing down thigns to memory that didn't need to be in memory, including severly wrong statements that then would confuse it later. At some point it got re-enabled automatically which had me ask claude itself to "turn it the fuck off" by which it promptly figured out that both ("autoMemoryEnabled": false, "autoDreamEnabled": false) are necessary and need to be at the user home settings, not in a project override (which is what I had with the original setup that eventually got ignored by a CC update). I agree with other commenters here, if anything is worth being rememebered, it will be in code comments, git commit messages, CLAUDE.md or other formal documentation. The auto memory system just causes confusion and leaves stale and outdated information written down. Its an interesting thought experiment as well, I originally thought that having the model write down memory files by itself would be a nice addition, but after playing around with it, it became clear to me that good as an idea turns out bad in practice because the model can't correctly gauge what deserves being stored as a memory.
- andai 4mo ago
  > "turn it the fuck off" -> "autoDreamEnabled": false So you told it don't go the fuck to sleep ;)
- estetlinus 4mo ago
  Ugh, agentic _dreaming_. Why on earth would I want that?
grimcompanion 4mo ago
> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself. This is infuriatingly common wrt talking/writing about how to use AI effectively. All of the "this is how you write an AGENTS.md" and "you need to talk to it like X to optimize it". Like sure, you can believe that as much as you want but unless you provide some evidence you can keep your shitty CLAUDE.md to yourself and don't pollute the whole company's git repo, thanks.
- estetlinus 4mo ago
  When nobody actually knows (how to write a CLAUDE.md), everyone’s an expert. Infuriating, indeed. Even more so when people vibe code those files without proofreading.
mastax 4mo ago
I found that if you allow any low value things into memory, Claude will notice that established pattern and start trying to add low value memories at an ever increasing pace.
- jvuygbbkuurx 4mo ago
  Explains why it made so many memories on my work machine, but never made one on my personal one. Maybe the project size also affects it.
charcircuit 4mo ago
>We have found zero performance benefit on SWE tasks when agents have search access to their previous transcript sessions I refuse to believe this is true. The ability for an agent to find information from before a compaction is incredibly useful. At compaction time it's impossible to know what exactly may be still needed.
- theahura 4mo ago
  With the million-context-window models we never hit compaction, observed over hundreds of sessions. What are you doing that has you hitting compaction regularly?
  - charcircuit 4mo ago
    For me logs can chew through a lot of tokens. And when the agent is trying a bunch of different experiments and then it may need to refer to what happened previously. Million context models also are still not effective for the entire context size.
- deleted 4mo ago
  [deleted]
linsomniac 4mo ago
I like the memory system, in general. For reference I'm using mostly Opus 4.8 + Max effort. It will often pull things out of memory that are relevant. Like I'll ask it to come up with a few options I should consider for, say, a self-hosted OIDC provider and it'll say things like "Considering the size of your operations team, this might be a better fit because of X and Y". Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that. It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.
- fc417fc802 4mo ago
  It seems like a prerequisite is a certain level of world model and associated reasoning ability. Your examples are entirely dependent on the past context being relevant to the current situation. That's particularly tricky if you regularly ask about hypotheticals or problems that you're assisting someone else with. A human would probably ask clarifying questions such as "is this for the operations team at X? are they still size Y?" and "is this app proxied like the others you mentioned in the past?" rather than assuming. There's also a noticable hierarchy to such context that needs to be correctly modeled - you could for example be involved with multiple teams of different sizes that are subject to different rules which is something a human would understand naturally.
  - linsomniac 4mo ago
    >A human would probably ask clarifying questions You say that, and that's a possibility, and maybe in your experience that is true. But I'd say I've found actual humans will frequently use existing internal models of something unless they have a reason to believe something has changed to invalidate that. We even have a well worn joke about what happens when you assume. >you could for example be involved with multiple teams of different sizes that are subject to different rules Just to clarify, my experience with the memory of these tools doesn't provide any data (for or against) how well/poorly the memory may work with someone who is involved with multiple teams.
estetlinus 4mo ago
I remember when OpenAI announced ChatGPT now will remember stuff between sessions. Oh, you mean find random trivia about me and copy paste it between prompts without out my explicit consent. ”compare these three cars. Oh btw I am a data engineer, and my moms maiden name is Joana, and I am allergic to bad poetry. And code should be DRY, I prefer SQL over Python and what’s the most poisonous flower in Scandinavia?”. I’ve had so much wierd output because context is ”””memorized””” and bleeding into completely unrelated projects and conversations. It’s the first feature I turn off.
- hmry 4mo ago
  I have to imagine these types of features are for people who treat chatgpt like their friend/therapist/girlfriend/assistant/... instead of people who use it to answer questions
  - ffsm8 4mo ago
    You shouldn't really use it to answer questions either though, because then you won't know how wrong it is when it's hallucinating... I mean I know the draw, it's just so easy to get suckered into it... After all, it's usually mostly correct! And since the Advent of llms searching via Google has gotten essentially impossible with 1000 slop articles per one genuine article... But the danger is real, even with eg fable
abratabia 4mo ago
[flagged]
saagarjha 4mo ago
I mean, it’s pretty clear the people who work on Claude Code aren’t actually looking at what they’re implementing. The thought behind this feature seems like it goes nowhere beyond “oh wouldn’t it be nice if Claude could remember things about you? Ok Claude go implement this” and nobody bothered to see if it was useful or helpful.
bluegatty 4mo ago
There's a lot of valuable information in there, its' too noisy.
zackify 4mo ago
This is an annoying problem. It keeps making fake assumptions just because of hypothetical questions I've asked in the past. It'll assume I own a datacenter and have lots of gpus just because I asked to research things.
dijksterhuis 4mo ago
non-deterministic system behaves non deterministically. in other news, water is wet.
consumer451 4mo ago
[flagged]
adverbly 4mo ago
I'm not sure how well this take will age. Its certainly true at the moment, but give it 10 years and we might have systems that are much cheaper and much better at context management than they are now. (Apologies to anyone who is under the impression that we were very likely going to be at the singularity in 10 years time. Possible != very likely)
- FuckButtons 4mo ago
  Sure, but it’s equally likely that we hit a point where scaling becomes economically unviable because we can’t come up with enough algorithmic improvements to break free of the tyranny of log linear scaling. (I’m not sure how many 2x in token cost people would be willing to pay)
8note 4mo ago
Thats interesting, but what was the methodology? is the conclusion really that its just more important to create proper artifacts from any tricks that got the llm to understand the code better? is the tool for searching the history just bad?
shepherdjerred 4mo ago
Strong disagree on this. I have Claude/Codex keep logs [1]. It's just prompted in my AGENTS.md [0]. > Every session must produce one of: a session log OR a plan, and end with a written summary appended to it. Default to a log; reserve plans for substantive design work. It's incredibly valuable. For example today I started a few sessions off like this: - What's the status of my work on Renovate? - I was recently working on X, find that - Did we fix the issue with backups? What are the next steps? - This bug came up again. Didn't we fix it already? [0]: https://github.com/shepherdjerred/monorepo/blob/main/AGENTS.md#documentation-discipline--per-session https://github.com/shepherdjerred/monorepo/blob/main/AGENTS.... [1]: https://github.com/shepherdjerred/monorepo/tree/main/packages/docs/logs https://github.com/shepherdjerred/monorepo/tree/main/package...
lukeschlather 4mo ago
At the core this is a hardware problem. 1M tokens is simply not enough context to understand a codebase the way a human would understand it. Being able to selectively forget is potentially a very valuable power, but right now it's a substitute for a human's ability to remember the rough shape of something, decide it's uninteresting, and remember that it is uninteresting. They talk about memory only being useful when guided by a human, I think the proper solution is deeper than that, it probably involves feeding the entire codebase and every agent session into a finetuning of the model, though at that point you might want some guidance to avoid feeding certain sessions into the model. Or maybe not, maybe the bitter lesson applies.
- kolinko 4mo ago
  1M context - at least with most of the projects I ever worked with, 1M, or even 100k would be enough to explain in broad strokes the class/project/deployment structure, and a window of 200-500k to explain the specific issue at hand.
greenoracle9 4mo ago
[flagged]
PeterStuer 4mo ago
I must admit lingering long since retired 'memories' are currently one of the biggest pitfalls of the setup. Wiping all 'memory.md' often leads to better sustain.
SaltyAstronaut 4mo ago
Those small, random items that pop-up later on in conversation actually make the experience feel better. But that's just my own personal experience.
exabrial 4mo ago
Hilariously, I'm working on a Claude Desktop replacement that does all of things. It's the best parts of Claude Desktop, Code, Cowork, and MCP connectors, but uses a client/server design. It's written in JavaFx so it's lightweight, fast, cross platform, and not another damned electron app. Ideal outcome is this turns into a startup. I think there's a real need for team-oriented AI to avoid siloing of knowledge.
scosman 4mo ago
The top of my ~/.claude/CLAUDE.md: > Don't turn a one-off or area-specific comment into a durable memory without my explicit confirmation. You have a history of over-indexing on one-offs, and those memories end up getting cited to override well-tuned skills.
- dofm 4mo ago
  Does that second sentence provide value beyond getting that off your chest? ;-) (Semi-serious question)
  - scosman 4mo ago
    heh, who knows. I often give agents the motivation, which I generally helpful. Have not bothered to measure this case.
tracyhenry 4mo ago
The only times I found the memory feature useful are in "projects" I created myself. In a project my questions are usually revolved around the same topic. Having context carried across threads actually make a lot of sense. In the general mode where I'm expecting models to be *stateless*, having memory is very annoying.
wrs 4mo ago
My current technique, which seems to have improved maintainability, is to guide Claude to write commit messages specifically focused on "why this was done", "what changed in the theory of operation", and "what changed in the code". Then just reviewing the commits for a file or dir gives it a ton of useful context distilled from the sessions that produced them. Also, making a docs dir with concise .md files explaining the theory of operation and updating them with every commit.
chickensong 4mo ago
I don't do anything with full session transcripts, but I find value when Claude writes a memory. I don't actually want Claude to have those memories, but they often point to gaps in my harness. I'll occasionally sweep the memories, pick out what should go into the harness or CLAUDE.md, then delete the memories.

180 points by theahura 4mo ago

Settings > Capabilities > "Generate memory from chat history" Toggle it off and never think about it again.

The author says >> We don't really write code by hand anymore. The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself. Damn, I didn't see this coming. Its first the build the intelligent builder. We will figure out what we want to build later. Edit: Before more people take it seriously. This is sarcasm. I don't wish this.

jmalicki 4mo ago
> We will figure out what we want to build later. Once the automator automates itself fast enough, we won't have the ability to opine what gets built. The LLM will decide. Just like right now sometimes LLMs delete tests so they pass, they could just delete humanity if humans get in their way.
otabdeveloper4 4mo ago
> The software world is very close to building a super intelligent senior software developer. Yeah. Two more weeks, as they say. Just need to iron out some kinks.
- andai 4mo ago
  It's the error rate. That's what everyone found when they were trying to go Full Auto with OpenClaw in February. You can rely on it like 95% of the time but that means if you keep it running continuously the error rate rapidly approaches 100%. That's getting a little better with each release, and it might actually hit the point where you can more or less trust it indefinitely (on well defined workflows). Or at least it would, if context window permitted...
  - otabdeveloper4 4mo ago
    With an infinite context window, yes. :)
rvz 4mo ago
> The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself. Except Claude is more expensive than an actual senior software developer. Otherwise, why are many companies terrified of the usage bill that gets printed on the invoice? The nonsense in "tokenmaxxing" was a complete marketing scam and illusion of cheap tokens which in reality were heavily subsidized. The entire point is detecting bad code before it reaches production. [0] AI generated or not. [0] https://sketch.dev/blog/our-first-outage-from-llm-written-code https://sketch.dev/blog/our-first-outage-from-llm-written-co...

beepbooptheory 4mo ago

There has been this slow transition inside me, as someone who likes to not touch the AI as much as possible, where I've gone from skeptical and argumentative about it all to starting to just feel sad for all the Claude et al heads. Like, this is such a ridiculous house of cards you have to deal with all the time, which isn't even directly concerning the task at hand, presumably. Like you're cooking yourself a meal but its just nuking a burrito and then still somehow needing to wash the dishes for an hour. Not that this isolated article is super damning or anything, but the accumulated set of all these reports has left me only empathetic, I think, of these other devs. Like, I just want to tell them, "it can be ok, it doesn't need to be like this.."

andai 4mo ago
I've been having a very nice time with Fable. I cooked up an Anki clone in like half an hour, with tech it's not familiar with. Nothing too ground breaking, but I was very pleased! I think Opus might be on similar level for most of what I'm doing, but I haven't used it much recently, so I can't remember the difference. So I guess I'll find out on the 7th when they pull the plug again! (Free-ish trial of Fable ending.) That being said, I tried using other frontier models to help with a Pong clone the other day and they were introducing new bugs at approximately the same rate as they were fixing it. On Pong!! I found that amusing because I couldn't think of a simpler game, so it didn't inspire confidence. Fable's doing just fine on an online multiplayer game though. I have no idea how that works. (Maybe it would fail Pong too?? I haven't tested that!)

general_reveal 4mo ago

Isn’t this just a form of the bitter lesson? Our attempts to make engineered context and agents will simply be made obsolete with bigger and better models. Those transcripts are probably extremely useful for lesser capable models, and near unnecessary for frontier ones, maybe?

andai 4mo ago
Yeah, the question is whether this applies to all of context management. I've been using a custom harness based on https://minimal-agent.com/ https://minimal-agent.com/ (itself based on swe-mini-agent), which is like 50 lines for the core logic. Bash is all you need. For small tasks, I find it's about 8x faster (and uses 8x fewer tokens) than the standard harness for each model. For bigger tasks I haven't tested it much. It seems to work too but I think they're a bit less focused and productive in that case. It could be that those big harnesses' 20k token system prompts are doing something important with regard to steering software development workflows. (e.g. I heard Fable has a custom system prompt in Claude Code which might explain its markedly more proactive behavior.) So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release (since they're fine tuned on mostly non stupid behavior and need less hand holding).
- sdesol 4mo ago
  > So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release I can't see how it would diminish unless you are literally working on public domain stuff. Unless stuffing context becomes cost effective and will not affect AI reasoning (this will be much harder), I don't see why context engineering is here to stay until we have close to AGI.
- irthomasthomas 4mo ago
  In think in all cases where I've seen it compared CC performed worse than a minimal harness.
HarHarVeryFunny 4mo ago
I don't think so - I think we'll find that to build a brain you need more built-in structure and biases, not less. Bear in mind that brain architecture is learnt too - just over a much longer timescale than an individual lifetime.

wongarsu 4mo ago

I agree with the take not to bother with a sophisticated memory system. Anything worth remembering should be in docs, guides, source comments, commit messages or tickets. You don't need another layer, every conceivable granularity is already covered by existing best practices

throwup238 4mo ago
Especially a layer that is largely out of band in a project (i.e. ~/.claude/…). In any project where I’ve needed memory I just add a line to AGENTS.md telling it to use MEMORY.md to save memories or STATUS.md to track progress.
- andai 4mo ago
  I've been enjoying having a little todo file the agent updates as it goes along, because then I can keep track of progress without scrolling through aeons of "Combobulating..." Also if context runs out you can just do "cat todo.md | agent" and you're off to the races again.
  - mock-possum 4mo ago
    Yep all my projects start with a PLAN.md at the root, and that acts as the ‘save file’ recording our progress over time. My session always ends with updating the plan file with what’s been done, and the next session always begins, as you suggest, with consuming the current state of the plan doc.
sdesol 4mo ago
> You don't need another layer I do think we need another layer, but it should be a routing layer. I am finalizing my pi-brains extension for Pi (https://github.com/earendil-works/pi https://github.com/earendil-works/pi) which does this: https://github.com/gitsense/pi-brains https://github.com/gitsense/pi-brains Right now "humans" need to define the routing rules for how to access information, but I will support what I call "knowledge agents" that can monitor conversations to inject context when needed.

dofm 4mo ago

Blog posts like this just blow me away. > I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself. > […] > We don't really write code by hand anymore. Honestly, isn't this just influencer spam? What possible value is there in reading about people who used to have products, but no longer write their own code, complaining about the inscrutable prediction machine they have handed that job and their livelihoods to? Like, if you have complaints about the thing, perhaps you should address them to your supplier directly. None of your readers can help, and nobody's magic folk solution to your problem is better than yours. And there are so many of these sorts of posts. Are we not entirely cooked? (I think I have concluded that if people writing about AI aren't writing about interesting things they have achieved with small, local LLMs — which for clarity I am fully interested in reading - then I'm done reading. This whole blogging-about-cloud-AI genre is just weird and irresponsible now)

fortyseven 4mo ago
[flagged]
- dofm 4mo ago
  I'm not the Blog Police, I'm a very naughty boy. I have opinions people apparently don't like, for no subscriber money.
LPisGood 4mo ago
I have to ask: do you still write a lot of code yourself? I and most people I know do not.
- walt_grata 4mo ago
  I write code by hand every day. I do the main part of the feature implementation myself and leave comments for the code i want the agent to write. I have some skills and a command that sets the stage to get the agent to fill in the rest
- 4mo ago

semiquaver 4mo ago

Strongly agree here. claude-code’s memory system is occasionally useful but much more often harmful, pulling in obsolete info that muddies the waters about current tasks. I’ve frequently seen Claude’s own memories severely mislead it. My guess is that has something to do with the training process leaving models unable to differentiate between “what’s happening now” and “what happened before”. Perhaps if making inferences from memories was actually part of the training process things would be different but my sense is that as an inference-time-only feature this just gets the models confused.

pennomi 4mo ago
Humans make memories constantly, but they also forget things that are no longer relevant. Until Claude can do that, it means the LLM will have an ever-increasing, ever more fragmented context. And LLMs are NOT intelligent enough to survive even mild context poisoning.
dwaltrip 4mo ago
I'd add on that the models have a very poor sense of time and the complex changes in world state that occur as time passes. Training with memory is an interesting idea...
celrod 4mo ago
When claude goes down a wrong path, I tend to clear context and write a new prompt that helps guide it down the correct path. Whatever thinking or context that led it there has inertia and tends to be sticky, otherwise. Pretty annoying when it brings those up again later from memory...

oefrha 4mo ago

I have this in my global CLAUDE.md after being annoyed by all the random crap memories. > Don't start generating an auto-memory entry before asking me. Ask first, write only if I confirm — no speculative drafting. No more crap after this. Incidentally I don’t recall Opus 4.8 asking me once in the past few weeks. Older models did ask semi-frequently.

aranw 4mo ago

t once had to tell claude 3-4 times to stop assuming the state of a system was the way it kept iterating it was cause it was in it's memory. I repeatably told it to otherwise and it just never updated it's memory and instead kept referencing it's memory about the state of a particular system

syntheticcdo 4mo ago
Did you try to delete the memory yourself?
andai 4mo ago
In my harness I have all the code auto injected at startup (doing mostly very small codebases). I found that every model will still manually check every file/function, they immediately assume that anything in context is stale. That's sensible because often the user edits stuff while they're running. What it does is save it from having to grep blindly about the codebase. But I think I'd get roughly the same benefit by just dumping the function headers then.
ErroneousBosh 4mo ago
Yeah but you also know humans that do that too, right? I know I do.

zahirbmirza 4mo ago

Even with memory off this occurs within a conversation. It is like an annoying friend, who remembers something from a past conversation, that you have grown and developed from, but they still want to hold it against you.

trjordan 4mo ago

It's because it mostly doesn't matter what you are trying to get the code to do. What matters is what the code does. Session logs can absolutely be useful, but not when building further. It's just that that the place they slot in is during validation. You know, that place between the markdown plan and CI passing, where there's 800 new lines of code and it all seems sort of fine when you click around? Session logs can show you what sort of manual validation happened. CI will run the tests you had, and the code will show you what new unit tests were added, but session logs can show you that the agent drove the app with Playwright, or that the agent read and considered the prod config as well as the dev config. Nothing bulletproof, but not every piece of validation work merits a test in the repo that lives forever. We've gotten a lot of mileage out of re-analyzing the sessions, figuring out where the agent made decisions without asking, and forcing the agent to consider validation for those decisions. That's the sort of thing that's hard to dictate up front but easy to highlight with the session logs.

Fabricio20 4mo ago

I specifically disabled claude memory in a project because it kept writing down thigns to memory that didn't need to be in memory, including severly wrong statements that then would confuse it later. At some point it got re-enabled automatically which had me ask claude itself to "turn it the fuck off" by which it promptly figured out that both ("autoMemoryEnabled": false, "autoDreamEnabled": false) are necessary and need to be at the user home settings, not in a project override (which is what I had with the original setup that eventually got ignored by a CC update). I agree with other commenters here, if anything is worth being rememebered, it will be in code comments, git commit messages, CLAUDE.md or other formal documentation. The auto memory system just causes confusion and leaves stale and outdated information written down. Its an interesting thought experiment as well, I originally thought that having the model write down memory files by itself would be a nice addition, but after playing around with it, it became clear to me that good as an idea turns out bad in practice because the model can't correctly gauge what deserves being stored as a memory.

andai 4mo ago
> "turn it the fuck off" -> "autoDreamEnabled": false So you told it don't go the fuck to sleep ;)
estetlinus 4mo ago
Ugh, agentic _dreaming_. Why on earth would I want that?

grimcompanion 4mo ago

> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself. This is infuriatingly common wrt talking/writing about how to use AI effectively. All of the "this is how you write an AGENTS.md" and "you need to talk to it like X to optimize it". Like sure, you can believe that as much as you want but unless you provide some evidence you can keep your shitty CLAUDE.md to yourself and don't pollute the whole company's git repo, thanks.

estetlinus 4mo ago
When nobody actually knows (how to write a CLAUDE.md), everyone’s an expert. Infuriating, indeed. Even more so when people vibe code those files without proofreading.

mastax 4mo ago

I found that if you allow any low value things into memory, Claude will notice that established pattern and start trying to add low value memories at an ever increasing pace.

jvuygbbkuurx 4mo ago
Explains why it made so many memories on my work machine, but never made one on my personal one. Maybe the project size also affects it.

charcircuit 4mo ago

>We have found zero performance benefit on SWE tasks when agents have search access to their previous transcript sessions I refuse to believe this is true. The ability for an agent to find information from before a compaction is incredibly useful. At compaction time it's impossible to know what exactly may be still needed.

theahura 4mo ago
With the million-context-window models we never hit compaction, observed over hundreds of sessions. What are you doing that has you hitting compaction regularly?
- charcircuit 4mo ago
  For me logs can chew through a lot of tokens. And when the agent is trying a bunch of different experiments and then it may need to refer to what happened previously. Million context models also are still not effective for the entire context size.
deleted 4mo ago
[deleted]

linsomniac 4mo ago

I like the memory system, in general. For reference I'm using mostly Opus 4.8 + Max effort. It will often pull things out of memory that are relevant. Like I'll ask it to come up with a few options I should consider for, say, a self-hosted OIDC provider and it'll say things like "Considering the size of your operations team, this might be a better fit because of X and Y". Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that. It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.

fc417fc802 4mo ago
It seems like a prerequisite is a certain level of world model and associated reasoning ability. Your examples are entirely dependent on the past context being relevant to the current situation. That's particularly tricky if you regularly ask about hypotheticals or problems that you're assisting someone else with. A human would probably ask clarifying questions such as "is this for the operations team at X? are they still size Y?" and "is this app proxied like the others you mentioned in the past?" rather than assuming. There's also a noticable hierarchy to such context that needs to be correctly modeled - you could for example be involved with multiple teams of different sizes that are subject to different rules which is something a human would understand naturally.
- linsomniac 4mo ago
  >A human would probably ask clarifying questions You say that, and that's a possibility, and maybe in your experience that is true. But I'd say I've found actual humans will frequently use existing internal models of something unless they have a reason to believe something has changed to invalidate that. We even have a well worn joke about what happens when you assume. >you could for example be involved with multiple teams of different sizes that are subject to different rules Just to clarify, my experience with the memory of these tools doesn't provide any data (for or against) how well/poorly the memory may work with someone who is involved with multiple teams.

estetlinus 4mo ago

I remember when OpenAI announced ChatGPT now will remember stuff between sessions. Oh, you mean find random trivia about me and copy paste it between prompts without out my explicit consent. ”compare these three cars. Oh btw I am a data engineer, and my moms maiden name is Joana, and I am allergic to bad poetry. And code should be DRY, I prefer SQL over Python and what’s the most poisonous flower in Scandinavia?”. I’ve had so much wierd output because context is ”””memorized””” and bleeding into completely unrelated projects and conversations. It’s the first feature I turn off.

hmry 4mo ago
I have to imagine these types of features are for people who treat chatgpt like their friend/therapist/girlfriend/assistant/... instead of people who use it to answer questions
- ffsm8 4mo ago
  You shouldn't really use it to answer questions either though, because then you won't know how wrong it is when it's hallucinating... I mean I know the draw, it's just so easy to get suckered into it... After all, it's usually mostly correct! And since the Advent of llms searching via Google has gotten essentially impossible with 1000 slop articles per one genuine article... But the danger is real, even with eg fable

abratabia 4mo ago

[flagged]

saagarjha 4mo ago

I mean, it’s pretty clear the people who work on Claude Code aren’t actually looking at what they’re implementing. The thought behind this feature seems like it goes nowhere beyond “oh wouldn’t it be nice if Claude could remember things about you? Ok Claude go implement this” and nobody bothered to see if it was useful or helpful.

bluegatty 4mo ago

There's a lot of valuable information in there, its' too noisy.

zackify 4mo ago

This is an annoying problem. It keeps making fake assumptions just because of hypothetical questions I've asked in the past. It'll assume I own a datacenter and have lots of gpus just because I asked to research things.

dijksterhuis 4mo ago

non-deterministic system behaves non deterministically. in other news, water is wet.

consumer451 4mo ago

[flagged]

adverbly 4mo ago

I'm not sure how well this take will age. Its certainly true at the moment, but give it 10 years and we might have systems that are much cheaper and much better at context management than they are now. (Apologies to anyone who is under the impression that we were very likely going to be at the singularity in 10 years time. Possible != very likely)

FuckButtons 4mo ago
Sure, but it’s equally likely that we hit a point where scaling becomes economically unviable because we can’t come up with enough algorithmic improvements to break free of the tyranny of log linear scaling. (I’m not sure how many 2x in token cost people would be willing to pay)

8note 4mo ago

Thats interesting, but what was the methodology? is the conclusion really that its just more important to create proper artifacts from any tricks that got the llm to understand the code better? is the tool for searching the history just bad?

shepherdjerred 4mo ago

Strong disagree on this. I have Claude/Codex keep logs [1]. It's just prompted in my AGENTS.md [0]. > Every session must produce one of: a session log OR a plan, and end with a written summary appended to it. Default to a log; reserve plans for substantive design work. It's incredibly valuable. For example today I started a few sessions off like this: - What's the status of my work on Renovate? - I was recently working on X, find that - Did we fix the issue with backups? What are the next steps? - This bug came up again. Didn't we fix it already? [0]: https://github.com/shepherdjerred/monorepo/blob/main/AGENTS.md#documentation-discipline--per-session https://github.com/shepherdjerred/monorepo/blob/main/AGENTS.... [1]: https://github.com/shepherdjerred/monorepo/tree/main/packages/docs/logs https://github.com/shepherdjerred/monorepo/tree/main/package...

lukeschlather 4mo ago

At the core this is a hardware problem. 1M tokens is simply not enough context to understand a codebase the way a human would understand it. Being able to selectively forget is potentially a very valuable power, but right now it's a substitute for a human's ability to remember the rough shape of something, decide it's uninteresting, and remember that it is uninteresting. They talk about memory only being useful when guided by a human, I think the proper solution is deeper than that, it probably involves feeding the entire codebase and every agent session into a finetuning of the model, though at that point you might want some guidance to avoid feeding certain sessions into the model. Or maybe not, maybe the bitter lesson applies.

kolinko 4mo ago
1M context - at least with most of the projects I ever worked with, 1M, or even 100k would be enough to explain in broad strokes the class/project/deployment structure, and a window of 200-500k to explain the specific issue at hand.

greenoracle9 4mo ago

[flagged]

PeterStuer 4mo ago

I must admit lingering long since retired 'memories' are currently one of the biggest pitfalls of the setup. Wiping all 'memory.md' often leads to better sustain.

SaltyAstronaut 4mo ago

Those small, random items that pop-up later on in conversation actually make the experience feel better. But that's just my own personal experience.

exabrial 4mo ago

Hilariously, I'm working on a Claude Desktop replacement that does all of things. It's the best parts of Claude Desktop, Code, Cowork, and MCP connectors, but uses a client/server design. It's written in JavaFx so it's lightweight, fast, cross platform, and not another damned electron app. Ideal outcome is this turns into a startup. I think there's a real need for team-oriented AI to avoid siloing of knowledge.

scosman 4mo ago

The top of my ~/.claude/CLAUDE.md: > Don't turn a one-off or area-specific comment into a durable memory without my explicit confirmation. You have a history of over-indexing on one-offs, and those memories end up getting cited to override well-tuned skills.

dofm 4mo ago
Does that second sentence provide value beyond getting that off your chest? ;-) (Semi-serious question)
- scosman 4mo ago
  heh, who knows. I often give agents the motivation, which I generally helpful. Have not bothered to measure this case.

tracyhenry 4mo ago

The only times I found the memory feature useful are in "projects" I created myself. In a project my questions are usually revolved around the same topic. Having context carried across threads actually make a lot of sense. In the general mode where I'm expecting models to be *stateless*, having memory is very annoying.

wrs 4mo ago

My current technique, which seems to have improved maintainability, is to guide Claude to write commit messages specifically focused on "why this was done", "what changed in the theory of operation", and "what changed in the code". Then just reviewing the commits for a file or dir gives it a ton of useful context distilled from the sessions that produced them. Also, making a docs dir with concise .md files explaining the theory of operation and updating them with every commit.

chickensong 4mo ago

I don't do anything with full session transcripts, but I find value when Claude writes a memory. I don't actually want Claude to have those memories, but they often point to gaps in my harness. I'll occasionally sweep the memories, pick out what should go into the harness or CLAUDE.md, then delete the memories.