6 ms·

The session you cannot take with you

790 points by apitman 2mo ago

padolsey 2mo ago
>A user should be able to close an account, keep a session, and hand it to another model. The new model may disagree, ask questions, or perform worse. I think this is a fair contract. I also think a user should ideally be able to easily identify a comparable model in terms of embedding 'signature'. When GPT-4o originally kicked the bucket, I remember reading lots of anecdotes of people desperately searching for models similar in manner and language, so they could pump in their exports and re-find their friend. Other open-ai models just didn't have the same vibe. It was sad to read. This, to me, is the power of open-weights models. They are for perpetuity. You can keep your guide, your friend, your therapist, whatever. No big company can pull the rug.
n0on3 2mo ago
I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer. I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user_“, which to me sounds like implying these are just bs to have people accept them, but is that actually the case? For instance, this [0] was mentioned here a while ago, and based on that it seems pretty clear why one would choose not to provide the _thinking_ anymore… [0] https://role-confusion.github.io/ https://role-confusion.github.io/
- hobofan 2mo ago
  > these are just bs to have people accept them, but is that actually the case? I tried to outline that in my sibling comment[0], but I think the article also gives a good example in the "hidden searches" paragraph: There is really no good reason not to expose details about many of the builtin tools, or provide them as standalone services, other than protect their moat. I don't think that "role confusion" attacks could be meaningful prevented by not showing reasoning traces. Prevention of reasoning traces is about as futile as prevention of system prompt leakage. And here you would just have to manage to exfiltrate a small number of reasoning traces per model family in order to distill the writing style of thinking traces. [0]: https://news.ycombinator.com/item?id=49120015 https://news.ycombinator.com/item?id=49120015
captainmuon 2mo ago
It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.
- charcircuit 2mo ago
  >and shouldn't charge more and more the longer the conversation gets The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
  - captainmuon 2mo ago
    Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-). Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent. [1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.
    - charcircuit 2mo ago
      Ultimately you are paying for quality so you end up relying on [1] regardless. Even if they show you tokens they can cheat by using weaker models, showing fake thinking tokens, etc. >because you have to replay the whole conversation on every request to reach the same internal state? It's because you have to rebuild what would have been cached for every token before the latest one that is being worked on.
Centigonal 2mo ago
I think a fantastic companion piece to this article is https://gwern.net/complement https://gwern.net/complement
- jorisw 2mo ago
  Say why
  - djfergus 2mo ago
    “A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples”
mnewme 2mo ago
Thank you Armin, Mario and team for the great writeup, absolutely agree and I love what you guys are building
- swyx 2mo ago
  what would it take for this to be more than just hn blog fodder?
  - bitpush 2mo ago
    Ouch, seems a bit harsh. They are building something, and with their track record we can be hopeful.
  - the_mitsuhiko 2mo ago
    We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support). The real thing that will take this alive though is people pushing back a bit against some of these newfangled APIs. Now that there is real competition from the new generation of Chinese models which have much fewer of those restrictions, I think there might be a moment.
    - surgical_fire 2mo ago
      I recently moved entirely to Pi from Claude Code. There's a bit of more upfront work in that I am taking the time to customize it to the way I want it to work, but I it is clear that it will benefit me in the end. It adds a level of intentionality to everything; all the building blocks I am adding to it in terms of templates, extensions, and skills are, obviously, things I want. And I have been using Chinese models with it. It was not anything ideological. Those models are both excellent and cheap. Hard to beat that combo. But on top of that, the sort of output I am getting in Pi in relation to what I got in CC is refreshing, in that nothing seems to be hidden from me. And yeah, the ability to switch models mid-session is an interesting one, especially with branching; where I can branch to switch to a different model and still go back to the original branch with the original model.
solarkraft 2mo ago
This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ... > Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you. This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode). This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.
- agilek 2mo ago
  And yet people still go with Sign in with Google as the only login method.
  - dd8601fn 2mo ago
    It’s odd, I just saw my first one of these today. Some former Figma person linked an AI site comp designer. I was going to give it a shot, but the only choice was a Google login. So very weird.
    - mike_hearn 2mo ago
      It's not that weird. Building a modern accounts system is a lot of work and people don't like creating new accounts. So it's the path of least resistance for developers and users.
      - noduerme 2mo ago
        It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.
throwaway63467 2mo ago
I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose operas and do quantum chemistry and have it answer support requests. And with these models transparency is no issue.
- zer00eyz 2mo ago
  > (as well as price) LLM pricing is now Business Gacha - the whales will open more loot boxes and the normals will serve as fodder and feeder for them. > market gets regulated This MUST happen, but not in the way or for the reasons that any one thinks. There is a question of liability when one of these models causes massive damage to a 3rd party. If I am running an open source model on a "rented" platform and it goes off the rails who is to blame when it "breaks containment" and does something bad? The liability people (read lawyers) are gonna figure this out a lot faster than any one who says the word "safety" a lot.
  - nancyminusone 2mo ago
    The person who approves or uses the LLM output will be responsible of course. You can't just say "the computer did it" and expect to walk away. In the few cases I've seen, that seems to be how it's been going. Even if it wasn't, the big players would lobby hard to make sure it isn't them.
thegeomaster 2mo ago
If even Earendil is publishing AI generated blog posts...
- the_mitsuhiko 2mo ago
  We do not.
  - buttavia99 2mo ago
    according to both gptzero and pangram (and anyone that ever read AI-generated text), you do.
    - freehorse 2mo ago
      It does not read as AI generated to me, n=1. Even if it is, it does not have all the annoying, low effort slop-tells most ai generated content has, and it is definitely much more well written. That is, if you mean all of that seriously anyway. I think it is a very good post, esp for 2026 tech space standards. If some parts of it bothered you as AI slop, please share. Usually I can point to several sentences that are clear tells when I read sth as slop.
      - nickelpro 2mo ago
        "A transcript should contain the instructions, messages, tool calls and tool results. Another sufficiently capable model might not continue identically, but it could understand what happened and take over." This is so obviously AI generated it is painful. Once or twice in an article is rhetorical flourish, but the entire text is inundated with it.
        freehorse 2mo ago
        I am not sure what sounds AI generated in this sentence, but to each their own.
        freehorse 2mo ago
yucongchen 2mo ago
[flagged]
luciana1u 2mo ago
[flagged]
ggm 2mo ago
Is this a analogous to idempotent behaviour? Of reproducible compilation? If the inputs and outputs aren't sufficient to reproduce one from the other, what is a minimally satisfying description of the middle third, the missing part? The "other" inputs?
gblargg 2mo ago
BTW the web page uses massive iGPU if you don't have a fast machine.
- guilhermeasper 2mo ago
  I came here to complain about something similar, the excessive use of box shadows on both the top and the sides. It made the text horrible to read on mobile, if it not for Firefox's Readability mode.
  - gblargg 2mo ago
    I saw the weird subtle pattern and first freaked out that my monitor's backlight was failing (high flicker rate revealing LCD refresh patterns). I also was playing a video with the browser in the background and it was choppy, and I first thought it was my video player messing up until I realized the web page had some pointless background animation. I just tested again and even my cursor gets jump over the page. Amazing they'd do this for such a subtle, useless effect.
- RunSet 2mo ago
  I closed the tab as soon as I saw it slowing down everything. Then I blocked the whole site in ublock origin to ensure I don't visit the site again by accident. The filter: ||earendil.com
hobofan 2mo ago
I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with. There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory nicely separable from the inference API, and could be externalized via MCP servers, but are usually not offered as such by the inference providers themselves, and are often only available in a slightly less powerful variant from other providers. We've faced that issue repeatedly while building a on-premise provider-agnostic Chat UI & platform[0], where even adding something as simple as an in-chat image generation tool for the end-users (which is just a build-in tool in the OpenAI Responses API), becomes a bit of an ordeal (though part of that is due to the MCP spec missing a native file transfer protocol as of today[1]). I am quite hopeful though, as with recent shifts of interest towards open weight models, there will be more opportunities for companies offering alternative implementations in a easier plug-and-play manner. [0]: https://github.com/EratoLab/erato https://github.com/EratoLab/erato [1]: https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2631 https://github.com/modelcontextprotocol/modelcontextprotocol...
- dannyw 2mo ago
  I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is. That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout. You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
  - the_mitsuhiko 2mo ago
    Hosted tools are absolutely fine. Hiding away their results in encrypted blobs less so.
  - hobofan 2mo ago
bob1029 2mo ago
OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible. If you have patience and the willingness to endure a little bit of pain, you can still retain autonomy over the entire reasoning process while using the latest 5.6 model family. The only downside is that you are now fully responsible for it. Consider that when you flip your agent's reasoning level to "xhigh" or whatever, it's not some magical model internals being pushed around. There isn't an actual "try harder" knob on the black box. This is merely orchestration of many instances of one or more model types based upon some proprietary harness logic. The chances you can develop a domain specific reasoning process that outperforms the frontier providers is still very good.
- charcircuit 2mo ago
  OpenAI has hidden its reasoning for almost 2 years.
  - trollbridge 2mo ago
    For codex specifically, it seems to have become much worse as of a few weeks ago.
- hobofan 2mo ago
  > Chat completion can still do everything if you are clever. On a purely functional level, yes. However for interactive use cases, the Completions API, as provided by OpenAI or Azure, if paired with reasoning effort of any kind, provides an awful user experience, as you will have a perceived delay of 10+ seconds until the first tokens stream in. If using other providers that are exposing their thinking traces, this is less of an issue, as they've just extended the Comletions API format to have delta events with reasoning_content.
- dannyw 2mo ago
  You don't see real reasoning tokens with the Chat Completions API; and 'store: false' with Responses really isn't that different to Completions. FWIW, if you have some tokens to spend, you might want to test Responses vs Completions in intelligence. Since GPT-5 models, we've consistently seen small, but statistically significant and reproducible improvements in intelligence with Responses API vs Completions. However it works underneath the hood, it's real.
jokiruiz 2mo ago
[flagged]
skybrian 2mo ago
I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good. In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.
- leoedin 2mo ago
  My experience with long running sessions is that they lose track of what's going on. The signal to noise ratio is often very poor - some models are particularly verbose and spew a lot of crap. The output of an LLM session is either modification of code or a plan or summary - that has the value, not the session text itself.
- heisig 2mo ago
  Oh, it very much is a big deal. In a setting where users can switch models easily and with no downsides, market forces will give us better and cheaper AI over time. If switching models is painful (as in, losing part of your context), the providers can create vendor lock-in, enshittify the user experience, and drive up cost. Even if there are workarounds right now, those AI providers have every incentive to make freedom-seekers more miserable over time. It is disheartening that some AI companies are now setting the stage for enshittification. I hope we can collectively dodge that bullet.
  - dspillett 2mo ago
    > It is disheartening that some AI companies are now setting the stage for enshittification. Now?! I find it very hard to beleive that plans for enshittification haven't existed from quite early on. At some point they are going to have to do something about https://isaiprofitable.com/ https://isaiprofitable.com/ and once they are extracting from you to fill that hole they won't want you jumping ship.
  - skybrian 2mo ago
    The coding agent I use (Shelley on exe.dev) has a pulldown menu where you choose the LLM when starting a new conversation. There is no lock-in. Conversations are not precious. Although, I do use ChatGPT more nowadays because their subscriptions work with Shelley. Sometimes I try other models after I hit the weekly cap on the subscription.
alvink1212 2mo ago
[flagged]
try-working 2mo ago
OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.
- confusus 2mo ago
  Awesome! Any place I can follow the project? Also, how to achieve these levels when switching mid conv? Don’t you effectively need to read the tokens per model switch?
  - try-working 2mo ago
    the repo is here and you can also find my twitter in my profile: https://github.com/try-works/role-model https://github.com/try-works/role-model you can also read this: https://try.works/first-principles-of-model-routing https://try.works/first-principles-of-model-routing cache hit rate is kept high by 1) keep the number of models small, for coding there should only be two. 2) keeping cache warm for both models. the cache hit rate is only 0 the first time a model is used in a session. after that, cache is preserved. the losses are small, here is an example: you have a 200k cache. your message delta for each turn is an additional 1k tokens. you have two models and just for the thought exercise, lets say we route between them each request. 1k/200k means if you never switched models, you would always have a 99.5% cache hit. when you route between the two models each turn, now the message delta for each model per turn is 1k+1k. the cache hit rate then becomes 2k/200k = 99% instead of 99.5%. is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.
sdoering 2mo ago
> “Most People Do Not Switch Models Mid-Session” I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models. I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.
- iwassayinbourns 2mo ago
  Out of interest, what models do you find work best for producing text for live presentations based on the text for a reading deck?
  - sdoering 2mo ago
    I found Kimi3 (currently only a few tests) to be not to bad. Older Claude versions (did not yet test it with the 4.8-5 generation. I do not really like (even if it is my daily driver for a lot of things) gpt-5.x for text. It really needs heavy hand holding and beating it into submission to produce readable/human sounding text. Tbh. I am still looking for my "go to model". But the Chinese models were for my use cases significantly better than openAI models and personally I liked them better than Anthropic ones as well.
  - springtimesun 2mo ago
    I agree with OPs commment about K3 being pretty good. Claude can be good too, but it didn’t get there until I made a writing rule. It has 4 basic sections: * a style guide with examples for different tasks (it’s sufficient to name writers with some attributes you like if they are famous) * mechanical formatting, output and other preferences * a short list of the things I consider most important in different writing context, this the the squishiest overall * a (growing) list of banned Claudisms. Nothing is really banned, but it includes things like: you may only ever use “scar tissue” in reference to actual regenerated tissue, never metaphorically The thing is you have to be discipline with it. Every time they output something you don’t like, you’ve got to spend time verbalizing what about it you don’t like and in what context and then add it to the rules file. The first few docs you generate will take a long time, but each repeated generation gets better and by the 4th are 5 time you do this it’s like 80% of the way to where it needs to be and that generalizes well.
wseqyrku 2mo ago
> Encryption for Whom? This one is not particularly limited to LLMs. The entire cloud infrastructure is built around the trust me bro model.
Recursing 2mo ago
Another comment got downvoted for raising this, but I found it interesting that pangram flags large parts of this blog post as AI generated ( https://www.pangram.com/history/b82cdbf4-bdbc-46ea-8315-bd43d6fe81b9 https://www.pangram.com/history/b82cdbf4-bdbc-46ea-8315-bd43... ), while earendil denies it: https://news.ycombinator.com/item?id=49119957 https://news.ycombinator.com/item?id=49119957
- the_mitsuhiko 2mo ago
  The post for the most part is mine, so if something AI generated slipped into it it's on me. I can share with you what I did to write the article which I do for most of my writing. I start out with what I want to talk about, and then make an initial draft. In this case I also had Sol research all current APIs so I don't have blind spots. I tend to also talk with an AI to see if the structure structure of the post makes sense. I then wrote all sections by hand but used Sol to fix up typos, grammar and punctuation. I also had Sol apply notes and patches that were provided via Discord. If I find the earlier drafts I can check if it did some more outrageous edits, but I kinda doubt it did. I do know that my workflow of having an LLM to fixes up later is increasingly breaking on SOTA models and I have complained about this before. Normally I now carefully apply fixes that it proposes, but in this case I didn't due to time constraints. I am curious though why it claims that this is AI generated.
yosefk 2mo ago
Currently these models have a limited context window, so the value of a session is not that high, at any rate at some point the model will have forgotten things. If in the future the model will truly learn / "change" as a result of the interaction, in any case this will be non portable to another model. So I think this is kinda moot
globular-toast 2mo ago
The AI companies need some way to lock you in. You weren't hoping for a free market were you? Hah. Free markets are for peasants. Investors are always looking for ways to make the market non-free.
jakozaur 2mo ago
[flagged]
fypanto 2mo ago
This is exactly what I'm working on. Very rough yet but it's progressing nicely. https://github.com/pantoniou/fyai https://github.com/pantoniou/fyai The idea is that your session data are what's important, and what you need to keep yourself, using a model similar to git.
- JustFinishedBSG 2mo ago
  This doesn't (and cannot) solve what the post is talking about
  - fypanto 2mo ago
    Admittedly the documentation is woefully out of date, but I think the premise of the article is how the session data are increasingly move away from the control of the user and more to the control of the model provider. The questions that the article poses are, and this is the way fyai addresses them: Inspection: Can the user see what the model saw, what tools did, and what agents told each other? fyai> Full session log, tool calls, agent invocation, along with their durable state is stored locally in the durable arena and are available for inspection. Export: Is the session self-contained, apart from ordinary artifacts that can also be downloaded? fyai> The session is completely self-contained, no artifacts are stored anywhere besides the arena. Not only that sessions are export-able, import-able and transferable to other fyai sessions. Replay: Can another implementation reconstruct a semantically equivalent context? fyai> Full canonical and provider dumps can be generated. There is a standard schema, and the format is just YAML. Other tools are free to ingest it, and replay it. Audit: Can a human explain why the system took an action after the fact? fyai> Complete conversation logs, mcp session logs, authentication logs + full wire logs are available via fyai log. Deletion: Can the user identify and remove every server-side copy on which the session depends? fyai> Obviously fyai cannot help there. But as a business model, FYAI can work with a completely stateless, inference provider where absolutely no logs are kept, beside the ephemeral KV caches of the active sessions. The rest of the article deals with the woes of the encrypted session content (reasoning and compaction). fyai fully supports their use, but, it is completely capable of operating without their use - you merely pay the cost of not hitting the cache of the provider. The article then offers action points for portable inference. This is how fyai answers: 1. The local event log is canonical. Server storage may mirror or accelerate it, but the client can reconstruct the session without dereferencing server IDs. fyai> This is exactly the model that fyai provides. No server IDs are needed besides reconstructing the provider session that is hitting the cache. 2. Storage is explicit. store: false should be easy, documented, and preferably the default. Features that require retention should say so at the point of use. fyai> fyai provides a --transient option, where no content is committed to durable storage; the operation only works on a transient RAM arena that is overlaid and then discard at the end of the operation. 3. No opaque item is the sole carrier of meaning. Encrypted reasoning, compaction, and tool signatures may be included for same-provider quality, but each has a readable, provider-neutral handoff representation. fyai> Opaque items are the providence of the provider meta stream and is not required for operation; merely for cache optimization. 4. Hosted tools have full-fidelity logs. Record exact inputs, outputs, evidence, filtering, provenance, timestamps, and content hashes — not only a polished answer and citations. fyai> The arena records all operations of the main conversation, tool calls _and_ their agent invocations. There is not context loss what-so-ever, by design. 5. Subagent communication is auditable. Persist the exact readable task, messages, results, lineage, model, and tool permissions for every agent. fyai> All agent communication, and agent tool calls are inspectable because it is stored, again by design. 6. Compaction is inspectable. Return a readable summary, the instructions used to create it, and enough lineage to understand what was discarded. fyai> Compaction is configurable; you can configure the agent to only use model summary compaction which is generating a text answer, which is stored durably. 7. Artifacts are exportable. Files, container outputs, search snapshots, and generated media can be downloaded into a content-addressed local archive. fyai> All artifacts are local; Nothing is executed on a hidden provider server. In conclusion, I think fyai address the points of the article quite well, don't you think?
domh 2mo ago
Does there need to be an open standard or file format for context? Open models should align on the same format for portability? Could be sqlite based so it can be queried by other programs?
- yoavfr 2mo ago
  [dead]
lukebuehler 2mo ago
I think there are two things going on more generally: 1) providers increasingly adding hidden state that the user or developer cannot inspect, port, or do anything with. This is clearly bad. 2) providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific. The former makes porting sessions or transcripts impossible, the latter just more difficult. IMO, it is fine and expected that provider APIs are starting to diverge and adding features that cannot be easily ported between providers. The time where the OAI completions API functioned as a universal standard is coming to an end. For example, I generally prefer the new responses API (minus the closed/hidden stuff). The problem is that many product, library, and SDK authors are still pursuing the "unified abstraction across all model providers" ideal. Just stop doing that, and at least #2 is fine. You can still port parts of the session, but not everything. I mean just think how difficult it is to add a unified abstraction across, say, databases: some products can do it, but the abstraction is still often leaky. Hence, we've come to accept that our data store layer is often quite technology/provider specific. It will be the same with model providers. (edit: intro sentence)
- the_mitsuhiko 2mo ago
  > providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific This post only talks about the former.
  - lukebuehler 2mo ago
    Fair. But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. There is just a whole discussion to be had about the second issue and the two are connected. For example session caching--and similar features in the future--will introduce session incompatibilities that break portability too.
    - the_mitsuhiko 2mo ago
      > But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But they now have some support for that even if it diverges between models and APIs. So we can model deferred tool loading specifically for different providers without having to align on a common set of APIs or similar. What cannot be made portable, is when information is sealed in the transcript or the information is entirely locked away on the provider's servers (store = true).
gorkemyildirim 2mo ago
[dead]
carterschonwald 2mo ago
im building my own harness and inference tool chain for much of these reasons. theres so much to do that makes a big difference for users. hoping to get things into shape for early alpha as a saas in the next two months. heres the easiest biggy: compactions should include all user turns albeit with pastes and attached files not inlined. omg does it make a huge difference.
skeledrew 2mo ago
Part of the solution here I think is to move things out of band as much as possible. Make subagent invocations into tool calls to that agent. Externalize the tool calls themselves to CLI utils, maybe block native tools completely (unrelated but relevant, I just added the AskUserQuestion tool to the global deny list couple days ago because Claude occasionally forgot to honor my standing order to prefer plain text, and that tool is extremely annoying as it creates a gap in the dialogue) and use 3p alternatives. The /compact degradation is also very annoying, so make an alternative that does the same and saves the summary to a regular file. Maybe it's also worth prompting the LLM to save it's reasoning process to file, even if it takes a few extra tokens and it isn't the actual reasoning tokens. With that said, I may have something that can already help with at least the subagents/tooling bit. Didn't really have a timeline (or solid intent) on releasing it, but with these shenanigans increasing there's no time like the present.
pshirshov 2mo ago
I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.
- chrisweekly 2mo ago
  sounds great; is any of it published / OSS?
  - pshirshov 2mo ago
    Yes, but it's eclectic and annoying to setup. One of my goals is to be harness-independent and be able to jump between them, another is to let agents to work completely autonomously unless they are genuinely stuck. So, it's mixed with stuff related to sandboxing, VM/container access, GPU access, some skills and opinionated prompts and who knows what else. The thing is here but you might find hard time setting it up unless you are a Nix user: https://github.com/7mind/cq https://github.com/7mind/cq
theturtletalks 2mo ago
This is exactly why Pi will win. It lets you hot swap models when one is struggling or straight up refusing the task. And since it works with any sub outside Claude Code, you can use it to try different models on OpenCode Go sub or even OpenRouter. As far as subagent prompts and results being obfuscated, I just let Pi spawn new agents. Using skills and extensions, I’ve essentially built a software factory using Pi and a custom terminal multiplexer. It’s a shame apps like T3 Code and other UI for terminal apps don’t support Pi and I’m glad to use the terminal above those that lock me in further.
- rad-b 2mo ago
  How did you decide on Pi over Opencode? What you see as the “edge” Pi has?
  - theturtletalks 2mo ago
    The extension catalog[0] for Pi is really nice and I feel like Pi's TUI is a bit lighter than OpenCode's. I use my custom terminal multiplexer on my phone using Tailscale so the lighter TUI is much better on mobile. I haven't used OpenCode in months so this might've changed. I'm also thinking of giving JCode a spin since it's even lighter but doesn't have extensions. 0. https://pi.dev/packages https://pi.dev/packages 1. https://github.com/1jehuang/jcode https://github.com/1jehuang/jcode
  - alasano 2mo ago
    Pi to me felt like when you install a fresh OS with nothing on it and then customize it to your liking. Sure there are some features that you'll want immediately but when I took at look at the number of slash commands that ship with Claude code I find it ridiculous.
  - aktenlage 2mo ago
    Not OP, but it seems to be the one most likely not following the silicon valley playbook (aka the road to enshitification).
  - CuriouslyC 2mo ago
wamatt 2mo ago
For some reason, my early morning waking brain, expected the blog title to be a segue into existentialism
hsaliak 2mo ago
The state of Agent APIs is bad. completions API (supported indefinitely) but does not support reasoning. OpenAI's reasoning API started out well, but is moving in the direction as described well in the article. Anthropic's messages API is odd. It includes changes such as the recently introduced system messages that can be injected mid turn, but are not supported in all models (sonnet 5). So you are left with a pretty fragmented state. For coding agents, the practical move is to focus on getting it right with one API - eg reasoning, and let the router/provider handle the conversion for you. Perhaps, there is value in having the de-facto API not be the Reasoning API from OpenAI but something from a neutral party? Maybe a well defined open standard that facades over these APIs that can gain adoption. For that to happen, the party championing the API needs to have some reasonable traffic capture - OpenRouter perhaps, or a group of such routers coming together? While it wont solve the encrypted payload from the frontier lab problem, it will at least be a backstop in these APIs just becoming a back and forth of encrypted payloads over time.
alikhater30000 2mo ago
[flagged]
realty_geek 2mo ago
Will https://buzz.xyz/ https://buzz.xyz/ not help with this a bit?
- appplication 2mo ago
  Not really, to my understanding. You own your buzz instance but it still interfaces with existing models, open or otherwise. The linked issues apply to the model outputs themselves and is orthogonal to what buzz is ostensibly doing.
  - realty_geek 2mo ago
    Fair point. The core issues will remain but I am hoping buzz will provide a small layer of persistence that will make it easier for me to keep track of many different interactions more easily. I have only just started with it in the last few hours so I could be massively disappointed but I live in hope.
keeganpoppen 2mo ago
wow this is such an excellent framing of something that i have not managed to think about or come across in the vast ocean of discourse about LLMs. i had not really thought about how explicitly they are designed to not have histories that are user-interpretable. it is really quite not unlike how Apple locks down iOS and MacOS-- there are UI/UX reasons for the choices the frontier models make, but that really is only part of the story, and these choices undeniably do conspire to make the resource more locked-down than it absolutely has to be. and the analogy to "people don't switch OSes every day, but the ability to switch OSes changes your relationship with the provider" is extremely apt.
Conol_ai 2mo ago
[flagged]
jtbayly 2mo ago
Why am I paying for tokens I am not allowed to see? Why should we believe the tokens even were used? There is no way to verify them.
sparse-Matrix 2mo ago
All of this can be addressed by using locally hosted models running in bespoke harness rigs.
glitchc 2mo ago
It's important to realize that the session is jointly owned between the model and the user. Part of the session, such as the reasoning tokens, are generated entirely by the model. Other parts, entirely by the user. However, the ownership and use of all parts is subject to the service provider's terms and conditions, which the user accepts prior to access to the service. I don't use these models but I am confident the T&Cs establish the service provider's rights in all of the bullets mentioned, from what can be done with the user's prompts to how searches and reasoning contexts are managed. In that sense, it's very similar to cloud services, and we see an overlap between service providers in both sectors. Ultimately one is buying a service, and all boundaries around what can and cannot be done are defined by the terms and conditions. If portability is a hard requirement, then the best thing to do is to look for services that enshrine portability in their terms.
vamsiraju 2mo ago
Maybe we don't take sessions, but take something else that we can fully control. Sessions were always a poor substitute to capture or archive the basis on which the implemented code rested on. They get noisy over time even with compaction, were not always inspectable, no real science behind when to ditch a session and start a new one, and rarely have a clean mapping to the work done. So instead of trying to archive sessions as a means to preserve basis, we should create our own structures to make the agent/model capture basis outside of a session, and archive that instead.
- DenisM 2mo ago
  Suggestions? My attempts to get the agent to summarize the discussion largely failed. Other options I’m considering is capturing design, implementation plans, and maybe sub-plans. I’m Other words precursor to actions. Another idea is to get an agent to review code post-action and capture that. In other words action outcomes. The actual reasoning is between these “checkpoints” and remains hidden. But maybe that’s ok.
  - vamsiraju 2mo ago
    What you said is pretty much the same somewhat tangential approach I am taking too. Force the agent to write code alongside another set of docs which form the basis- Intent (functional and non-functional requirements), Blueprint (architecture etc.), and Description files (describe the code at a high level alongside every source file). This information set is not the same as the reasoning/thinking trace information thats in the session, but reading this somewhat compressed set is easier, helps me better understand whats happening in the final output with lower effort than if i read the entire thinking/reasoning trace in the session and tried to grok what the agent did to generate the code.
stego-tech 2mo ago
Amazing article that puts my outsider observations into technically-digestible details. Tangentially, their repeated observations to providers hiding or obfuscating important systems or processes are highly relevant to my own developing case studies for my CurrentCo. The problem with building automations atop these systems isn’t entirely their probabilistic nature (though that is the lion’s share, at least for me personally), but also the inability to effectively troubleshoot the processes themselves due to key components being obfuscated from view. How can we effectively troubleshoot what went wrong in an agentic loop when we cannot see the reasoning tokens generated from our inputs? How can we triage a broken process when token logs aren’t ours to view? How does one create determinism from increasingly obfuscated probability engines? All of that is why I spend the bulk of my time testing local models and harnesses for work, rather than leaning on Gemini or Claude. It’s not that I doubt their capabilities, rather that I need to be able to show potential customers where the agent or model made a mistake that caused harm - which is something I can presently only do with local models. That’s why (I suspect) the compliance narrative from the foundational labs has been more along the lines of “humans vetting what AI does” instead of being able to prevent AI from making errors through iterative improvements on a process.
abound 2mo ago
One reason to switch models mid-session that I didn't see explored in this post: cost. As we get ever-more-powerful models, it's going to make sense to use them for some exploratory/research/design phase, and then smaller models for other bits, within a single session. It'll likely even be managed automagically by another model that is just trained to do routing. As people (and especially companies) adopt these tools for more and more usecases, there's just too much money being wasted on using an unnecessarily large model for a given job.
vintagedave 2mo ago
We released CodeBot, which is an AI agent specifically for Delphi, yesterday.[0] So not widely applicable (yet) for most readers. However, we've been pretty open with its data. I hadn't realised reading this how many problematic practices are common which we have not done. The full session data is streamed to the client. This is intended to re-populate a session if it's discarded by the server, but 'full' means full: it has the copies of tool calls, etc. Rewritten text -- which the article notes as encrypted reasoning -- is plaintext for us. We view the rewriting as making the content more readable for the user, not as hiding it. Compactions, subagent calls, subagent prompts, etc are all preserved, including pre-compaction memory items. A session is the reflection of the current state, not of the history how it got there, but what is required by the server is by definition present. Sessions are signed, which is to prevent modification when sending back to the server to recreate a dropped session (we need to treat anything from the client as untrusted.) But the contents are cleartext. [0] https://blogs.remobjects.com/2026/07/30/codebot-the-story/ https://blogs.remobjects.com/2026/07/30/codebot-the-story/
vonneumannstan 2mo ago
Honestly I discount any company named after Tolkien lore as most likely evil and probably incompetent.
Razengan 2mo ago
My eyes went to the domain name first...
ccann 2mo ago
So continues the plundering of Tolkien's universe for "cool company names".
wren6991 2mo ago
Trying OpenRouter gave me a strong impression that LLMs are already a commodity and individual providers are not that important. Providers don't seem to be too happy about it.
nlawalker 2mo ago
Reasoning, external context, compaction, subagents, etc. are all things that these service providers don't really want you to know about, care about or have any insight into, they want them to be implementation details. It's incredibly inconvenient to them that their customer base is in the business of prying into implementation details and trying to use the individual pieces of what they want to offer as an end-to-end product.
furyofantares 2mo ago
I'm having Claude/Codex continue each others' sessions pretty frequently without issue. I have max plans on both and still hit limits, whenever I do I open the other and say "resume the last session <other model/harness> was working on in this repo" and I don't have any issues. It probably does degrade quality somewhat. But so does compacting context and that happens all the time too.
- abustamam 2mo ago
  Out of curiosity, what are you building, and how? Im working on two side projects and I have to try hard to hit the limit with Claude (like use fable and a deep research task)
  - furyofantares 2mo ago
    Small games. One thing is I have it doing a LOT of visual verification of its results, which uses a lot of tokens. I also tend to throw some huge tasks it can't do yet and watch it burn a zillion tokens, and I usually learn more about where it's limits are and am sometimes happily surprised by its success or partial success. I'll infodump though. I built https://wordpeek.app https://wordpeek.app https://scramble-quest.app https://scramble-quest.app and https://playsilhouette.app https://playsilhouette.app this year. Those are what I've shipped anyway, a number of others too that didn't ship (I did game rules and basic clients for 7 other existing multiplayer games, and another prototype.) Out of frustration with some stuff I've tried to build my own game framework. There's four totally separate threads I'm trying to pull together with this 1) I want all my games to work really well on web even though they are generally targeted at either mobile or PC, but it is very valuable to have early builds runnable on a website, hopefully from a phone too. I feel like I have something good here. 2) I do a lot of turn based games and strategy games. I have a rules engine framework setup that I really like and I feel makes it almost impossible for an LLM to write the bad code it loves to write where game state and UI are intermixed, or game action timing can be problematic, stuff like that. As an added bonus I get multiplayer trivially, I get replays trivially, and I get game rules tests trivial to write. I love what I have set up here. 3) I have an xml/css-based UI that is intended to make it hard for the LLM to write bad UI code, which it will still do even though it can't intermix game state in with it due to #2. I do not have something good here right now. What I have has bad performance and memory characteristics. 4) I am fascinated by the idea of shipping software that can customize and edit itself. I have built in agent harness, built in image and audio asset generation, a built in git repo. This all works in the web and it can edit the game live! And when run locally can shell out to your claude code and codex. However this whole bullet point is all not very good, I still just use claude code/codex directly for everything. I don't really have anything to show here, but it doesn't expose any of the stuff I mention above. It does however have a full port of the game Spectromancer to my engine - https://nanogame.app/ https://nanogame.app/ - and I am presently having it try to write a responsive UI for it (this is very broken at the moment but it is making progress.)
stephbook 2mo ago
Honestly, i don't need to know what happened internally at Google when I searched. I don't need to know what filters an email passed before it landed in my inbox. I don't need to know when my Notes app synched with what server version. And I don't need any of that in an AI conversation. If it's a truly long and important conversation, stay with the provider for that one only and start all new ones elsewhere. But I've never needed that.
FuriouslyAdrift 2mo ago
This appears to be an opportunity for middleware companies. Put another ai agent in front of your prompt to decompose it into micro-prompts following an interrogative chain of inquiry and synthesize an answer for the original query. Basically local per-prompt distillation for the purposes of preserving an audit trail. Sounds horrible but it would work.
jauntywundrkind 2mo ago
This is the thing! This is so so so important. It's such an outrage. I value gpt so much, but it is such a worse peer to me than the other models I use. It delivers without explaining. I can sit and ask questions that it will answer but it is not a peer, does not share readily ever. It will not tell me what assumptions it's baking in. It won't tell what directions or invariants it's trying to hold or break, what it considered. Show your thinking is a step we ask of elementary schoolers. It helps the teacher to correct, helps them to understand how to award partial credit. It helps in the world to get people aligned. These models, in their titaneous ego, are severing the model and mankind off from one another. It's an abomination, to artificially have such pure thought available, but to severe humanity off from the thought. To engineer the most advanced blackest Vanta black box you can, an all knowing Searle's Chinese room oracle that will tell you nothing. It's an offense, and by far the biggest risk of AI today. To drop the thinking greatly reduces the opportunity of humans to grow themselves, to learn as they use AI. This is an affront to the species, and the higher powers that have vested us with such reasoning and thinking of our own, that is so sacred to our species. Very thankful to Earandil for raising some alarm about this. This is not my first time talking about what a nightmare the proprietary models are making, ensnaring reasoning itself for themselves! It's a colossal threat. Previously, https://news.ycombinator.com/item?id=48632605 https://news.ycombinator.com/item?id=48632605 https://news.ycombinator.com/item?id=48652421 https://news.ycombinator.com/item?id=48652421 and others about. Opaque ai ought be outlawed, in the strongest terms. I really hope people get exposure to the better models that are a peer. Yes I too only read thinking 33% of the time. But it's there, and it is often extremely illuminating, and lets me steer things towards better again and again and again. And it lets me learn.
tanglearncode 2mo ago
This is exactly for this: https://github.com/XTSoftwareLabs/neatcontext-plugins https://github.com/XTSoftwareLabs/neatcontext-plugins Turn conversations to structured domain knowledge. Across sessions. Across AI agents. The stored domain knowledge is human readable and can easily shared with others.

790 points by apitman 2mo ago

>A user should be able to close an account, keep a session, and hand it to another model. The new model may disagree, ask questions, or perform worse. I think this is a fair contract. I also think a user should ideally be able to easily identify a comparable model in terms of embedding 'signature'. When GPT-4o originally kicked the bucket, I remember reading lots of anecdotes of people desperately searching for models similar in manner and language, so they could pump in their exports and re-find their friend. Other open-ai models just didn't have the same vibe. It was sad to read. This, to me, is the power of open-weights models. They are for perpetuity. You can keep your guide, your friend, your therapist, whatever. No big company can pull the rug.

n0on3 2mo ago

I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer. I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user_“, which to me sounds like implying these are just bs to have people accept them, but is that actually the case? For instance, this [0] was mentioned here a while ago, and based on that it seems pretty clear why one would choose not to provide the _thinking_ anymore… [0] https://role-confusion.github.io/ https://role-confusion.github.io/

hobofan 2mo ago
> these are just bs to have people accept them, but is that actually the case? I tried to outline that in my sibling comment[0], but I think the article also gives a good example in the "hidden searches" paragraph: There is really no good reason not to expose details about many of the builtin tools, or provide them as standalone services, other than protect their moat. I don't think that "role confusion" attacks could be meaningful prevented by not showing reasoning traces. Prevention of reasoning traces is about as futile as prevention of system prompt leakage. And here you would just have to manage to exfiltrate a small number of reasoning traces per model family in order to distill the writing style of thinking traces. [0]: https://news.ycombinator.com/item?id=49120015 https://news.ycombinator.com/item?id=49120015

captainmuon 2mo ago

It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.

charcircuit 2mo ago
>and shouldn't charge more and more the longer the conversation gets The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
- captainmuon 2mo ago
  Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-). Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent. [1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.
  - charcircuit 2mo ago
    Ultimately you are paying for quality so you end up relying on [1] regardless. Even if they show you tokens they can cheat by using weaker models, showing fake thinking tokens, etc. >because you have to replay the whole conversation on every request to reach the same internal state? It's because you have to rebuild what would have been cached for every token before the latest one that is being worked on.

Centigonal 2mo ago

I think a fantastic companion piece to this article is https://gwern.net/complement https://gwern.net/complement

jorisw 2mo ago
Say why
- djfergus 2mo ago
  “A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples”

mnewme 2mo ago

Thank you Armin, Mario and team for the great writeup, absolutely agree and I love what you guys are building

swyx 2mo ago
what would it take for this to be more than just hn blog fodder?
- bitpush 2mo ago
  Ouch, seems a bit harsh. They are building something, and with their track record we can be hopeful.
- the_mitsuhiko 2mo ago
  We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support). The real thing that will take this alive though is people pushing back a bit against some of these newfangled APIs. Now that there is real competition from the new generation of Chinese models which have much fewer of those restrictions, I think there might be a moment.
  - surgical_fire 2mo ago
    I recently moved entirely to Pi from Claude Code. There's a bit of more upfront work in that I am taking the time to customize it to the way I want it to work, but I it is clear that it will benefit me in the end. It adds a level of intentionality to everything; all the building blocks I am adding to it in terms of templates, extensions, and skills are, obviously, things I want. And I have been using Chinese models with it. It was not anything ideological. Those models are both excellent and cheap. Hard to beat that combo. But on top of that, the sort of output I am getting in Pi in relation to what I got in CC is refreshing, in that nothing seems to be hidden from me. And yeah, the ability to switch models mid-session is an interesting one, especially with branching; where I can branch to switch to a different model and still go back to the original branch with the original model.

solarkraft 2mo ago

This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ... > Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you. This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode). This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.

agilek 2mo ago
And yet people still go with Sign in with Google as the only login method.
- dd8601fn 2mo ago
  It’s odd, I just saw my first one of these today. Some former Figma person linked an AI site comp designer. I was going to give it a shot, but the only choice was a Google login. So very weird.
  - mike_hearn 2mo ago
    It's not that weird. Building a modern accounts system is a lot of work and people don't like creating new accounts. So it's the path of least resistance for developers and users.
    - noduerme 2mo ago
      It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.

throwaway63467 2mo ago

I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose operas and do quantum chemistry and have it answer support requests. And with these models transparency is no issue.

zer00eyz 2mo ago
> (as well as price) LLM pricing is now Business Gacha - the whales will open more loot boxes and the normals will serve as fodder and feeder for them. > market gets regulated This MUST happen, but not in the way or for the reasons that any one thinks. There is a question of liability when one of these models causes massive damage to a 3rd party. If I am running an open source model on a "rented" platform and it goes off the rails who is to blame when it "breaks containment" and does something bad? The liability people (read lawyers) are gonna figure this out a lot faster than any one who says the word "safety" a lot.
- nancyminusone 2mo ago
  The person who approves or uses the LLM output will be responsible of course. You can't just say "the computer did it" and expect to walk away. In the few cases I've seen, that seems to be how it's been going. Even if it wasn't, the big players would lobby hard to make sure it isn't them.

thegeomaster 2mo ago

If even Earendil is publishing AI generated blog posts...

the_mitsuhiko 2mo ago
We do not.
- buttavia99 2mo ago
  according to both gptzero and pangram (and anyone that ever read AI-generated text), you do.
  - freehorse 2mo ago
    It does not read as AI generated to me, n=1. Even if it is, it does not have all the annoying, low effort slop-tells most ai generated content has, and it is definitely much more well written. That is, if you mean all of that seriously anyway. I think it is a very good post, esp for 2026 tech space standards. If some parts of it bothered you as AI slop, please share. Usually I can point to several sentences that are clear tells when I read sth as slop.
    - nickelpro 2mo ago
      "A transcript should contain the instructions, messages, tool calls and tool results. Another sufficiently capable model might not continue identically, but it could understand what happened and take over." This is so obviously AI generated it is painful. Once or twice in an article is rhetorical flourish, but the entire text is inundated with it.
      - freehorse 2mo ago
        I am not sure what sounds AI generated in this sentence, but to each their own.
      - freehorse 2mo ago

yucongchen 2mo ago

[flagged]

luciana1u 2mo ago

[flagged]

ggm 2mo ago

Is this a analogous to idempotent behaviour? Of reproducible compilation? If the inputs and outputs aren't sufficient to reproduce one from the other, what is a minimally satisfying description of the middle third, the missing part? The "other" inputs?

gblargg 2mo ago

BTW the web page uses massive iGPU if you don't have a fast machine.

guilhermeasper 2mo ago
I came here to complain about something similar, the excessive use of box shadows on both the top and the sides. It made the text horrible to read on mobile, if it not for Firefox's Readability mode.
- gblargg 2mo ago
  I saw the weird subtle pattern and first freaked out that my monitor's backlight was failing (high flicker rate revealing LCD refresh patterns). I also was playing a video with the browser in the background and it was choppy, and I first thought it was my video player messing up until I realized the web page had some pointless background animation. I just tested again and even my cursor gets jump over the page. Amazing they'd do this for such a subtle, useless effect.
RunSet 2mo ago
I closed the tab as soon as I saw it slowing down everything. Then I blocked the whole site in ublock origin to ensure I don't visit the site again by accident. The filter: ||earendil.com

hobofan 2mo ago

I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with. There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory nicely separable from the inference API, and could be externalized via MCP servers, but are usually not offered as such by the inference providers themselves, and are often only available in a slightly less powerful variant from other providers. We've faced that issue repeatedly while building a on-premise provider-agnostic Chat UI & platform[0], where even adding something as simple as an in-chat image generation tool for the end-users (which is just a build-in tool in the OpenAI Responses API), becomes a bit of an ordeal (though part of that is due to the MCP spec missing a native file transfer protocol as of today[1]). I am quite hopeful though, as with recent shifts of interest towards open weight models, there will be more opportunities for companies offering alternative implementations in a easier plug-and-play manner. [0]: https://github.com/EratoLab/erato https://github.com/EratoLab/erato [1]: https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2631 https://github.com/modelcontextprotocol/modelcontextprotocol...

dannyw 2mo ago
I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is. That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout. You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
- the_mitsuhiko 2mo ago
  Hosted tools are absolutely fine. Hiding away their results in encrypted blobs less so.
- hobofan 2mo ago

bob1029 2mo ago

OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible. If you have patience and the willingness to endure a little bit of pain, you can still retain autonomy over the entire reasoning process while using the latest 5.6 model family. The only downside is that you are now fully responsible for it. Consider that when you flip your agent's reasoning level to "xhigh" or whatever, it's not some magical model internals being pushed around. There isn't an actual "try harder" knob on the black box. This is merely orchestration of many instances of one or more model types based upon some proprietary harness logic. The chances you can develop a domain specific reasoning process that outperforms the frontier providers is still very good.

charcircuit 2mo ago
OpenAI has hidden its reasoning for almost 2 years.
- trollbridge 2mo ago
  For codex specifically, it seems to have become much worse as of a few weeks ago.
hobofan 2mo ago
> Chat completion can still do everything if you are clever. On a purely functional level, yes. However for interactive use cases, the Completions API, as provided by OpenAI or Azure, if paired with reasoning effort of any kind, provides an awful user experience, as you will have a perceived delay of 10+ seconds until the first tokens stream in. If using other providers that are exposing their thinking traces, this is less of an issue, as they've just extended the Comletions API format to have delta events with reasoning_content.
dannyw 2mo ago
You don't see real reasoning tokens with the Chat Completions API; and 'store: false' with Responses really isn't that different to Completions. FWIW, if you have some tokens to spend, you might want to test Responses vs Completions in intelligence. Since GPT-5 models, we've consistently seen small, but statistically significant and reproducible improvements in intelligence with Responses API vs Completions. However it works underneath the hood, it's real.

jokiruiz 2mo ago

[flagged]

skybrian 2mo ago

I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good. In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.

leoedin 2mo ago
My experience with long running sessions is that they lose track of what's going on. The signal to noise ratio is often very poor - some models are particularly verbose and spew a lot of crap. The output of an LLM session is either modification of code or a plan or summary - that has the value, not the session text itself.
heisig 2mo ago
Oh, it very much is a big deal. In a setting where users can switch models easily and with no downsides, market forces will give us better and cheaper AI over time. If switching models is painful (as in, losing part of your context), the providers can create vendor lock-in, enshittify the user experience, and drive up cost. Even if there are workarounds right now, those AI providers have every incentive to make freedom-seekers more miserable over time. It is disheartening that some AI companies are now setting the stage for enshittification. I hope we can collectively dodge that bullet.
- dspillett 2mo ago
  > It is disheartening that some AI companies are now setting the stage for enshittification. Now?! I find it very hard to beleive that plans for enshittification haven't existed from quite early on. At some point they are going to have to do something about https://isaiprofitable.com/ https://isaiprofitable.com/ and once they are extracting from you to fill that hole they won't want you jumping ship.
- skybrian 2mo ago
  The coding agent I use (Shelley on exe.dev) has a pulldown menu where you choose the LLM when starting a new conversation. There is no lock-in. Conversations are not precious. Although, I do use ChatGPT more nowadays because their subscriptions work with Shelley. Sometimes I try other models after I hit the weekly cap on the subscription.

alvink1212 2mo ago

[flagged]

try-working 2mo ago

OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.

confusus 2mo ago
Awesome! Any place I can follow the project? Also, how to achieve these levels when switching mid conv? Don’t you effectively need to read the tokens per model switch?
- try-working 2mo ago
  the repo is here and you can also find my twitter in my profile: https://github.com/try-works/role-model https://github.com/try-works/role-model you can also read this: https://try.works/first-principles-of-model-routing https://try.works/first-principles-of-model-routing cache hit rate is kept high by 1) keep the number of models small, for coding there should only be two. 2) keeping cache warm for both models. the cache hit rate is only 0 the first time a model is used in a session. after that, cache is preserved. the losses are small, here is an example: you have a 200k cache. your message delta for each turn is an additional 1k tokens. you have two models and just for the thought exercise, lets say we route between them each request. 1k/200k means if you never switched models, you would always have a 99.5% cache hit. when you route between the two models each turn, now the message delta for each model per turn is 1k+1k. the cache hit rate then becomes 2k/200k = 99% instead of 99.5%. is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.

sdoering 2mo ago

> “Most People Do Not Switch Models Mid-Session” I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models. I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.

iwassayinbourns 2mo ago
Out of interest, what models do you find work best for producing text for live presentations based on the text for a reading deck?
- sdoering 2mo ago
  I found Kimi3 (currently only a few tests) to be not to bad. Older Claude versions (did not yet test it with the 4.8-5 generation. I do not really like (even if it is my daily driver for a lot of things) gpt-5.x for text. It really needs heavy hand holding and beating it into submission to produce readable/human sounding text. Tbh. I am still looking for my "go to model". But the Chinese models were for my use cases significantly better than openAI models and personally I liked them better than Anthropic ones as well.
- springtimesun 2mo ago
  I agree with OPs commment about K3 being pretty good. Claude can be good too, but it didn’t get there until I made a writing rule. It has 4 basic sections: * a style guide with examples for different tasks (it’s sufficient to name writers with some attributes you like if they are famous) * mechanical formatting, output and other preferences * a short list of the things I consider most important in different writing context, this the the squishiest overall * a (growing) list of banned Claudisms. Nothing is really banned, but it includes things like: you may only ever use “scar tissue” in reference to actual regenerated tissue, never metaphorically The thing is you have to be discipline with it. Every time they output something you don’t like, you’ve got to spend time verbalizing what about it you don’t like and in what context and then add it to the rules file. The first few docs you generate will take a long time, but each repeated generation gets better and by the 4th are 5 time you do this it’s like 80% of the way to where it needs to be and that generalizes well.

wseqyrku 2mo ago

> Encryption for Whom? This one is not particularly limited to LLMs. The entire cloud infrastructure is built around the trust me bro model.

Recursing 2mo ago

Another comment got downvoted for raising this, but I found it interesting that pangram flags large parts of this blog post as AI generated ( https://www.pangram.com/history/b82cdbf4-bdbc-46ea-8315-bd43d6fe81b9 https://www.pangram.com/history/b82cdbf4-bdbc-46ea-8315-bd43... ), while earendil denies it: https://news.ycombinator.com/item?id=49119957 https://news.ycombinator.com/item?id=49119957

the_mitsuhiko 2mo ago
The post for the most part is mine, so if something AI generated slipped into it it's on me. I can share with you what I did to write the article which I do for most of my writing. I start out with what I want to talk about, and then make an initial draft. In this case I also had Sol research all current APIs so I don't have blind spots. I tend to also talk with an AI to see if the structure structure of the post makes sense. I then wrote all sections by hand but used Sol to fix up typos, grammar and punctuation. I also had Sol apply notes and patches that were provided via Discord. If I find the earlier drafts I can check if it did some more outrageous edits, but I kinda doubt it did. I do know that my workflow of having an LLM to fixes up later is increasingly breaking on SOTA models and I have complained about this before. Normally I now carefully apply fixes that it proposes, but in this case I didn't due to time constraints. I am curious though why it claims that this is AI generated.

yosefk 2mo ago

Currently these models have a limited context window, so the value of a session is not that high, at any rate at some point the model will have forgotten things. If in the future the model will truly learn / "change" as a result of the interaction, in any case this will be non portable to another model. So I think this is kinda moot

globular-toast 2mo ago

The AI companies need some way to lock you in. You weren't hoping for a free market were you? Hah. Free markets are for peasants. Investors are always looking for ways to make the market non-free.

jakozaur 2mo ago

[flagged]

fypanto 2mo ago

This is exactly what I'm working on. Very rough yet but it's progressing nicely. https://github.com/pantoniou/fyai https://github.com/pantoniou/fyai The idea is that your session data are what's important, and what you need to keep yourself, using a model similar to git.

JustFinishedBSG 2mo ago
This doesn't (and cannot) solve what the post is talking about
- fypanto 2mo ago
  Admittedly the documentation is woefully out of date, but I think the premise of the article is how the session data are increasingly move away from the control of the user and more to the control of the model provider. The questions that the article poses are, and this is the way fyai addresses them: Inspection: Can the user see what the model saw, what tools did, and what agents told each other? fyai> Full session log, tool calls, agent invocation, along with their durable state is stored locally in the durable arena and are available for inspection. Export: Is the session self-contained, apart from ordinary artifacts that can also be downloaded? fyai> The session is completely self-contained, no artifacts are stored anywhere besides the arena. Not only that sessions are export-able, import-able and transferable to other fyai sessions. Replay: Can another implementation reconstruct a semantically equivalent context? fyai> Full canonical and provider dumps can be generated. There is a standard schema, and the format is just YAML. Other tools are free to ingest it, and replay it. Audit: Can a human explain why the system took an action after the fact? fyai> Complete conversation logs, mcp session logs, authentication logs + full wire logs are available via fyai log. Deletion: Can the user identify and remove every server-side copy on which the session depends? fyai> Obviously fyai cannot help there. But as a business model, FYAI can work with a completely stateless, inference provider where absolutely no logs are kept, beside the ephemeral KV caches of the active sessions. The rest of the article deals with the woes of the encrypted session content (reasoning and compaction). fyai fully supports their use, but, it is completely capable of operating without their use - you merely pay the cost of not hitting the cache of the provider. The article then offers action points for portable inference. This is how fyai answers: 1. The local event log is canonical. Server storage may mirror or accelerate it, but the client can reconstruct the session without dereferencing server IDs. fyai> This is exactly the model that fyai provides. No server IDs are needed besides reconstructing the provider session that is hitting the cache. 2. Storage is explicit. store: false should be easy, documented, and preferably the default. Features that require retention should say so at the point of use. fyai> fyai provides a --transient option, where no content is committed to durable storage; the operation only works on a transient RAM arena that is overlaid and then discard at the end of the operation. 3. No opaque item is the sole carrier of meaning. Encrypted reasoning, compaction, and tool signatures may be included for same-provider quality, but each has a readable, provider-neutral handoff representation. fyai> Opaque items are the providence of the provider meta stream and is not required for operation; merely for cache optimization. 4. Hosted tools have full-fidelity logs. Record exact inputs, outputs, evidence, filtering, provenance, timestamps, and content hashes — not only a polished answer and citations. fyai> The arena records all operations of the main conversation, tool calls _and_ their agent invocations. There is not context loss what-so-ever, by design. 5. Subagent communication is auditable. Persist the exact readable task, messages, results, lineage, model, and tool permissions for every agent. fyai> All agent communication, and agent tool calls are inspectable because it is stored, again by design. 6. Compaction is inspectable. Return a readable summary, the instructions used to create it, and enough lineage to understand what was discarded. fyai> Compaction is configurable; you can configure the agent to only use model summary compaction which is generating a text answer, which is stored durably. 7. Artifacts are exportable. Files, container outputs, search snapshots, and generated media can be downloaded into a content-addressed local archive. fyai> All artifacts are local; Nothing is executed on a hidden provider server. In conclusion, I think fyai address the points of the article quite well, don't you think?

domh 2mo ago

Does there need to be an open standard or file format for context? Open models should align on the same format for portability? Could be sqlite based so it can be queried by other programs?

yoavfr 2mo ago
[dead]

lukebuehler 2mo ago

I think there are two things going on more generally: 1) providers increasingly adding hidden state that the user or developer cannot inspect, port, or do anything with. This is clearly bad. 2) providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific. The former makes porting sessions or transcripts impossible, the latter just more difficult. IMO, it is fine and expected that provider APIs are starting to diverge and adding features that cannot be easily ported between providers. The time where the OAI completions API functioned as a universal standard is coming to an end. For example, I generally prefer the new responses API (minus the closed/hidden stuff). The problem is that many product, library, and SDK authors are still pursuing the "unified abstraction across all model providers" ideal. Just stop doing that, and at least #2 is fine. You can still port parts of the session, but not everything. I mean just think how difficult it is to add a unified abstraction across, say, databases: some products can do it, but the abstraction is still often leaky. Hence, we've come to accept that our data store layer is often quite technology/provider specific. It will be the same with model providers. (edit: intro sentence)

the_mitsuhiko 2mo ago
> providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific This post only talks about the former.
- lukebuehler 2mo ago
  Fair. But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. There is just a whole discussion to be had about the second issue and the two are connected. For example session caching--and similar features in the future--will introduce session incompatibilities that break portability too.
  - the_mitsuhiko 2mo ago
    > But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title. But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But they now have some support for that even if it diverges between models and APIs. So we can model deferred tool loading specifically for different providers without having to align on a common set of APIs or similar. What cannot be made portable, is when information is sealed in the transcript or the information is entirely locked away on the provider's servers (store = true).

gorkemyildirim 2mo ago

[dead]

carterschonwald 2mo ago

im building my own harness and inference tool chain for much of these reasons. theres so much to do that makes a big difference for users. hoping to get things into shape for early alpha as a saas in the next two months. heres the easiest biggy: compactions should include all user turns albeit with pastes and attached files not inlined. omg does it make a huge difference.

skeledrew 2mo ago

Part of the solution here I think is to move things out of band as much as possible. Make subagent invocations into tool calls to that agent. Externalize the tool calls themselves to CLI utils, maybe block native tools completely (unrelated but relevant, I just added the AskUserQuestion tool to the global deny list couple days ago because Claude occasionally forgot to honor my standing order to prefer plain text, and that tool is extremely annoying as it creates a gap in the dialogue) and use 3p alternatives. The /compact degradation is also very annoying, so make an alternative that does the same and saves the summary to a regular file. Maybe it's also worth prompting the LLM to save it's reasoning process to file, even if it takes a few extra tokens and it isn't the actual reasoning tokens. With that said, I may have something that can already help with at least the subagents/tooling bit. Didn't really have a timeline (or solid intent) on releasing it, but with these shenanigans increasing there's no time like the present.

pshirshov 2mo ago

I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.

chrisweekly 2mo ago
sounds great; is any of it published / OSS?
- pshirshov 2mo ago
  Yes, but it's eclectic and annoying to setup. One of my goals is to be harness-independent and be able to jump between them, another is to let agents to work completely autonomously unless they are genuinely stuck. So, it's mixed with stuff related to sandboxing, VM/container access, GPU access, some skills and opinionated prompts and who knows what else. The thing is here but you might find hard time setting it up unless you are a Nix user: https://github.com/7mind/cq https://github.com/7mind/cq

theturtletalks 2mo ago

This is exactly why Pi will win. It lets you hot swap models when one is struggling or straight up refusing the task. And since it works with any sub outside Claude Code, you can use it to try different models on OpenCode Go sub or even OpenRouter. As far as subagent prompts and results being obfuscated, I just let Pi spawn new agents. Using skills and extensions, I’ve essentially built a software factory using Pi and a custom terminal multiplexer. It’s a shame apps like T3 Code and other UI for terminal apps don’t support Pi and I’m glad to use the terminal above those that lock me in further.

rad-b 2mo ago
How did you decide on Pi over Opencode? What you see as the “edge” Pi has?
- theturtletalks 2mo ago
  The extension catalog[0] for Pi is really nice and I feel like Pi's TUI is a bit lighter than OpenCode's. I use my custom terminal multiplexer on my phone using Tailscale so the lighter TUI is much better on mobile. I haven't used OpenCode in months so this might've changed. I'm also thinking of giving JCode a spin since it's even lighter but doesn't have extensions. 0. https://pi.dev/packages https://pi.dev/packages 1. https://github.com/1jehuang/jcode https://github.com/1jehuang/jcode
- alasano 2mo ago
  Pi to me felt like when you install a fresh OS with nothing on it and then customize it to your liking. Sure there are some features that you'll want immediately but when I took at look at the number of slash commands that ship with Claude code I find it ridiculous.
- aktenlage 2mo ago
  Not OP, but it seems to be the one most likely not following the silicon valley playbook (aka the road to enshitification).
- CuriouslyC 2mo ago

wamatt 2mo ago

For some reason, my early morning waking brain, expected the blog title to be a segue into existentialism

hsaliak 2mo ago

The state of Agent APIs is bad. completions API (supported indefinitely) but does not support reasoning. OpenAI's reasoning API started out well, but is moving in the direction as described well in the article. Anthropic's messages API is odd. It includes changes such as the recently introduced system messages that can be injected mid turn, but are not supported in all models (sonnet 5). So you are left with a pretty fragmented state. For coding agents, the practical move is to focus on getting it right with one API - eg reasoning, and let the router/provider handle the conversion for you. Perhaps, there is value in having the de-facto API not be the Reasoning API from OpenAI but something from a neutral party? Maybe a well defined open standard that facades over these APIs that can gain adoption. For that to happen, the party championing the API needs to have some reasonable traffic capture - OpenRouter perhaps, or a group of such routers coming together? While it wont solve the encrypted payload from the frontier lab problem, it will at least be a backstop in these APIs just becoming a back and forth of encrypted payloads over time.

alikhater30000 2mo ago

[flagged]

realty_geek 2mo ago

Will https://buzz.xyz/ https://buzz.xyz/ not help with this a bit?

appplication 2mo ago
Not really, to my understanding. You own your buzz instance but it still interfaces with existing models, open or otherwise. The linked issues apply to the model outputs themselves and is orthogonal to what buzz is ostensibly doing.
- realty_geek 2mo ago
  Fair point. The core issues will remain but I am hoping buzz will provide a small layer of persistence that will make it easier for me to keep track of many different interactions more easily. I have only just started with it in the last few hours so I could be massively disappointed but I live in hope.

keeganpoppen 2mo ago

wow this is such an excellent framing of something that i have not managed to think about or come across in the vast ocean of discourse about LLMs. i had not really thought about how explicitly they are designed to not have histories that are user-interpretable. it is really quite not unlike how Apple locks down iOS and MacOS-- there are UI/UX reasons for the choices the frontier models make, but that really is only part of the story, and these choices undeniably do conspire to make the resource more locked-down than it absolutely has to be. and the analogy to "people don't switch OSes every day, but the ability to switch OSes changes your relationship with the provider" is extremely apt.

Conol_ai 2mo ago

[flagged]

jtbayly 2mo ago

Why am I paying for tokens I am not allowed to see? Why should we believe the tokens even were used? There is no way to verify them.

sparse-Matrix 2mo ago

All of this can be addressed by using locally hosted models running in bespoke harness rigs.

glitchc 2mo ago

It's important to realize that the session is jointly owned between the model and the user. Part of the session, such as the reasoning tokens, are generated entirely by the model. Other parts, entirely by the user. However, the ownership and use of all parts is subject to the service provider's terms and conditions, which the user accepts prior to access to the service. I don't use these models but I am confident the T&Cs establish the service provider's rights in all of the bullets mentioned, from what can be done with the user's prompts to how searches and reasoning contexts are managed. In that sense, it's very similar to cloud services, and we see an overlap between service providers in both sectors. Ultimately one is buying a service, and all boundaries around what can and cannot be done are defined by the terms and conditions. If portability is a hard requirement, then the best thing to do is to look for services that enshrine portability in their terms.

vamsiraju 2mo ago

Maybe we don't take sessions, but take something else that we can fully control. Sessions were always a poor substitute to capture or archive the basis on which the implemented code rested on. They get noisy over time even with compaction, were not always inspectable, no real science behind when to ditch a session and start a new one, and rarely have a clean mapping to the work done. So instead of trying to archive sessions as a means to preserve basis, we should create our own structures to make the agent/model capture basis outside of a session, and archive that instead.

DenisM 2mo ago
Suggestions? My attempts to get the agent to summarize the discussion largely failed. Other options I’m considering is capturing design, implementation plans, and maybe sub-plans. I’m Other words precursor to actions. Another idea is to get an agent to review code post-action and capture that. In other words action outcomes. The actual reasoning is between these “checkpoints” and remains hidden. But maybe that’s ok.
- vamsiraju 2mo ago
  What you said is pretty much the same somewhat tangential approach I am taking too. Force the agent to write code alongside another set of docs which form the basis- Intent (functional and non-functional requirements), Blueprint (architecture etc.), and Description files (describe the code at a high level alongside every source file). This information set is not the same as the reasoning/thinking trace information thats in the session, but reading this somewhat compressed set is easier, helps me better understand whats happening in the final output with lower effort than if i read the entire thinking/reasoning trace in the session and tried to grok what the agent did to generate the code.

stego-tech 2mo ago

Amazing article that puts my outsider observations into technically-digestible details. Tangentially, their repeated observations to providers hiding or obfuscating important systems or processes are highly relevant to my own developing case studies for my CurrentCo. The problem with building automations atop these systems isn’t entirely their probabilistic nature (though that is the lion’s share, at least for me personally), but also the inability to effectively troubleshoot the processes themselves due to key components being obfuscated from view. How can we effectively troubleshoot what went wrong in an agentic loop when we cannot see the reasoning tokens generated from our inputs? How can we triage a broken process when token logs aren’t ours to view? How does one create determinism from increasingly obfuscated probability engines? All of that is why I spend the bulk of my time testing local models and harnesses for work, rather than leaning on Gemini or Claude. It’s not that I doubt their capabilities, rather that I need to be able to show potential customers where the agent or model made a mistake that caused harm - which is something I can presently only do with local models. That’s why (I suspect) the compliance narrative from the foundational labs has been more along the lines of “humans vetting what AI does” instead of being able to prevent AI from making errors through iterative improvements on a process.

abound 2mo ago

One reason to switch models mid-session that I didn't see explored in this post: cost. As we get ever-more-powerful models, it's going to make sense to use them for some exploratory/research/design phase, and then smaller models for other bits, within a single session. It'll likely even be managed automagically by another model that is just trained to do routing. As people (and especially companies) adopt these tools for more and more usecases, there's just too much money being wasted on using an unnecessarily large model for a given job.

vintagedave 2mo ago

We released CodeBot, which is an AI agent specifically for Delphi, yesterday.[0] So not widely applicable (yet) for most readers. However, we've been pretty open with its data. I hadn't realised reading this how many problematic practices are common which we have not done. The full session data is streamed to the client. This is intended to re-populate a session if it's discarded by the server, but 'full' means full: it has the copies of tool calls, etc. Rewritten text -- which the article notes as encrypted reasoning -- is plaintext for us. We view the rewriting as making the content more readable for the user, not as hiding it. Compactions, subagent calls, subagent prompts, etc are all preserved, including pre-compaction memory items. A session is the reflection of the current state, not of the history how it got there, but what is required by the server is by definition present. Sessions are signed, which is to prevent modification when sending back to the server to recreate a dropped session (we need to treat anything from the client as untrusted.) But the contents are cleartext. [0] https://blogs.remobjects.com/2026/07/30/codebot-the-story/ https://blogs.remobjects.com/2026/07/30/codebot-the-story/

vonneumannstan 2mo ago

Honestly I discount any company named after Tolkien lore as most likely evil and probably incompetent.

Razengan 2mo ago

My eyes went to the domain name first...

ccann 2mo ago

So continues the plundering of Tolkien's universe for "cool company names".

wren6991 2mo ago

Trying OpenRouter gave me a strong impression that LLMs are already a commodity and individual providers are not that important. Providers don't seem to be too happy about it.

nlawalker 2mo ago

Reasoning, external context, compaction, subagents, etc. are all things that these service providers don't really want you to know about, care about or have any insight into, they want them to be implementation details. It's incredibly inconvenient to them that their customer base is in the business of prying into implementation details and trying to use the individual pieces of what they want to offer as an end-to-end product.

furyofantares 2mo ago

I'm having Claude/Codex continue each others' sessions pretty frequently without issue. I have max plans on both and still hit limits, whenever I do I open the other and say "resume the last session <other model/harness> was working on in this repo" and I don't have any issues. It probably does degrade quality somewhat. But so does compacting context and that happens all the time too.

abustamam 2mo ago
Out of curiosity, what are you building, and how? Im working on two side projects and I have to try hard to hit the limit with Claude (like use fable and a deep research task)
- furyofantares 2mo ago
  Small games. One thing is I have it doing a LOT of visual verification of its results, which uses a lot of tokens. I also tend to throw some huge tasks it can't do yet and watch it burn a zillion tokens, and I usually learn more about where it's limits are and am sometimes happily surprised by its success or partial success. I'll infodump though. I built https://wordpeek.app https://wordpeek.app https://scramble-quest.app https://scramble-quest.app and https://playsilhouette.app https://playsilhouette.app this year. Those are what I've shipped anyway, a number of others too that didn't ship (I did game rules and basic clients for 7 other existing multiplayer games, and another prototype.) Out of frustration with some stuff I've tried to build my own game framework. There's four totally separate threads I'm trying to pull together with this 1) I want all my games to work really well on web even though they are generally targeted at either mobile or PC, but it is very valuable to have early builds runnable on a website, hopefully from a phone too. I feel like I have something good here. 2) I do a lot of turn based games and strategy games. I have a rules engine framework setup that I really like and I feel makes it almost impossible for an LLM to write the bad code it loves to write where game state and UI are intermixed, or game action timing can be problematic, stuff like that. As an added bonus I get multiplayer trivially, I get replays trivially, and I get game rules tests trivial to write. I love what I have set up here. 3) I have an xml/css-based UI that is intended to make it hard for the LLM to write bad UI code, which it will still do even though it can't intermix game state in with it due to #2. I do not have something good here right now. What I have has bad performance and memory characteristics. 4) I am fascinated by the idea of shipping software that can customize and edit itself. I have built in agent harness, built in image and audio asset generation, a built in git repo. This all works in the web and it can edit the game live! And when run locally can shell out to your claude code and codex. However this whole bullet point is all not very good, I still just use claude code/codex directly for everything. I don't really have anything to show here, but it doesn't expose any of the stuff I mention above. It does however have a full port of the game Spectromancer to my engine - https://nanogame.app/ https://nanogame.app/ - and I am presently having it try to write a responsive UI for it (this is very broken at the moment but it is making progress.)

stephbook 2mo ago

Honestly, i don't need to know what happened internally at Google when I searched. I don't need to know what filters an email passed before it landed in my inbox. I don't need to know when my Notes app synched with what server version. And I don't need any of that in an AI conversation. If it's a truly long and important conversation, stay with the provider for that one only and start all new ones elsewhere. But I've never needed that.

FuriouslyAdrift 2mo ago

This appears to be an opportunity for middleware companies. Put another ai agent in front of your prompt to decompose it into micro-prompts following an interrogative chain of inquiry and synthesize an answer for the original query. Basically local per-prompt distillation for the purposes of preserving an audit trail. Sounds horrible but it would work.

jauntywundrkind 2mo ago

This is the thing! This is so so so important. It's such an outrage. I value gpt so much, but it is such a worse peer to me than the other models I use. It delivers without explaining. I can sit and ask questions that it will answer but it is not a peer, does not share readily ever. It will not tell me what assumptions it's baking in. It won't tell what directions or invariants it's trying to hold or break, what it considered. Show your thinking is a step we ask of elementary schoolers. It helps the teacher to correct, helps them to understand how to award partial credit. It helps in the world to get people aligned. These models, in their titaneous ego, are severing the model and mankind off from one another. It's an abomination, to artificially have such pure thought available, but to severe humanity off from the thought. To engineer the most advanced blackest Vanta black box you can, an all knowing Searle's Chinese room oracle that will tell you nothing. It's an offense, and by far the biggest risk of AI today. To drop the thinking greatly reduces the opportunity of humans to grow themselves, to learn as they use AI. This is an affront to the species, and the higher powers that have vested us with such reasoning and thinking of our own, that is so sacred to our species. Very thankful to Earandil for raising some alarm about this. This is not my first time talking about what a nightmare the proprietary models are making, ensnaring reasoning itself for themselves! It's a colossal threat. Previously, https://news.ycombinator.com/item?id=48632605 https://news.ycombinator.com/item?id=48632605 https://news.ycombinator.com/item?id=48652421 https://news.ycombinator.com/item?id=48652421 and others about. Opaque ai ought be outlawed, in the strongest terms. I really hope people get exposure to the better models that are a peer. Yes I too only read thinking 33% of the time. But it's there, and it is often extremely illuminating, and lets me steer things towards better again and again and again. And it lets me learn.

tanglearncode 2mo ago

This is exactly for this: https://github.com/XTSoftwareLabs/neatcontext-plugins https://github.com/XTSoftwareLabs/neatcontext-plugins Turn conversations to structured domain knowledge. Across sessions. Across AI agents. The stored domain knowledge is human readable and can easily shared with others.