11 ms·

LLMs corrupt your documents when you delegate

479 points by rbanffy 6mo ago

simonreiff 6mo ago
[dead]
jonmoore 6mo ago
I really liked the evaluation method here - testing fidelity by round-tripping through chains of invertible steps. It was striking how even frontier models accumulated errors on seemingly computer-friendly tasks. It would be interesting to know if the stronger results on Python are not just an artefact of the Python-specific evaluation, if they carry over to other common general-purpose languages, and if they are driven by something specific in the training processes.
causal 6mo ago
Yeah I've been saying this for a while: AI-washing any text will degrade it, compounding with each pass. "Semantic ablation" is my favorite term for it: https://www.theregister.com/software/2026/02/16/semantic-ablation-why-ai-writing-is-boring-and-dangerous/4414930 https://www.theregister.com/software/2026/02/16/semantic-abl...
- polskibus 6mo ago
  By „with each pass” do you mean within the same session, or with new session (context window) each time?
  - sebastiennight 6mo ago
    In my experience, it happens with each edit of the document, whether or not you clear the context window. You can somewhat mitigate this, at the same moment you ask for the new edit, by adding new info or specifying the lost meaning you want to add back. But other things will still get washed out. Nuances will drift, sharp corners will be ablated. You're doing a Xerox copy of your latest Xerox copy, so even if you add your comments with a sharpie, anything that was there right before will be slightly blurrier in the next version.
    - atoav 6mo ago
      Which is why I think AI assisted writing is better then just letting it write the full text (if you care about the quality of the result). The act of writing isn't just the production of text, it is about wrangling a topic, rotating it in your mind and finding the perfect expression for a thought you have and that you want to convey to others. Some of those things can't be known by the LLM since you don't know them yourself by the point you started out. Often that thinking bit itself provides value to the person doing it, beyond the text itself. By letting a LLM do it for you, you rob yourself of the change of thought and the new findings you may encounter. Working with LLMs just makes it quicker to get going, bit you need to be a ruthless editor.
  - adampunk 6mo ago
    Each edit, even with unrelated edits. I had a README referring to something as "the cathedral of s*t" (some HN commentators don't care for the swearing, which is systemically bad news but w/e) and the robot would lift that phrase out in drive-bys, repeatedly. Occasionally it would report the action, sometimes it would not bother to report it. It never reached into the README on an unrelated doc edit, but if it was touching the README, that line was getting excised.
    - crooked-v 6mo ago
      That kind of passive-aggressive pseudo-moralizing is a common feature of all the current 'frontier' models. Try to do something like get one to summarize A Song of Ice and Fire text and it's likely to try and covertly sand off all the 'offensive' rough edges without even saying it's doing so.
- mohamedkoubaa 6mo ago
  I've been calling it meanwit reversion
- atoav 6mo ago
  It is X, not Y!
- m463 6mo ago
  code too. Someone AI'd and "extended" some code I wrote, and it did what day-1 jr programmer did - moved all the whitespace lines around and destroyed readability. Like "cleaning up" the mona lisa :)
cyanydeez 6mo ago
I played around with a local LLM to try and build a wiki like DAG. It made a lot of stupid errors from vague generic things like interpreting based on file names to not following redirects and placing the redirect response in them. I've also had them convert to markdown something like an excel formatted document. It worked pretty well as long as I was examining the output. But the longer it ran in context, the more likely it was to try in slip things in that seemed related but wasn't part of the break down. The only way I've found to mitigate some of it is to make every file a small-purpose built doc. This way you can definitely use git to revert changes but also limit the damage every time they touch them to the small context. Anyone who thinks they're a genius creating docs or updating them isnt actually reading the output.
- sebastiennight 6mo ago
  > I've also had them convert to markdown something like an excel formatted document. This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion. Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unwittingly inject their own biases and ideas into it whenever the source data is counter-intuitive to them.
  - trollbridge 6mo ago
    I see people cut and paste from Excep into a chat, as an image, and ask it to sum up numbers.
    - somewhatgoated 6mo ago
      I’ve seen people drink their own recycled piss and inject coffee into their ass - what’s your point?
      - sebastiennight 6mo ago
        In the first half, I thought you were an astronaut, but the second half has me double-guessing myself.
woeirua 6mo ago
It's an interesting paper, but I'd like to see a lot more about the types of errors that the LLM makes. Are they happening in the forward pass or the inverse pass? My guess is the inverse pass.
- daveguy 6mo ago
  This sounds like wishful thinking to me. The tasks are designed to be reversible. Whether it stochastic parrots in the forward direction or reverse direction is irrelevant. Especially considering these are inference engines. Every pass is a forward pass from the perspective of the LLM / agent. There is no feedback loop, and part of the reason why it's so easy for these things to mangle tasks. They are plausible sounding sentence/sequence generators.
adampunk 6mo ago
LLMs will make mistakes on every turn. The mistakes will have little to no apparent connection to "difficulty" or what may or may not be prevalent in the training data. They will be mistakes at all levels of operation, from planning to code writing to reporting. Whether those mistakes matter and whether you catch them is mostly up to you. I have yet to find a model that does not make mistakes each turn. I suspect that this kind of error is fundamentally incorrigible. The most interesting thing about LLMs is that despite the above (and its non-determinism) they're still useful.
- pyrolistical 6mo ago
  As a human I make typos all the time
  - adampunk 6mo ago
    I do too! I also make higher level design errors and get too enthusiastic about projects before code is written. We are, in a sense, fallible machines who have designed a planet-wide computational fabric around that fact.
    - junofan 6mo ago
      [flagged]
  - dangus 6mo ago
    A human can sit down and say “I’m going to make sure this is correct on the first pass and make sure I make an exact copy.” They have cognitive awareness of which tasks are highly critical and need more checking and re-checking without being prompted to think that way. For a human, time doesn’t stop when the first pass of the prompt and response is over. An LLM effectively wipes its memory of what it just did unless something is keeping track of a highly resource constrained context. An LLM is like an author of a book that immediately closes its eyes and wipes its memory after writing a chapter. Sure, it can pull some of that back in the next query via context, and it can regain context very quickly, but it effectively has no memory of the exact thing it just did. When a human is doing these tasks there is a lot of room for mistakes but there’s also a wildly higher capacity for flowing through time.
timacles 6mo ago
Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording here and there. LLMs are mean reversion machines, the more 'outside of their training' the context/work load they are currently dealing with, the more they will tend to gradually pull that into some homogenous abstract equilibrium
- Twirrim 6mo ago
  A coworker talks about LLMs as "bullshit" layers. Not exactly dismissing them or being derogatory about them, but emphasising that each time you feed something through an LLM, what comes out the other side may not be what you expect/want. Like that guy at the pub sharing what he'd seen online somewhere, after a few pints. Might be accurate, but carries notable risk it's not. So e.g., don't use an LLM to call an API to gather data and produce a report on it, as that's feeding deterministic data through a "bullshit" layer, meaning you can't trust what comes out the other side. Instead use the LLM to help you write the code that will produce a deterministic output from deterministic data. I've seen co-workers use LLMs to summarise deterministic data coming from APIs and have reports be wildly off the mark as often as they are accurate. Depending on what they're looking at that can have catastrophic risk.
  - ben_w 6mo ago
    Similar experience. I wouldn't say it even needs to be like some random person in the local pub: this behaviour is what you'd get from any game of telephone, book authors will say how you need to be blunt and direct about points in the book because readers will miss subtlety, anyone who has been quoted in a newspaper will have a story about the paper getting it wrong, etc. However, there's a reason pre-computing bureaucracy came with paper trails and meeting minutes getting written up, why court cases are increasingly cautious about the reliability of eye witnesses. It is ironic, the more AI becomes like us and less it acts like a traditional computer program, the worse it is at many things we want to use it for, but because collectively we're oblivious to our cognitive limitations we race into completely avoidable failures like this.
simonw 6mo ago
I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We note this is not an optimized state-of-the-art agent system; future work could explore more sophisticated harnesses. And yeah, their basic harness consists of read_file() and write_file() - that's just round-tripping with an extra step! The modern coding agent harnesses put a LOT of work into the design of their tools for editing files. My favorite current example of that is the Claude edit suite described here: https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool#text-editor-tool-commands https://platform.claude.com/docs/en/agents-and-tools/tool-us... The str_replace and insert commands are essential for avoiding round-trip risky edits of the whole file. They do at least provide a run_python() tool, so it's possible the better models figured out how to run string replacement using that. I'd like to see their system prompt and if it encouraged Python-based manipulation over reading and then writing the file. Update: found that harness code here https://github.com/microsoft/delegate52/blob/main/model_agentic.py https://github.com/microsoft/delegate52/blob/main/model_agen... The relevant prompt fragment is: You can approach the task in whatever way you find most effective: programmatically or directly by writing files As with so many papers like this, the results of the paper reflect more on the design of the harness that the paper's authors used than on the models themselves. I'm confident an experienced AI engineer / prompt engineer / pick your preferred title could get better results on this test by iterating on the harness itself.
- threethirtytwo 6mo ago
  People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have to ingest the document hundreds of times to achieve a state called "memorization". The equivalent in an LLM is called training. If you train a document into an LLM you can get parity with the memorized human edit in this case. But the above is irrelevant. The point is LLMs have certain similarities with humans. You need to design a harness such that an LLM edits a document the same way a human would: Search and surgical edits. All coding agents edit this way, so this paper isn't relevant.
meander_water 6mo ago
> We find that models are not failing due to “death by a thousand cuts” (i.e., many small errors). Instead, they main- tain near-perfect reconstruction in some rounds, and experience critical failures in a few rounds, typically losing 10-30+ points in a single round trip > We find that weaker models’ degradation originates primarily from content deletion, while frontier models’ degradation is attributable to corruption of content. I think we largely already knew this. This is why we fudge around with harnesses and temperature etc.
threethirtytwo 6mo ago
This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read the whole document. 2. They come up with a patch. A diff of the section they want to edit. 3. They change THAT section only. This is NOT what that experiment was doing. A 25% degradation rate would render the whole industry dead. No one would be using claude code because of that. The reality is... everyone is using claude code. AI is alien to the human brain, but in many ways it is remarkably. This is one aspect of similarity in that we cannot edit a whole document holistically to produce one edit. It has to be targeted surgical edits rather then a regurgitation of the entire document with said edit.
- leptons 6mo ago
  >IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. Except that isn't how humans edit documents, and it isn't how LLMs work either. When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory. They have the document, either physically printed out, or in a word processor. To make edits they either cross-out and write in the edit, or in a word processor just delete the text and replace it with something better. There's no need to keep the entire document in a human's memory for them to reproduce it from memory. The same goes for the LLM, it has access to the original document at all times. It can remove sections and replace them. But the LLM hallucinates. And if you give a document to a human high on LSD to edit, you might get some weird edits back.
  - threethirtytwo 6mo ago
    >Except that isn't how humans edit documents, Bro. That's my point. >and it isn't how LLMs work either. This is also my point. To be more technical about it, the harness around the LLM pushes it to do surgical edits rather then regurgitation, so my point is this experiment is garbage and testing an impractical and rarely used use case. >When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory. No shit sherlock. The point of that sentence was to illustrate the absurdity of doing that which in turn illustrates the absurdity of this scientific paper. You're kind of lost.
arian_ 6mo ago
[flagged]
carterschonwald 6mo ago
this is literally just “leave a child at the work computer with a real doc open playing office”. otoh it is good to design benchmarks tonground these things. on the flip side if you’re literally just using a bare bones harness on top of a stochastic parrot, of course stochastic errors accumulate. theres a lot of ways for improving text faithfulness through harness tool designs, and my incremental experiments seem promising. but unless work is gated on shit like “the script used must type checked ghc haskell or lean4”, unsupervised stuff is gonna decay
- rhubarbtree 6mo ago
  It’s not a stochastic parrot.
  - kgwgk 6mo ago
    It’s a stochastic goblin.
    - carterschonwald 6mo ago
      good one
twobitshifter 6mo ago
I thought this was going to be about a problem we saw recently. Someone used an LLM to update the comment block at the start of each source file, and the LLM programmed its own tool that ended up changing ALL of the line endings when it output again with the corrected comment block. Instead of an LLM we could have used find and replace, but people are thinking LLM is the only tool.
wtetzner 6mo ago
I think the problem is that we're using LLMs to do too much of the work. We should aim to design agents that use the LLM as the thinnest possible layer to translate the natural language intent into a deterministic process, minimizing round trips to the LLM as much as possible.
- mohamedkoubaa 6mo ago
  A process isn't automated until neither human nor genie is in the loop.
- whatisthiseven 6mo ago
  This becomes clear to anyone that wants to do marginally complex work. Developing pipelines that combine pre-processing flows, semantic targeting, and minimal contextual calls to an LLM API gets you powerful automated steps. Combined with separate validation steps, LLMs go from toys to useful.
danielvaughn 6mo ago
I've spent the last few months reading a lot of AI-generated code. It's extremely difficult. It's like how psychopaths are eerie because there's nothing behind their eyes. AI-generated code is eerie because there's nothing between the lines. Code is in some sense theory building, and when you read a humans code you can (mostly) feel their theory working in the background. LLMs have no such theory, the code is just facts strewn about. Very weird experience to try and understand it.
- glaslong 6mo ago
  Thank you I've had trouble articulating this sense, but it's strong. An uncanny valley.
- leptons 6mo ago
  My company is moving to a workflow where we only write Jira tickets, the LLM writes all the code and submits a PR. Then we are supposed to review the code the LLM wrote. I'm looking for a new job.
  - 8note 6mo ago
    that doesnt seem particularly horrible, as long as you as the engineer can still go change things in the code package and surrounding infrastructure to improve the output, and make sure that the agent is actually making the right stuff the first time you see the outputs eg. setting up better feedback loops, improving CI/CD, breaking changes up at the right scale, etc. you i assume also can then put in more work up front, doing simulations of solutions, lean proofs, and so on? more engineering, less plumbing
    - leptons 6mo ago
      The change is turning me from someone who writes pretty good reliable code, to someone who has to read and review pretty bad code. If you think this is an improvement, you're nuts. It is inserting a pretty unreliable middle-man know for errors and hallucinations, that often just goes down and stops working for reasons we can't control into a workflow that has worked well for a decade, and we're paying extra to really break-even on the time spent creating new code. Just because "everyone else is doing it". Not because it's proving to be a boon in productivity.
BrightGirl 6mo ago
[flagged]
andrewljohnson 6mo ago
LLM editing should be done to produce deterministic output. That is, the LLM should produce a diff, and the user should accept the diff. It seems like a bad pattern to just tell the LLM to edit any long document without that sort of visibility. Same goes for prose as for code.
- alansaber 6mo ago
  This gets skipped because continual approvals break up user flow so we let LLMs make a few hundred line diffs then a user does a bulk review, and can just revert all/partially. It's naieve to assume user will review every LOC in every instance.
  - andrewljohnson 6mo ago
    I’m fine with bulk review, it just has to get reviewed before a merge. You don’t need to review the LLM output as you work except as it aids you to work.
- julianlam 6mo ago
  I always thought it was a little weird that LLMs aren't sophisticated enough to surgically edit files as needed. For example, if there is a code block that needs to be wrapped within another function call, it'll rewrite the entire function call and you'll just have to pray that the re-written code block wasn't subtly changed. I _think_ so far it hasn't introduced any changes....
  - andrewljohnson 6mo ago
    You can just look at the diff when you do a pull request, no prayer needed, and if you want it to be “surgical” in that way, your prompt (and agents.md) can be specific. You can also unit test the function to better assure behavior didn’t change.
    - julianlam 6mo ago
      Indeed, that's what I do. I inspect the diff, though if it's an indentation change the entire block will be marked changed. Still not an excuse to not read every line of course... Unit tests give me the confidence that at least those tested logic paths are unaffected. Sometimes with older codebases one cannot assume the paths have adequate test coverage.
OfekSh 6mo ago
[flagged]
bigstrat2003 6mo ago
We don't need a study to tell us that LLMs always make mistakes. We already knew that. Anyone with sense is not using LLMs because of that.
rmwaite 6mo ago
What I find fascinating about LLMs is that a lot of their failures seem strikingly similar to the failures that humans struggle with. I’m not sure what this “means” but I think it’s interesting that we can theoretically fix these failures for LLMs but for humans it is much harder. You pretty much need to educate / indoctrinate people for their entire lives and even then it’s messy and unpredictable and prone to failure—just like LLMs.
buffaloPizzaBoy 6mo ago
I typically tell my agents to only treat document writing as a last "rendering" pass. LLMs are so good at taking sparse knowledge and compiling it, that I prefer to store knowledge as composable ideas/facts. What has worked well in practice is giving the agent a directory, and tell it to make independent markdown files for facts/findings it locates - with each file having front-matter for easy search-ability. This de-complects most tasks from "research AND store iteratively in a final document format" to more cohesive tasks "research a set of facts and findings which may be helpful for a document", and "assemble the document". Only a partial mitigation, but find it leads to more versatile re-use of findings, same as if a human was working.
- xstas1 6mo ago
  Sounds like a good system. To use the analogy from ths other comment, this would be like running an image through JPEG compression twice. The issue happens then if you're updating the individual research files on a regular basis. (Or making a long series of commits on a starting code base.) Every edit has a chance of doing a drive-by cleanup on nearby lines. Over a long enough timeline, it'll ablate your logic into something featureless, like if you compress an image too many times.
Bmello11 6mo ago
[flagged]
rao-v 6mo ago
May your contexts always be short
- tieTYT 6mo ago
  Before I read some of the study, I thought that was relevant too, but each "step [was] conducted as an independent, single-turn session."
  - rao-v 6mo ago
    Good point!
tmaly 6mo ago
When AI generates code, we have the ability to easily verify it and test it. The same is not so easy with free form text. I have been thinking about this mainly around when agents write plans or edit plans, but I think figuring out how to do this in general would be a huge breakthrough. Logical English was one idea I came across and Runcible https://runcible.com/ https://runcible.com/ was another idea I recently stumbled on.
y3ahd0g 6mo ago
Yeah so I run my agents as a different user that do not have write perms to my /home Then I can diff what they wrote with my copy Users are the OG container. On Linux it's possible to constrain a user to a network namespace, cgroups. BPF can be used like docker compose to ensure a service running under a user is running TL;DR a lot of the userspace cruft we import to run software has been rolled into the kernel over the last 10-15 years. Ignore the terminology "user". Under the hood all the same constraint and boundary setting you want exists without downloading the entire internet
clearstack 6mo ago
[flagged]
Amber-chen 6mo ago
[dead]
deferredgrant 6mo ago
[flagged]
madprops 6mo ago
Simple quick blind test for fun: https://w.merkoba.com/pickabot/ https://w.merkoba.com/pickabot/
GhostDriftInc 6mo ago
[flagged]
pickleRick243 6mo ago
With this paper by Microsoft and the infamous paper by Apple last year, it seems the tech giants that don't have their own models are getting a bit insecure.
charlie90 6mo ago
Doesnt this apply to humans as well? Thats why children play the game "Telephone" and watch as a message gets corrupted. The solution is to provide single source of truth.
xiaosong001 6mo ago
[flagged]
enrique_mendez 6mo ago
I'm making tools for fighting this kind of degradation: https://github.com/JigSpec/JigSpec https://github.com/JigSpec/JigSpec
wg0 6mo ago
> Delegation requires trust - the expectation that the LLM will faithfully execute the task without introducing errors into documents. We introduce DELEGATE-52 to study the readiness of AI systems in delegated workflows. DELEGATE-52 simulates long delegated workflows that require in-depth document editing across 52 professional domains, such as coding, crystallography, and music notation. Our large-scale experiment with 19 LLMs reveals that current models degrade documents during delegation: even frontier models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt an average of 25% of document content by the end of long workflows, with other models failing more severely. Additional experiments reveal that agentic tool use does not improve performance on DELEGATE-52, and that degradation severity is exacerbated by document size, length of interaction, or presence of distractor files. Our analysis shows that current LLMs are unreliable delegates: they introduce sparse but severe errors that silently corrupt documents, compounding over long interaction. That's why harnesses and prompting rituals using dozens of markdown down files do not work as advertised and is pretty much snake oil otherwise known as "agentic engineering". Also, the agentic engineering is pretty much so called prompt engineering except that prompt is now spread across dozens of markdown files directories.
Art9681 6mo ago
Remind yourselves that most research papers are written by career students with no real world practical experience. That is all.
- LPisGood 6mo ago
  Spending some time in and around applied research labs and seeing how poorly the sausage looks before it gets made into a paper is quite distressing. I’m sure there are labs out there doing excellent work (especially those focused on theory), but most of the applied research I’ve seen up close and personal is very poor indeed.
peter_retief 6mo ago
I am surprised that not more people talk about this, I once had an ssh key deleted, so unexpected it took me a while to debug. We live and learn. Still a huge fan though.
30030 6mo ago
[flagged]
tim-projects 6mo ago
You can get around the problem by doing a git diff of the unstaged file and a previous commit. This works well for code regressions but also works for document writing. I've automated it at this point. A case where using the CLI agent is much better than using the web chat.
30030 6mo ago
[flagged]
deleted 6mo ago
[deleted]
Ozzie-D 6mo ago
[flagged]
YeGoblynQueenne 6mo ago
In my experience there's no longer any good reason to post research papers investigating limitations of LLMs on HN any more because they are always met with one, or all, of the following arguments that have now taken the status of thought-terminating clichés: 1. It's an older model. 2. You're prompting it wrong. 3. That's not what LLMs are for. 4. We knew that already. It's as if there LLMs have no limitations, which of course goes completely against number 1 in the list above, because if LLMs have no limitations then how are newer models better and why are AI companies constantly releasing new versions? But the debate has taken on an insidious identitarian character: it's no longer about understanding a technology, its strengths, its limitations, what makes it tick. It's a fractious internet fight between crowds of users who have attached this or that opinion to their very internet persona and will not budge from their entrenched positions. That is basically the death of curious debate. Obviously there's no point in discussing any research under those conditions: good or bad, flawed or not, we're just not going to get any signal out of the noise on HN anymore.
toshikatsu-oga 6mo ago
[flagged]
handoflixue 6mo ago
> Our main experiment is a round-trip relay with N = 10 consecutive round-trips per environment, simulating 20 delegated interactions. In each interaction, the model receives all work environment documents as text in its context window in a single turn The LLM isn't being given an actual file system they can work with - they're expected to receive the document as text in the prompt, perform a task, and then re-output text into the conversation? Maybe I'm misunderstanding the methodology, but this feels a lot like the human game of Telephone - or perhaps, asking one to do a similar editing task using only Microsoft Outlook with copy/paste disabled. I'd imagine that one gets radically different results if one uses the appropriate desktop tools, just like humans do much better outside games of Telephone.
chris_explicare 6mo ago
[flagged]
fatso784 6mo ago
My problem with this kind of work is—-obviously they do. Did anyone seriously think otherwise? I’m shocked why these are even questions deserving scientific scrutiny. Have people truly lost their critical thinking that badly already?
- roryokane 6mo ago
  In history, why did scientists research gravity for so long? Were they too stupid to realize that they were obviously being pulled towards the ground? No. They hoped to learn about the details. Eventually they learned details that were not apparent from everyday experience, such as the formula for how gravity scales with mass. It’s the same here. For example, this study concluded that most changes are safe and some are very bad, as opposed to most changes being slightly bad. That is not obvious, especially to infrequent LLM users. Also, even “obvious” conclusions are within the scope of science. I’ve spent too long writing this already to look up an example, but I bet there have been countries in the past whose leaders chose “obviously-good” monetary policies that economic research could have shown was counterproductive. The world is complicated, and without systems of communication such as academia, it’s hard to be sure if what you see is what everyone else sees.
belabartok39 6mo ago
LLMs corrupt your documents. Proceeds to use AI to WRITE PAPER and DESIGN experiments.
theuniverseson 6mo ago
[flagged]
limack0 6mo ago
[flagged]
scottsiu 6mo ago
[flagged]

479 points by rbanffy 6mo ago

[dead]

I really liked the evaluation method here - testing fidelity by round-tripping through chains of invertible steps. It was striking how even frontier models accumulated errors on seemingly computer-friendly tasks. It would be interesting to know if the stronger results on Python are not just an artefact of the Python-specific evaluation, if they carry over to other common general-purpose languages, and if they are driven by something specific in the training processes.

causal 6mo ago

Yeah I've been saying this for a while: AI-washing any text will degrade it, compounding with each pass. "Semantic ablation" is my favorite term for it: https://www.theregister.com/software/2026/02/16/semantic-ablation-why-ai-writing-is-boring-and-dangerous/4414930 https://www.theregister.com/software/2026/02/16/semantic-abl...

polskibus 6mo ago
By „with each pass” do you mean within the same session, or with new session (context window) each time?
- sebastiennight 6mo ago
  In my experience, it happens with each edit of the document, whether or not you clear the context window. You can somewhat mitigate this, at the same moment you ask for the new edit, by adding new info or specifying the lost meaning you want to add back. But other things will still get washed out. Nuances will drift, sharp corners will be ablated. You're doing a Xerox copy of your latest Xerox copy, so even if you add your comments with a sharpie, anything that was there right before will be slightly blurrier in the next version.
  - atoav 6mo ago
    Which is why I think AI assisted writing is better then just letting it write the full text (if you care about the quality of the result). The act of writing isn't just the production of text, it is about wrangling a topic, rotating it in your mind and finding the perfect expression for a thought you have and that you want to convey to others. Some of those things can't be known by the LLM since you don't know them yourself by the point you started out. Often that thinking bit itself provides value to the person doing it, beyond the text itself. By letting a LLM do it for you, you rob yourself of the change of thought and the new findings you may encounter. Working with LLMs just makes it quicker to get going, bit you need to be a ruthless editor.
- adampunk 6mo ago
  Each edit, even with unrelated edits. I had a README referring to something as "the cathedral of s*t" (some HN commentators don't care for the swearing, which is systemically bad news but w/e) and the robot would lift that phrase out in drive-bys, repeatedly. Occasionally it would report the action, sometimes it would not bother to report it. It never reached into the README on an unrelated doc edit, but if it was touching the README, that line was getting excised.
  - crooked-v 6mo ago
    That kind of passive-aggressive pseudo-moralizing is a common feature of all the current 'frontier' models. Try to do something like get one to summarize A Song of Ice and Fire text and it's likely to try and covertly sand off all the 'offensive' rough edges without even saying it's doing so.
mohamedkoubaa 6mo ago
I've been calling it meanwit reversion
atoav 6mo ago
It is X, not Y!
m463 6mo ago
code too. Someone AI'd and "extended" some code I wrote, and it did what day-1 jr programmer did - moved all the whitespace lines around and destroyed readability. Like "cleaning up" the mona lisa :)

cyanydeez 6mo ago

I played around with a local LLM to try and build a wiki like DAG. It made a lot of stupid errors from vague generic things like interpreting based on file names to not following redirects and placing the redirect response in them. I've also had them convert to markdown something like an excel formatted document. It worked pretty well as long as I was examining the output. But the longer it ran in context, the more likely it was to try in slip things in that seemed related but wasn't part of the break down. The only way I've found to mitigate some of it is to make every file a small-purpose built doc. This way you can definitely use git to revert changes but also limit the damage every time they touch them to the small context. Anyone who thinks they're a genius creating docs or updating them isnt actually reading the output.

sebastiennight 6mo ago
> I've also had them convert to markdown something like an excel formatted document. This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion. Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unwittingly inject their own biases and ideas into it whenever the source data is counter-intuitive to them.
- trollbridge 6mo ago
  I see people cut and paste from Excep into a chat, as an image, and ask it to sum up numbers.
  - somewhatgoated 6mo ago
    I’ve seen people drink their own recycled piss and inject coffee into their ass - what’s your point?
    - sebastiennight 6mo ago
      In the first half, I thought you were an astronaut, but the second half has me double-guessing myself.

woeirua 6mo ago

It's an interesting paper, but I'd like to see a lot more about the types of errors that the LLM makes. Are they happening in the forward pass or the inverse pass? My guess is the inverse pass.

daveguy 6mo ago
This sounds like wishful thinking to me. The tasks are designed to be reversible. Whether it stochastic parrots in the forward direction or reverse direction is irrelevant. Especially considering these are inference engines. Every pass is a forward pass from the perspective of the LLM / agent. There is no feedback loop, and part of the reason why it's so easy for these things to mangle tasks. They are plausible sounding sentence/sequence generators.

adampunk 6mo ago

LLMs will make mistakes on every turn. The mistakes will have little to no apparent connection to "difficulty" or what may or may not be prevalent in the training data. They will be mistakes at all levels of operation, from planning to code writing to reporting. Whether those mistakes matter and whether you catch them is mostly up to you. I have yet to find a model that does not make mistakes each turn. I suspect that this kind of error is fundamentally incorrigible. The most interesting thing about LLMs is that despite the above (and its non-determinism) they're still useful.

pyrolistical 6mo ago
As a human I make typos all the time
- adampunk 6mo ago
  I do too! I also make higher level design errors and get too enthusiastic about projects before code is written. We are, in a sense, fallible machines who have designed a planet-wide computational fabric around that fact.
  - junofan 6mo ago
    [flagged]
- dangus 6mo ago
  A human can sit down and say “I’m going to make sure this is correct on the first pass and make sure I make an exact copy.” They have cognitive awareness of which tasks are highly critical and need more checking and re-checking without being prompted to think that way. For a human, time doesn’t stop when the first pass of the prompt and response is over. An LLM effectively wipes its memory of what it just did unless something is keeping track of a highly resource constrained context. An LLM is like an author of a book that immediately closes its eyes and wipes its memory after writing a chapter. Sure, it can pull some of that back in the next query via context, and it can regain context very quickly, but it effectively has no memory of the exact thing it just did. When a human is doing these tasks there is a lot of room for mistakes but there’s also a wildly higher capacity for flowing through time.

timacles 6mo ago

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording here and there. LLMs are mean reversion machines, the more 'outside of their training' the context/work load they are currently dealing with, the more they will tend to gradually pull that into some homogenous abstract equilibrium

Twirrim 6mo ago
A coworker talks about LLMs as "bullshit" layers. Not exactly dismissing them or being derogatory about them, but emphasising that each time you feed something through an LLM, what comes out the other side may not be what you expect/want. Like that guy at the pub sharing what he'd seen online somewhere, after a few pints. Might be accurate, but carries notable risk it's not. So e.g., don't use an LLM to call an API to gather data and produce a report on it, as that's feeding deterministic data through a "bullshit" layer, meaning you can't trust what comes out the other side. Instead use the LLM to help you write the code that will produce a deterministic output from deterministic data. I've seen co-workers use LLMs to summarise deterministic data coming from APIs and have reports be wildly off the mark as often as they are accurate. Depending on what they're looking at that can have catastrophic risk.
- ben_w 6mo ago
  Similar experience. I wouldn't say it even needs to be like some random person in the local pub: this behaviour is what you'd get from any game of telephone, book authors will say how you need to be blunt and direct about points in the book because readers will miss subtlety, anyone who has been quoted in a newspaper will have a story about the paper getting it wrong, etc. However, there's a reason pre-computing bureaucracy came with paper trails and meeting minutes getting written up, why court cases are increasingly cautious about the reliability of eye witnesses. It is ironic, the more AI becomes like us and less it acts like a traditional computer program, the worse it is at many things we want to use it for, but because collectively we're oblivious to our cognitive limitations we race into completely avoidable failures like this.

simonw 6mo ago

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We note this is not an optimized state-of-the-art agent system; future work could explore more sophisticated harnesses. And yeah, their basic harness consists of read_file() and write_file() - that's just round-tripping with an extra step! The modern coding agent harnesses put a LOT of work into the design of their tools for editing files. My favorite current example of that is the Claude edit suite described here: https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool#text-editor-tool-commands https://platform.claude.com/docs/en/agents-and-tools/tool-us... The str_replace and insert commands are essential for avoiding round-trip risky edits of the whole file. They do at least provide a run_python() tool, so it's possible the better models figured out how to run string replacement using that. I'd like to see their system prompt and if it encouraged Python-based manipulation over reading and then writing the file. Update: found that harness code here https://github.com/microsoft/delegate52/blob/main/model_agentic.py https://github.com/microsoft/delegate52/blob/main/model_agen... The relevant prompt fragment is: You can approach the task in whatever way you find most effective: programmatically or directly by writing files As with so many papers like this, the results of the paper reflect more on the design of the harness that the paper's authors used than on the models themselves. I'm confident an experienced AI engineer / prompt engineer / pick your preferred title could get better results on this test by iterating on the harness itself.

threethirtytwo 6mo ago
People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have to ingest the document hundreds of times to achieve a state called "memorization". The equivalent in an LLM is called training. If you train a document into an LLM you can get parity with the memorized human edit in this case. But the above is irrelevant. The point is LLMs have certain similarities with humans. You need to design a harness such that an LLM edits a document the same way a human would: Search and surgical edits. All coding agents edit this way, so this paper isn't relevant.

meander_water 6mo ago

> We find that models are not failing due to “death by a thousand cuts” (i.e., many small errors). Instead, they main- tain near-perfect reconstruction in some rounds, and experience critical failures in a few rounds, typically losing 10-30+ points in a single round trip > We find that weaker models’ degradation originates primarily from content deletion, while frontier models’ degradation is attributable to corruption of content. I think we largely already knew this. This is why we fudge around with harnesses and temperature etc.

threethirtytwo 6mo ago

This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read the whole document. 2. They come up with a patch. A diff of the section they want to edit. 3. They change THAT section only. This is NOT what that experiment was doing. A 25% degradation rate would render the whole industry dead. No one would be using claude code because of that. The reality is... everyone is using claude code. AI is alien to the human brain, but in many ways it is remarkably. This is one aspect of similarity in that we cannot edit a whole document holistically to produce one edit. It has to be targeted surgical edits rather then a regurgitation of the entire document with said edit.

leptons 6mo ago
>IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. Except that isn't how humans edit documents, and it isn't how LLMs work either. When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory. They have the document, either physically printed out, or in a word processor. To make edits they either cross-out and write in the edit, or in a word processor just delete the text and replace it with something better. There's no need to keep the entire document in a human's memory for them to reproduce it from memory. The same goes for the LLM, it has access to the original document at all times. It can remove sections and replace them. But the LLM hallucinates. And if you give a document to a human high on LSD to edit, you might get some weird edits back.
- threethirtytwo 6mo ago
  >Except that isn't how humans edit documents, Bro. That's my point. >and it isn't how LLMs work either. This is also my point. To be more technical about it, the harness around the LLM pushes it to do surgical edits rather then regurgitation, so my point is this experiment is garbage and testing an impractical and rarely used use case. >When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory. No shit sherlock. The point of that sentence was to illustrate the absurdity of doing that which in turn illustrates the absurdity of this scientific paper. You're kind of lost.

arian_ 6mo ago

[flagged]

carterschonwald 6mo ago

this is literally just “leave a child at the work computer with a real doc open playing office”. otoh it is good to design benchmarks tonground these things. on the flip side if you’re literally just using a bare bones harness on top of a stochastic parrot, of course stochastic errors accumulate. theres a lot of ways for improving text faithfulness through harness tool designs, and my incremental experiments seem promising. but unless work is gated on shit like “the script used must type checked ghc haskell or lean4”, unsupervised stuff is gonna decay

rhubarbtree 6mo ago
It’s not a stochastic parrot.
- kgwgk 6mo ago
  It’s a stochastic goblin.
  - carterschonwald 6mo ago
    good one

twobitshifter 6mo ago

I thought this was going to be about a problem we saw recently. Someone used an LLM to update the comment block at the start of each source file, and the LLM programmed its own tool that ended up changing ALL of the line endings when it output again with the corrected comment block. Instead of an LLM we could have used find and replace, but people are thinking LLM is the only tool.

wtetzner 6mo ago

I think the problem is that we're using LLMs to do too much of the work. We should aim to design agents that use the LLM as the thinnest possible layer to translate the natural language intent into a deterministic process, minimizing round trips to the LLM as much as possible.

mohamedkoubaa 6mo ago
A process isn't automated until neither human nor genie is in the loop.
whatisthiseven 6mo ago
This becomes clear to anyone that wants to do marginally complex work. Developing pipelines that combine pre-processing flows, semantic targeting, and minimal contextual calls to an LLM API gets you powerful automated steps. Combined with separate validation steps, LLMs go from toys to useful.

danielvaughn 6mo ago

I've spent the last few months reading a lot of AI-generated code. It's extremely difficult. It's like how psychopaths are eerie because there's nothing behind their eyes. AI-generated code is eerie because there's nothing between the lines. Code is in some sense theory building, and when you read a humans code you can (mostly) feel their theory working in the background. LLMs have no such theory, the code is just facts strewn about. Very weird experience to try and understand it.

glaslong 6mo ago
Thank you I've had trouble articulating this sense, but it's strong. An uncanny valley.
leptons 6mo ago
My company is moving to a workflow where we only write Jira tickets, the LLM writes all the code and submits a PR. Then we are supposed to review the code the LLM wrote. I'm looking for a new job.
- 8note 6mo ago
  that doesnt seem particularly horrible, as long as you as the engineer can still go change things in the code package and surrounding infrastructure to improve the output, and make sure that the agent is actually making the right stuff the first time you see the outputs eg. setting up better feedback loops, improving CI/CD, breaking changes up at the right scale, etc. you i assume also can then put in more work up front, doing simulations of solutions, lean proofs, and so on? more engineering, less plumbing
  - leptons 6mo ago
    The change is turning me from someone who writes pretty good reliable code, to someone who has to read and review pretty bad code. If you think this is an improvement, you're nuts. It is inserting a pretty unreliable middle-man know for errors and hallucinations, that often just goes down and stops working for reasons we can't control into a workflow that has worked well for a decade, and we're paying extra to really break-even on the time spent creating new code. Just because "everyone else is doing it". Not because it's proving to be a boon in productivity.

BrightGirl 6mo ago

[flagged]

andrewljohnson 6mo ago

LLM editing should be done to produce deterministic output. That is, the LLM should produce a diff, and the user should accept the diff. It seems like a bad pattern to just tell the LLM to edit any long document without that sort of visibility. Same goes for prose as for code.

alansaber 6mo ago
This gets skipped because continual approvals break up user flow so we let LLMs make a few hundred line diffs then a user does a bulk review, and can just revert all/partially. It's naieve to assume user will review every LOC in every instance.
- andrewljohnson 6mo ago
  I’m fine with bulk review, it just has to get reviewed before a merge. You don’t need to review the LLM output as you work except as it aids you to work.
julianlam 6mo ago
I always thought it was a little weird that LLMs aren't sophisticated enough to surgically edit files as needed. For example, if there is a code block that needs to be wrapped within another function call, it'll rewrite the entire function call and you'll just have to pray that the re-written code block wasn't subtly changed. I _think_ so far it hasn't introduced any changes....
- andrewljohnson 6mo ago
  You can just look at the diff when you do a pull request, no prayer needed, and if you want it to be “surgical” in that way, your prompt (and agents.md) can be specific. You can also unit test the function to better assure behavior didn’t change.
  - julianlam 6mo ago
    Indeed, that's what I do. I inspect the diff, though if it's an indentation change the entire block will be marked changed. Still not an excuse to not read every line of course... Unit tests give me the confidence that at least those tested logic paths are unaffected. Sometimes with older codebases one cannot assume the paths have adequate test coverage.

OfekSh 6mo ago

[flagged]

bigstrat2003 6mo ago

We don't need a study to tell us that LLMs always make mistakes. We already knew that. Anyone with sense is not using LLMs because of that.

rmwaite 6mo ago

What I find fascinating about LLMs is that a lot of their failures seem strikingly similar to the failures that humans struggle with. I’m not sure what this “means” but I think it’s interesting that we can theoretically fix these failures for LLMs but for humans it is much harder. You pretty much need to educate / indoctrinate people for their entire lives and even then it’s messy and unpredictable and prone to failure—just like LLMs.

buffaloPizzaBoy 6mo ago

I typically tell my agents to only treat document writing as a last "rendering" pass. LLMs are so good at taking sparse knowledge and compiling it, that I prefer to store knowledge as composable ideas/facts. What has worked well in practice is giving the agent a directory, and tell it to make independent markdown files for facts/findings it locates - with each file having front-matter for easy search-ability. This de-complects most tasks from "research AND store iteratively in a final document format" to more cohesive tasks "research a set of facts and findings which may be helpful for a document", and "assemble the document". Only a partial mitigation, but find it leads to more versatile re-use of findings, same as if a human was working.

xstas1 6mo ago
Sounds like a good system. To use the analogy from ths other comment, this would be like running an image through JPEG compression twice. The issue happens then if you're updating the individual research files on a regular basis. (Or making a long series of commits on a starting code base.) Every edit has a chance of doing a drive-by cleanup on nearby lines. Over a long enough timeline, it'll ablate your logic into something featureless, like if you compress an image too many times.

Bmello11 6mo ago

[flagged]

rao-v 6mo ago

May your contexts always be short

tieTYT 6mo ago
Before I read some of the study, I thought that was relevant too, but each "step [was] conducted as an independent, single-turn session."
- rao-v 6mo ago
  Good point!

tmaly 6mo ago

When AI generates code, we have the ability to easily verify it and test it. The same is not so easy with free form text. I have been thinking about this mainly around when agents write plans or edit plans, but I think figuring out how to do this in general would be a huge breakthrough. Logical English was one idea I came across and Runcible https://runcible.com/ https://runcible.com/ was another idea I recently stumbled on.

y3ahd0g 6mo ago

Yeah so I run my agents as a different user that do not have write perms to my /home Then I can diff what they wrote with my copy Users are the OG container. On Linux it's possible to constrain a user to a network namespace, cgroups. BPF can be used like docker compose to ensure a service running under a user is running TL;DR a lot of the userspace cruft we import to run software has been rolled into the kernel over the last 10-15 years. Ignore the terminology "user". Under the hood all the same constraint and boundary setting you want exists without downloading the entire internet

clearstack 6mo ago

[flagged]

Amber-chen 6mo ago

[dead]

deferredgrant 6mo ago

[flagged]

madprops 6mo ago

Simple quick blind test for fun: https://w.merkoba.com/pickabot/ https://w.merkoba.com/pickabot/

GhostDriftInc 6mo ago

[flagged]

pickleRick243 6mo ago

With this paper by Microsoft and the infamous paper by Apple last year, it seems the tech giants that don't have their own models are getting a bit insecure.

charlie90 6mo ago

Doesnt this apply to humans as well? Thats why children play the game "Telephone" and watch as a message gets corrupted. The solution is to provide single source of truth.

xiaosong001 6mo ago

[flagged]

enrique_mendez 6mo ago

I'm making tools for fighting this kind of degradation: https://github.com/JigSpec/JigSpec https://github.com/JigSpec/JigSpec

wg0 6mo ago

> Delegation requires trust - the expectation that the LLM will faithfully execute the task without introducing errors into documents. We introduce DELEGATE-52 to study the readiness of AI systems in delegated workflows. DELEGATE-52 simulates long delegated workflows that require in-depth document editing across 52 professional domains, such as coding, crystallography, and music notation. Our large-scale experiment with 19 LLMs reveals that current models degrade documents during delegation: even frontier models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt an average of 25% of document content by the end of long workflows, with other models failing more severely. Additional experiments reveal that agentic tool use does not improve performance on DELEGATE-52, and that degradation severity is exacerbated by document size, length of interaction, or presence of distractor files. Our analysis shows that current LLMs are unreliable delegates: they introduce sparse but severe errors that silently corrupt documents, compounding over long interaction. That's why harnesses and prompting rituals using dozens of markdown down files do not work as advertised and is pretty much snake oil otherwise known as "agentic engineering". Also, the agentic engineering is pretty much so called prompt engineering except that prompt is now spread across dozens of markdown files directories.

Art9681 6mo ago

Remind yourselves that most research papers are written by career students with no real world practical experience. That is all.

LPisGood 6mo ago
Spending some time in and around applied research labs and seeing how poorly the sausage looks before it gets made into a paper is quite distressing. I’m sure there are labs out there doing excellent work (especially those focused on theory), but most of the applied research I’ve seen up close and personal is very poor indeed.

peter_retief 6mo ago

I am surprised that not more people talk about this, I once had an ssh key deleted, so unexpected it took me a while to debug. We live and learn. Still a huge fan though.

30030 6mo ago

[flagged]

tim-projects 6mo ago

You can get around the problem by doing a git diff of the unstaged file and a previous commit. This works well for code regressions but also works for document writing. I've automated it at this point. A case where using the CLI agent is much better than using the web chat.

30030 6mo ago

[flagged]

deleted 6mo ago

[deleted]

Ozzie-D 6mo ago

[flagged]

YeGoblynQueenne 6mo ago

In my experience there's no longer any good reason to post research papers investigating limitations of LLMs on HN any more because they are always met with one, or all, of the following arguments that have now taken the status of thought-terminating clichés: 1. It's an older model. 2. You're prompting it wrong. 3. That's not what LLMs are for. 4. We knew that already. It's as if there LLMs have no limitations, which of course goes completely against number 1 in the list above, because if LLMs have no limitations then how are newer models better and why are AI companies constantly releasing new versions? But the debate has taken on an insidious identitarian character: it's no longer about understanding a technology, its strengths, its limitations, what makes it tick. It's a fractious internet fight between crowds of users who have attached this or that opinion to their very internet persona and will not budge from their entrenched positions. That is basically the death of curious debate. Obviously there's no point in discussing any research under those conditions: good or bad, flawed or not, we're just not going to get any signal out of the noise on HN anymore.

toshikatsu-oga 6mo ago

[flagged]

handoflixue 6mo ago

> Our main experiment is a round-trip relay with N = 10 consecutive round-trips per environment, simulating 20 delegated interactions. In each interaction, the model receives all work environment documents as text in its context window in a single turn The LLM isn't being given an actual file system they can work with - they're expected to receive the document as text in the prompt, perform a task, and then re-output text into the conversation? Maybe I'm misunderstanding the methodology, but this feels a lot like the human game of Telephone - or perhaps, asking one to do a similar editing task using only Microsoft Outlook with copy/paste disabled. I'd imagine that one gets radically different results if one uses the appropriate desktop tools, just like humans do much better outside games of Telephone.

chris_explicare 6mo ago

[flagged]

fatso784 6mo ago

My problem with this kind of work is—-obviously they do. Did anyone seriously think otherwise? I’m shocked why these are even questions deserving scientific scrutiny. Have people truly lost their critical thinking that badly already?

roryokane 6mo ago
In history, why did scientists research gravity for so long? Were they too stupid to realize that they were obviously being pulled towards the ground? No. They hoped to learn about the details. Eventually they learned details that were not apparent from everyday experience, such as the formula for how gravity scales with mass. It’s the same here. For example, this study concluded that most changes are safe and some are very bad, as opposed to most changes being slightly bad. That is not obvious, especially to infrequent LLM users. Also, even “obvious” conclusions are within the scope of science. I’ve spent too long writing this already to look up an example, but I bet there have been countries in the past whose leaders chose “obviously-good” monetary policies that economic research could have shown was counterproductive. The world is complicated, and without systems of communication such as academia, it’s hard to be sure if what you see is what everyone else sees.

belabartok39 6mo ago

LLMs corrupt your documents. Proceeds to use AI to WRITE PAPER and DESIGN experiments.

theuniverseson 6mo ago

[flagged]

limack0 6mo ago

[flagged]

scottsiu 6mo ago

[flagged]