4 ms·

> done by my clankers or clankers of other people I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users,

> done by my clankers or clankers of other people I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users, project docs, business plan docs. A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff. They're able to generate this stuff way faster than we can review. It used to be that it would take a significant part of a day for a project manager to come up with a planning doc - now they can generate one in a few minutes and send it out for review. It's just really tiring.

PsylentKnight 3mo ago
Thats gonna be a no from me dog. I don’t expect anyone to read something I didn’t read myself
aakresearch 3mo ago
Wait, what? I thought everyone agrees that modern models post September 2025 (or whenever Opus or whatever 5.6789 was released) do not hallucinate, make things up, contradict themselves and can review their own output into perfection regardless of task, goal or context???? /s
- UncleOxidant 3mo ago
  In general I think from the coding side they're more robust now. However, people generating docs are maybe not as experienced with how to prompt in ways that avoid having the LLM tell you what you want to hear. I think this is still a pitfall that can easily be fallen into. Those of us who are doing LLM-assisted coding for the last couple of years are more aware of this now. Those who are planning/management folks are still kind of susceptible depending on how much experience they've had dealing with LLMs.
- KronisLV 3mo ago
  What a take with no nuance. > do not hallucinate They do, just less. To the degree of being usable, as long as there are guardrails and they're used responsibly. For example, if there's code being output, there should be type checking and compilation, as well as code tests that prove that it works or that it doesn't - seeing how abysmal code coverage is in most of the projects I've seem, for whatever reason people thought that they didn't really need it much. They were wrong. This also implies you need SOTA models on max reasoning. > make things up Same as above. Ideally you'd give them some way to verify their claims, like web search or browsing and referencing docs, Jira tickets etc., basically improve the signal to noise ratio. > contradict themselves They do so way less than before, as long as the above is true. > can review their own output into perfection They are pretty good at reviewing things, especially if you make them do adversarial review! It will never be perfect, but can be close in quality to human output (e.g. the code they produce, when used properly and with intent, is better than the code I've seen many developers write and ship before LLMs were a thing). This also more or less scales with how much compute you give them - three parallel review agents will turn one output artifact into something good with higher confidence than two, and definitely better than with no review. There's a cost vs quality balance and it seems that all those xhigh and max reasoning modes are still geared way too much towards cost, instead of quality. So you have to make up for that shortcoming yourself. > regardless of task, goal or context???? Garbage in, garbage out. I won't be an asshole and say that you're holding it wrong, nor will I say that anyone should listen to the claims marketing AI (absolutely delusional takes, meant to attract investors), but we're slowly getting to a better position in regards to LLMs, year by year. It's just a shame that the peak of inflated expectations hit while the technology still hasn't fully plateaued and reached whatever its ceiling is. I probably also shouldn't ignore the fact that some people will not care about any of it and send AI generated slop verbatim and to an outside observer there's no way to easily tell apart the difference between the two, unless you make a technical report contain exact references to where the data is sourced from, for example (and then either verify the references yourself, or make another agent do it).
  - MaKey 3mo ago
    I think you missed the /s (for sarcasm) at the end.
sdesol 3mo ago
> now they can generate one in a few minutes and send it out for review. I think we will very soon move to a prove to me you've read it protocol and/or introduce speed bumps to slow things down.
jaggederest 3mo ago
Feed it into an AI and ask it to adversarially criticize it, doc for doc, send back 30 responses in a zip folder, wipe hands on pants, return to HN.
- da_grift_shift 3mo ago
  This may actually be a solid way to tackle the bullshit asymmetry problem caused by drive-by LLM sloppers.
  - jaggederest 3mo ago
    I always laugh because I've been practicing prompting every single day for the last few years, if they want to start a prompt fight, brother let me at 'em.
- nswango 3mo ago
  Don't know if you are serious, but why become part of the problem? Why not just review a single document quickly, find an error which invalidates the document, and send it back saying "Policy paper 1 mentions X as being on the business plan for Y, it's not on the plan, please can you fix."
  - reed1234 3mo ago
    Why do your colleagues work when they could at least attempt it first?
  - zufallsheld 3mo ago
    Because they'll just paste your remarks in their llm, let it correct the text and send it back to you.
    - deleted 3mo ago
      [deleted]
  - lesostep 3mo ago
    Because they will fix it and send again. Unless you can write a good-sounding reason why it's on them to review a LLM output before sending it to you, they will outsources this reviewing to you, and it's a lot of reviewing.
soonnow 3mo ago
Yeah I heard a similar thing recently at a presentation. At that point wouldn't it be easier to just send the prompt around?
intended 3mo ago
Rate of generation/Rate of verification is a proxy for signal to noise ratios, just for work. That ratio has changed, and verification is the hard part. Verification is the point of all markets (and a decent part of human civ as well). And review isn’t cost less - https://en.wikipedia.org/wiki/Ironies_of_Automation https://en.wikipedia.org/wiki/Ironies_of_Automation
- layer8 3mo ago
  > Rate of generation/Rate of verification is a proxy for signal to noise ratios Hopefully you mean Rate of verification/Rate of generation.
  - intended 3mo ago
    Yes! it should be: Verification/generation
groby_b 3mo ago
Write an LLM script to review them. Tell it to find at least three severe issues. Set to auto-reply. He who brings the slop cannon shall be drowned by slop rain.
- tanseydavid 3mo ago
  > He who brings the slop cannon shall be drowned by slop rain. If this approach gets widely adopted, then I think you should probably start building an ARK. Make it big enough to hold two of every animal species. /s
AgentMatt 3mo ago
Seems like for such requests it's necessary to get some proof of work: require a meeting where for every artifact they sent you to review, they briefly explain the gist and point out the motivation for creating the artifact.
- AnimalMuppet 3mo ago
  Side benefit: They get public humiliation for the problems in what they sent around. It could create some social pressure to not send out garbage.
mathw 3mo ago
The only way to even start to counter that is to make it a firm company policy that if you use an LLM to hallucinate any documents you absolutely must thoroughly review them yourself before you send them to anybody else, and that you are still responsible for the quality of LLM-generated content. Getting an LLM to vomit out a bunch of documents and sending them straight to another colleague is absolutely unacceptable behaviour.
- theideaofcoffee 3mo ago
  This is going to just run up against the insanity that is tokenmaxxing every moment of the day. When people are incentivized (upon pain of firing) to get the LLM to vomit out as much as possible, they're hardly going to stop and ponder if schlepping the slop over the wall is acceptable if the alternative is a pink slip. Which is going to win?
  - thinkingemote 3mo ago
    We employees need to remember that most software projects fail. So the work we produce should have lower value than we give it. We also need to be motivated to stay in our jobs. Most developers like their projects and value their work. But the chances are that it's for nothing. Many developers know they work on bad products (gambling industry, military, surveillance, whatever) and so it's here that they focus on their technologies, tools and frameworks rather than the work they produce. "Agentic engineering" for example. Id be curious to see what and how Googlers are doing with their 20% time.
    - soulofmischief 3mo ago
      The 20% policy at Google is effectively dead.
  - AnimalMuppet 3mo ago
    If tokenmaxxing wins in your company, your company is going to lose. There's an external reality out there, outside your company, and your company has to produce things that actually work out there. Hallucinated AI slop does not help you do that. It leads you to unworkable plans, and if the plans produce, they produce unsellable products. If you're an employee in that situation, push back if you can. If you can't, put your resume on the street. (But that may not work, these days. If it doesn't, all I can say is ride it out as best you can, and try to maintain both your job and your sanity. How? I don't know.)
jjav 3mo ago
> I'm getting so many requests to review LLM-generated documents That's the other nightmare of AI slop. So easy to generate endless content. Who will review? Just today the boss request I review slides for a presentation. But it's all AI slop, generated from querying tickets and docs and who knows what. It's mostly sort of correct but also plenty misleading and incorrect. So now I have to fact check all this slop which will take hours (even with my AI assistance) and rewrite most of it. If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.
- AnimalMuppet 3mo ago
  > If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting. Maybe, depending on the boss. Some would have spent five minutes describing what they wanted, and someone else would have spent three hours creating the deck.
  - jjav 3mo ago
    I'm fine with that. The company doesn't have infinite people so as long as someone spends 3 hours generating and I spend 5-15 minutes reviewing, that's fine. Problem with AI is that generation is so many orders of magnitude faster than reviewing so it's basically infinite monkeys on infinite typewriters.
freefaler 3mo ago
I got this problem with my own employees, LLM are fine, but lazy slop is not permitted. Current idea is to have a clear "best practice" template for most of the research/specs/problem definition they submit and it reduced the slop to a manageable level. But this might work in a smaller company where the management is reading and is strict about these things.
chadgpt3 3mo ago
It's spam, it's a DoS attack. The right way to handle a DoS attack is to blacklist the sender. But this doesn't work if the sender is paying you.
palmotea 3mo ago
> A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff You just discovered the unlock to massive AI-driven productivity increases: outsource the hard stuff to others, or just don't do it at all. Keep the easy tasks that generate a big volume of output for yourself.
deterministic 3mo ago
Reject documents that contain LLM hallucinations. Or use LLM's to generate 12+ pages of detailed reviews of those documents and return to sender.
senectus1 3mo ago
I find myself telling co-workers that if they couldn't be bothered writeing the email/document, why should i have to be bothered reading the document/email. this is just spam, people sending unsolcited data at you and expecting you to swoollow and process that data. its just rude and unreasonable, not to mention an unconscious (hopefully) act of sabotage.

PsylentKnight 3mo ago

Thats gonna be a no from me dog. I don’t expect anyone to read something I didn’t read myself

aakresearch 3mo ago

Wait, what? I thought everyone agrees that modern models post September 2025 (or whenever Opus or whatever 5.6789 was released) do not hallucinate, make things up, contradict themselves and can review their own output into perfection regardless of task, goal or context???? /s

UncleOxidant 3mo ago
In general I think from the coding side they're more robust now. However, people generating docs are maybe not as experienced with how to prompt in ways that avoid having the LLM tell you what you want to hear. I think this is still a pitfall that can easily be fallen into. Those of us who are doing LLM-assisted coding for the last couple of years are more aware of this now. Those who are planning/management folks are still kind of susceptible depending on how much experience they've had dealing with LLMs.
KronisLV 3mo ago
What a take with no nuance. > do not hallucinate They do, just less. To the degree of being usable, as long as there are guardrails and they're used responsibly. For example, if there's code being output, there should be type checking and compilation, as well as code tests that prove that it works or that it doesn't - seeing how abysmal code coverage is in most of the projects I've seem, for whatever reason people thought that they didn't really need it much. They were wrong. This also implies you need SOTA models on max reasoning. > make things up Same as above. Ideally you'd give them some way to verify their claims, like web search or browsing and referencing docs, Jira tickets etc., basically improve the signal to noise ratio. > contradict themselves They do so way less than before, as long as the above is true. > can review their own output into perfection They are pretty good at reviewing things, especially if you make them do adversarial review! It will never be perfect, but can be close in quality to human output (e.g. the code they produce, when used properly and with intent, is better than the code I've seen many developers write and ship before LLMs were a thing). This also more or less scales with how much compute you give them - three parallel review agents will turn one output artifact into something good with higher confidence than two, and definitely better than with no review. There's a cost vs quality balance and it seems that all those xhigh and max reasoning modes are still geared way too much towards cost, instead of quality. So you have to make up for that shortcoming yourself. > regardless of task, goal or context???? Garbage in, garbage out. I won't be an asshole and say that you're holding it wrong, nor will I say that anyone should listen to the claims marketing AI (absolutely delusional takes, meant to attract investors), but we're slowly getting to a better position in regards to LLMs, year by year. It's just a shame that the peak of inflated expectations hit while the technology still hasn't fully plateaued and reached whatever its ceiling is. I probably also shouldn't ignore the fact that some people will not care about any of it and send AI generated slop verbatim and to an outside observer there's no way to easily tell apart the difference between the two, unless you make a technical report contain exact references to where the data is sourced from, for example (and then either verify the references yourself, or make another agent do it).
- MaKey 3mo ago
  I think you missed the /s (for sarcasm) at the end.

sdesol 3mo ago

> now they can generate one in a few minutes and send it out for review. I think we will very soon move to a prove to me you've read it protocol and/or introduce speed bumps to slow things down.

jaggederest 3mo ago

Feed it into an AI and ask it to adversarially criticize it, doc for doc, send back 30 responses in a zip folder, wipe hands on pants, return to HN.

da_grift_shift 3mo ago
This may actually be a solid way to tackle the bullshit asymmetry problem caused by drive-by LLM sloppers.
- jaggederest 3mo ago
  I always laugh because I've been practicing prompting every single day for the last few years, if they want to start a prompt fight, brother let me at 'em.
nswango 3mo ago
Don't know if you are serious, but why become part of the problem? Why not just review a single document quickly, find an error which invalidates the document, and send it back saying "Policy paper 1 mentions X as being on the business plan for Y, it's not on the plan, please can you fix."
- reed1234 3mo ago
  Why do your colleagues work when they could at least attempt it first?
- zufallsheld 3mo ago
  Because they'll just paste your remarks in their llm, let it correct the text and send it back to you.
  - deleted 3mo ago
    [deleted]
- lesostep 3mo ago
  Because they will fix it and send again. Unless you can write a good-sounding reason why it's on them to review a LLM output before sending it to you, they will outsources this reviewing to you, and it's a lot of reviewing.

soonnow 3mo ago

Yeah I heard a similar thing recently at a presentation. At that point wouldn't it be easier to just send the prompt around?

intended 3mo ago

Rate of generation/Rate of verification is a proxy for signal to noise ratios, just for work. That ratio has changed, and verification is the hard part. Verification is the point of all markets (and a decent part of human civ as well). And review isn’t cost less - https://en.wikipedia.org/wiki/Ironies_of_Automation https://en.wikipedia.org/wiki/Ironies_of_Automation

layer8 3mo ago
> Rate of generation/Rate of verification is a proxy for signal to noise ratios Hopefully you mean Rate of verification/Rate of generation.
- intended 3mo ago
  Yes! it should be: Verification/generation

groby_b 3mo ago

Write an LLM script to review them. Tell it to find at least three severe issues. Set to auto-reply. He who brings the slop cannon shall be drowned by slop rain.

tanseydavid 3mo ago
> He who brings the slop cannon shall be drowned by slop rain. If this approach gets widely adopted, then I think you should probably start building an ARK. Make it big enough to hold two of every animal species. /s

AgentMatt 3mo ago

Seems like for such requests it's necessary to get some proof of work: require a meeting where for every artifact they sent you to review, they briefly explain the gist and point out the motivation for creating the artifact.

AnimalMuppet 3mo ago
Side benefit: They get public humiliation for the problems in what they sent around. It could create some social pressure to not send out garbage.

mathw 3mo ago

The only way to even start to counter that is to make it a firm company policy that if you use an LLM to hallucinate any documents you absolutely must thoroughly review them yourself before you send them to anybody else, and that you are still responsible for the quality of LLM-generated content. Getting an LLM to vomit out a bunch of documents and sending them straight to another colleague is absolutely unacceptable behaviour.

theideaofcoffee 3mo ago
This is going to just run up against the insanity that is tokenmaxxing every moment of the day. When people are incentivized (upon pain of firing) to get the LLM to vomit out as much as possible, they're hardly going to stop and ponder if schlepping the slop over the wall is acceptable if the alternative is a pink slip. Which is going to win?
- thinkingemote 3mo ago
  We employees need to remember that most software projects fail. So the work we produce should have lower value than we give it. We also need to be motivated to stay in our jobs. Most developers like their projects and value their work. But the chances are that it's for nothing. Many developers know they work on bad products (gambling industry, military, surveillance, whatever) and so it's here that they focus on their technologies, tools and frameworks rather than the work they produce. "Agentic engineering" for example. Id be curious to see what and how Googlers are doing with their 20% time.
  - soulofmischief 3mo ago
    The 20% policy at Google is effectively dead.
- AnimalMuppet 3mo ago
  If tokenmaxxing wins in your company, your company is going to lose. There's an external reality out there, outside your company, and your company has to produce things that actually work out there. Hallucinated AI slop does not help you do that. It leads you to unworkable plans, and if the plans produce, they produce unsellable products. If you're an employee in that situation, push back if you can. If you can't, put your resume on the street. (But that may not work, these days. If it doesn't, all I can say is ride it out as best you can, and try to maintain both your job and your sanity. How? I don't know.)

jjav 3mo ago

> I'm getting so many requests to review LLM-generated documents That's the other nightmare of AI slop. So easy to generate endless content. Who will review? Just today the boss request I review slides for a presentation. But it's all AI slop, generated from querying tickets and docs and who knows what. It's mostly sort of correct but also plenty misleading and incorrect. So now I have to fact check all this slop which will take hours (even with my AI assistance) and rewrite most of it. If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.

AnimalMuppet 3mo ago
> If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting. Maybe, depending on the boss. Some would have spent five minutes describing what they wanted, and someone else would have spent three hours creating the deck.
- jjav 3mo ago
  I'm fine with that. The company doesn't have infinite people so as long as someone spends 3 hours generating and I spend 5-15 minutes reviewing, that's fine. Problem with AI is that generation is so many orders of magnitude faster than reviewing so it's basically infinite monkeys on infinite typewriters.

freefaler 3mo ago

I got this problem with my own employees, LLM are fine, but lazy slop is not permitted. Current idea is to have a clear "best practice" template for most of the research/specs/problem definition they submit and it reduced the slop to a manageable level. But this might work in a smaller company where the management is reading and is strict about these things.

chadgpt3 3mo ago

It's spam, it's a DoS attack. The right way to handle a DoS attack is to blacklist the sender. But this doesn't work if the sender is paying you.

palmotea 3mo ago

> A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff You just discovered the unlock to massive AI-driven productivity increases: outsource the hard stuff to others, or just don't do it at all. Keep the easy tasks that generate a big volume of output for yourself.

deterministic 3mo ago

Reject documents that contain LLM hallucinations. Or use LLM's to generate 12+ pages of detailed reviews of those documents and return to sender.

senectus1 3mo ago

I find myself telling co-workers that if they couldn't be bothered writeing the email/document, why should i have to be bothered reading the document/email. this is just spam, people sending unsolcited data at you and expecting you to swoollow and process that data. its just rude and unreasonable, not to mention an unconscious (hopefully) act of sabotage.