25 ms·

An approach I like to help solving this is antagonistic or review agents. The first agent decides that eye glows turn NPCs into enemies, the second agent is ful

by pitched 3mo ago

An approach I like to help solving this is antagonistic or review agents. The first agent decides that eye glows turn NPCs into enemies, the second agent is fully dedicated to deciding if that is valid. If the review fails, it leaves notes and the original agent tries again.

katzenversteher 3mo ago
So far I only tried it with a single LLM in the dungeon master role. Your approach sounds promising (and I will definitly try it) but also a bit like a complicated workaround. What I mean: In games with humans the dungeon master is usually one person, not a whole council ;)
- deleted 3mo ago
  [deleted]
- cortesoft 3mo ago
  The magic of computers is that a complicated workaround can become modularized functionality very easiily.
  - pylotlight 3mo ago
    The cost there is multiple rounds of review tokens making it both slow and expensive.
    - 9fry 3mo ago
      Man-computer symbiosis is the void: mechanically extended man or AI. Distilling LLMs are a reversal of that.
    - fc417fc802 3mo ago
      There are many models that are neither slow nor expensive that are suitable for targeted review tasks.
    - pitched 3mo ago
      These things are still very far from human-level intelligence. Maybe a touch beyond a golden retriever in processing power. Ratcheting the intelligence up a notch from there is expensive. It’s a lot cheaper to simplify the problem it’s solving instead.
- fc417fc802 3mo ago
  But any competent human is also playing multiple roles mentally, mentally asking an entire series of questions about any new information. Discreet rounds of review emulate that. Write down the human process as a flow chart and then each interior node in the chart becomes a discreet review step with its own prompt.
gabriel666smith 3mo ago
This is also the best approach I've found thus far when I'm seeing how well LLMs can form narrative content. I don't frame its prompt as antagonistic though - I've found in the past (with weaker models, so YMMV) that this can be overly officious, sometimes blocking more creative outputs that you'd want to retain. The structure I've found that works best is to have six or seven agents chained, each roughly mimicking a part of the mind, or a role in film production. Broadly: - A high-temp "Id" agent, tuned to output only vaguely related noise. This really helps creativity. - An "Ego" agent, who receives the "Id" noise and is then given the initial response task. - A low-temp "Super-Ego" or "script supervisor" agent, who can grep back across longer contexts to check detail, and is asked to ensure that the initial response is within narrative reason. Not telling it that one role of the dialogue was "user" and one was "assistant" really helps with it not siding with the user. - A "continuity editor" agent, who is explicitly tasked with world and character lore-checking, building and updating character & world MD docs, etc. - A "prose editor" agent, whose sole task is to ensure it's tonally in-line with initial guidelines. You can add more as needed, depending on what is important to you. I think expecting competent narrative from a single model is a big ask. When writing and telling or performing a story, you have to engage several different parts of the brain, with very different tasks. The creative part of the brain has to have lots of bad ideas in it to surface a compelling idea; the parts dealing with immersion and/or realism have to incredibly restrained. The Id agent is very important. By appending 100 tokens of noise to a prompt asking: "Write a short story about [subject]", then asking an LLM to blindly score the short stories generated across a range of creativity metrics (such as they can exist!) I personally saw a ~40% score increase vs control over 3k short stories.
- maxignol 3mo ago
  I’ve never seen the Id approach before, that’s a good idea ! Though I was wondering how do you manage to keep costs low within the 7 agents ?
  - gabriel666smith 3mo ago
    The whole thing was borne out of wanting to keep costs low! My favoured approach (last time I was doing this) is using only a tiny sliding context window based on message pairs, rather than tokens, and only for the agents that need it. Amending prose style, for example, shouldn't need context beyond the message it's working on, and then its system prompt. For the models that require context, I personally found combining a tiny sliding window with a lazy version of the "Recursive Language Models" approach broke immersion least often and had a significantly lower cost. That + the "Id noise" + the strict agents also allowed cheaper models to overperform for me personally. My lazy version of the RLM approach is basically just giving the agent a grep tool across the full message history & "lore" documentation created by agents, combined with repeated, low-context turns, and a "submit answer" tool for when it felt like it had finished working. When I looked at the internals of what each agent turn looked like, it did look like a complete mess - but the context window only needs to surface the things it actually needs to know each turn. Short outputs help a lot with immersion, too - brevity means there is a lot less you can get wrong, and also aids response time & cost. It does take me an awful lot of prompt tuning to get what I want creatively from LLMs in any format, especially weaker models working in this chain, but I think that's likely always going to be true. Art can have rules, but that doesn't make it science :-) The RLM approach is detailed here, and I've found it really useful for any cost-sensitive/long-context task: https://alexzhang13.github.io/blog/2025/rlm/ https://alexzhang13.github.io/blog/2025/rlm/
binary132 3mo ago
If all or most LLMs are susceptible to failing this test, why would an LLM be a good means of evaluating performance on this test without being given a rubric?
- layer8 3mo ago
  Because evaluating a performance is different task from creating a performance. Someone who plays an instrument badly often has a different perception from someone who has to listen to it. ;)

by pitched 3mo ago

katzenversteher 3mo ago

So far I only tried it with a single LLM in the dungeon master role. Your approach sounds promising (and I will definitly try it) but also a bit like a complicated workaround. What I mean: In games with humans the dungeon master is usually one person, not a whole council ;)

deleted 3mo ago
[deleted]
cortesoft 3mo ago
The magic of computers is that a complicated workaround can become modularized functionality very easiily.
- pylotlight 3mo ago
  The cost there is multiple rounds of review tokens making it both slow and expensive.
  - 9fry 3mo ago
    Man-computer symbiosis is the void: mechanically extended man or AI. Distilling LLMs are a reversal of that.
  - fc417fc802 3mo ago
    There are many models that are neither slow nor expensive that are suitable for targeted review tasks.
  - pitched 3mo ago
    These things are still very far from human-level intelligence. Maybe a touch beyond a golden retriever in processing power. Ratcheting the intelligence up a notch from there is expensive. It’s a lot cheaper to simplify the problem it’s solving instead.
fc417fc802 3mo ago
But any competent human is also playing multiple roles mentally, mentally asking an entire series of questions about any new information. Discreet rounds of review emulate that. Write down the human process as a flow chart and then each interior node in the chart becomes a discreet review step with its own prompt.

gabriel666smith 3mo ago

This is also the best approach I've found thus far when I'm seeing how well LLMs can form narrative content. I don't frame its prompt as antagonistic though - I've found in the past (with weaker models, so YMMV) that this can be overly officious, sometimes blocking more creative outputs that you'd want to retain. The structure I've found that works best is to have six or seven agents chained, each roughly mimicking a part of the mind, or a role in film production. Broadly: - A high-temp "Id" agent, tuned to output only vaguely related noise. This really helps creativity. - An "Ego" agent, who receives the "Id" noise and is then given the initial response task. - A low-temp "Super-Ego" or "script supervisor" agent, who can grep back across longer contexts to check detail, and is asked to ensure that the initial response is within narrative reason. Not telling it that one role of the dialogue was "user" and one was "assistant" really helps with it not siding with the user. - A "continuity editor" agent, who is explicitly tasked with world and character lore-checking, building and updating character & world MD docs, etc. - A "prose editor" agent, whose sole task is to ensure it's tonally in-line with initial guidelines. You can add more as needed, depending on what is important to you. I think expecting competent narrative from a single model is a big ask. When writing and telling or performing a story, you have to engage several different parts of the brain, with very different tasks. The creative part of the brain has to have lots of bad ideas in it to surface a compelling idea; the parts dealing with immersion and/or realism have to incredibly restrained. The Id agent is very important. By appending 100 tokens of noise to a prompt asking: "Write a short story about [subject]", then asking an LLM to blindly score the short stories generated across a range of creativity metrics (such as they can exist!) I personally saw a ~40% score increase vs control over 3k short stories.

maxignol 3mo ago
I’ve never seen the Id approach before, that’s a good idea ! Though I was wondering how do you manage to keep costs low within the 7 agents ?
- gabriel666smith 3mo ago
  The whole thing was borne out of wanting to keep costs low! My favoured approach (last time I was doing this) is using only a tiny sliding context window based on message pairs, rather than tokens, and only for the agents that need it. Amending prose style, for example, shouldn't need context beyond the message it's working on, and then its system prompt. For the models that require context, I personally found combining a tiny sliding window with a lazy version of the "Recursive Language Models" approach broke immersion least often and had a significantly lower cost. That + the "Id noise" + the strict agents also allowed cheaper models to overperform for me personally. My lazy version of the RLM approach is basically just giving the agent a grep tool across the full message history & "lore" documentation created by agents, combined with repeated, low-context turns, and a "submit answer" tool for when it felt like it had finished working. When I looked at the internals of what each agent turn looked like, it did look like a complete mess - but the context window only needs to surface the things it actually needs to know each turn. Short outputs help a lot with immersion, too - brevity means there is a lot less you can get wrong, and also aids response time & cost. It does take me an awful lot of prompt tuning to get what I want creatively from LLMs in any format, especially weaker models working in this chain, but I think that's likely always going to be true. Art can have rules, but that doesn't make it science :-) The RLM approach is detailed here, and I've found it really useful for any cost-sensitive/long-context task: https://alexzhang13.github.io/blog/2025/rlm/ https://alexzhang13.github.io/blog/2025/rlm/

binary132 3mo ago

If all or most LLMs are susceptible to failing this test, why would an LLM be a good means of evaluating performance on this test without being given a rubric?

layer8 3mo ago
Because evaluating a performance is different task from creating a performance. Someone who plays an instrument badly often has a different perception from someone who has to listen to it. ;)