3 ms·

Setting aside the choice of LLM, the constraint that the LLM must maintain a world-model-as-knowledge-graph solely by reading and re-reading its own chat histor

by btown 1y ago

Setting aside the choice of LLM, the constraint that the LLM must maintain a world-model-as-knowledge-graph solely by reading and re-reading its own chat history seems to be a less interesting experiment than providing it with tools that let it develop that world model explicitly? On page 5, Figure 1, the authors create a hand-written diagram showing the relationship between objects as a graph showing the directionality of edges in 3D space. To me, this implies that you could supply your LLM with a set of tools like getObjectsInGraph, updateGraphRelatingObjectPair, findObjectsRelativeToObject, describePathBetweenObjectsByName... and allow it to maintain that diagram as a structured DAG, and continually ask the game engine questions that let it update that graph in an agentic way. My prediction would be that they'd recreate that diagram, and enable goal seeking, with high fidelity. Asking an LLM to work without being able to "visualize" and "touch" its environment in its "mind's eye" is tying a hand behind its back. But I'm bullish that we'll find increasingly better ways of adapting 3D/4D world models into textual tools in a way that rapidly changes the possibilities of what LLMs can do.

daxfohl 1y ago
Or even just a notepad. It's well established that long context histories with scattered information are hard for LLMs to navigate. To distinguish whether it's using notes or context history, you could simply delete the context after each turn. The prompt could be something like "you're taking over this game from a previous player who has compiled these notes. (insert notes here). Play one turn, and update the notes with any new knowledge you have attained, relationships you have identified, inaccuracies you have confirmed, hypotheses you have, or anything else you this would be useful, so that the next player will be able use these notes to make the best next move.", and just clear the context after each move. Maybe also say there's a limit to the number of words on the notepad so that it doesn't just flood the notes with irrelevant information. For future iterations, maybe also give it a bitmap or svg canvas, or a database, or a code interpreter, and see if it uses any of those tools at all.
- daxfohl 1y ago
  I went ahead and did this, just using a maze where the only actions are "turn left". "turn right", and "go forward". The response is either OK or BLOCKED, and additionally which directions are open and which are walled in the current cell (relative directions: front, back, left, right. Not north east south west). When having it try to retain context of where it is via chat history alone, it failed. It went back into fully-explored paths and explored them again. When giving it a scratchpad to store knowledge, it didn't do any better. At least one common failure pattern I noticed is it got confused and updated its position when issuing a "turn" command. It never created a map, no matter how strongly I recommended doing so in the prompt, but generally just stored a list of cells it went to, and what it saw in each cell. I'd have played with it more and think I could have eventually gotten it working 99% better, but I'd already spent $12 and I'm unemployed so. Anyway, even if I got it working really really well, it'd still just be another example of "don't use an LLM when a simple state machine would be easier and work better". I think an LLM could be useful if the response from the maze game was natural language: "You went forward a step, to find yourself a T-intersection with passages off to each side.", and have the LLM translate that to a structured response that could be fed into the state machine. But it's still not reliable enough to serve as the state machine itself.
theptip 1y ago
This is a well-debated point; should you test the system on its own, or system+scaffold? How much custom scaffolding is allowed? Both avenues are interesting. For AGI presumably the “general” bit means you don’t get to build task-specific scaffolding ahead of time (though you can certainly build generic scaffolds like memory or knowledge layers). For safety/capability research, system+scaffolding is often more interesting because that is the frontier; if you conclude “LLMs cannot world-model” but actually LLM+CoT+memory can world model in a specific domain you care about, then you might underestimate capabilities and therefore deployment risk. The general point being: capabilities are jagged and prompt-dependent, just because you failed to elicit a capability, doesn’t mean you proved it cannot be elicited.
- daxfohl 1y ago
  FWIW I just tried making a maze game and having an LLM solve it step by step, where each move the LLM has to choose from "move forward", "turn left", "turn right", and is provided with a current state e.g., "right: blocked, left: open, forward: open, back: blocked" after each move. I wanted to see if the LLM could maintain some internal context so as not to explore paths that it has already seen before. In chat mode, it didn't. It would get stuck going back and forth along already-explored paths indefinitely. So I tried a "thinking" mode, where after each move, I asked it to summarize what it knew about the maze thus far. Then instead of continuing the chat I'd start a new one, providing the LLM with its previous instance's summary. With gpt4, the summaries were very verbose, and each subsequent iteration would append to it and maybe rephrase a bit. It got very slow, and didn't seem to help a whole lot. They were so verbose, it was hard to tell if they accurately represented the maze or not. "On step N, you went forward, and you found walls on two sides but not the front, so I would recommend trying going forward again for your subsequent move". With o3-mini, the summaries were more concise and more mathematical looking, easy to understand. "Current position: (4,3) orientation: E. History: (0,0): walls N,E; open S,W. (0,1): walls ...". But it would get them wrong sometimes, so didn't help in the long run. I think maybe I could improve the results by playing around with the prompts a bit, but I'm moderately convinced at this point that they're not good enough to maintain an internal state nor externalize a scratchpad state that would allow them to solve a maze one step at a time efficiently. Much less, make sense out of a full RPG.
mathiaspoint 1y ago
I don't think so. The attention should be able to manage as well as any external RAG tool, possibly better in some cases. It just needs training.

by btown 1y ago

daxfohl 1y ago

Or even just a notepad. It's well established that long context histories with scattered information are hard for LLMs to navigate. To distinguish whether it's using notes or context history, you could simply delete the context after each turn. The prompt could be something like "you're taking over this game from a previous player who has compiled these notes. (insert notes here). Play one turn, and update the notes with any new knowledge you have attained, relationships you have identified, inaccuracies you have confirmed, hypotheses you have, or anything else you this would be useful, so that the next player will be able use these notes to make the best next move.", and just clear the context after each move. Maybe also say there's a limit to the number of words on the notepad so that it doesn't just flood the notes with irrelevant information. For future iterations, maybe also give it a bitmap or svg canvas, or a database, or a code interpreter, and see if it uses any of those tools at all.

daxfohl 1y ago
I went ahead and did this, just using a maze where the only actions are "turn left". "turn right", and "go forward". The response is either OK or BLOCKED, and additionally which directions are open and which are walled in the current cell (relative directions: front, back, left, right. Not north east south west). When having it try to retain context of where it is via chat history alone, it failed. It went back into fully-explored paths and explored them again. When giving it a scratchpad to store knowledge, it didn't do any better. At least one common failure pattern I noticed is it got confused and updated its position when issuing a "turn" command. It never created a map, no matter how strongly I recommended doing so in the prompt, but generally just stored a list of cells it went to, and what it saw in each cell. I'd have played with it more and think I could have eventually gotten it working 99% better, but I'd already spent $12 and I'm unemployed so. Anyway, even if I got it working really really well, it'd still just be another example of "don't use an LLM when a simple state machine would be easier and work better". I think an LLM could be useful if the response from the maze game was natural language: "You went forward a step, to find yourself a T-intersection with passages off to each side.", and have the LLM translate that to a structured response that could be fed into the state machine. But it's still not reliable enough to serve as the state machine itself.

theptip 1y ago

This is a well-debated point; should you test the system on its own, or system+scaffold? How much custom scaffolding is allowed? Both avenues are interesting. For AGI presumably the “general” bit means you don’t get to build task-specific scaffolding ahead of time (though you can certainly build generic scaffolds like memory or knowledge layers). For safety/capability research, system+scaffolding is often more interesting because that is the frontier; if you conclude “LLMs cannot world-model” but actually LLM+CoT+memory can world model in a specific domain you care about, then you might underestimate capabilities and therefore deployment risk. The general point being: capabilities are jagged and prompt-dependent, just because you failed to elicit a capability, doesn’t mean you proved it cannot be elicited.

daxfohl 1y ago
FWIW I just tried making a maze game and having an LLM solve it step by step, where each move the LLM has to choose from "move forward", "turn left", "turn right", and is provided with a current state e.g., "right: blocked, left: open, forward: open, back: blocked" after each move. I wanted to see if the LLM could maintain some internal context so as not to explore paths that it has already seen before. In chat mode, it didn't. It would get stuck going back and forth along already-explored paths indefinitely. So I tried a "thinking" mode, where after each move, I asked it to summarize what it knew about the maze thus far. Then instead of continuing the chat I'd start a new one, providing the LLM with its previous instance's summary. With gpt4, the summaries were very verbose, and each subsequent iteration would append to it and maybe rephrase a bit. It got very slow, and didn't seem to help a whole lot. They were so verbose, it was hard to tell if they accurately represented the maze or not. "On step N, you went forward, and you found walls on two sides but not the front, so I would recommend trying going forward again for your subsequent move". With o3-mini, the summaries were more concise and more mathematical looking, easy to understand. "Current position: (4,3) orientation: E. History: (0,0): walls N,E; open S,W. (0,1): walls ...". But it would get them wrong sometimes, so didn't help in the long run. I think maybe I could improve the results by playing around with the prompts a bit, but I'm moderately convinced at this point that they're not good enough to maintain an internal state nor externalize a scratchpad state that would allow them to solve a maze one step at a time efficiently. Much less, make sense out of a full RPG.

mathiaspoint 1y ago

I don't think so. The attention should be able to manage as well as any external RAG tool, possibly better in some cases. It just needs training.