3 ms·
That's an interesting benchmark. It feels like it tests skills that are very relevant to digital assistants, story writing and role play. Some thoughts about t
by wongarsu 2y ago
That's an interesting benchmark. It feels like it tests skills that are very relevant to digital assistants, story writing and role play.
Some thoughts about the setup:
- the setup seems to give reasoning models an inherent advantage because only they have a private plan and a public text in the same output. I feel like giving all models the option to formulate plans and keep track of other players inside <think> or <secret> tags would level the playing field more.
- from personal experience with social tasks for LLMs it helps both reasoning and non-reasoning LLMs to explicitly ask them to plan their next steps, in a way they are assured is kept hidden from all other players. That might be a good addition here either before or after the public subround
- the individual rounds are pretty short. Humans would struggle to coordinate in so few exchanges with so few words. If this was done for context limitations, asking models to summarize the game state from their perspective, then giving them only the current round, the previous round and their own summary of the game before that might be a good strategy.
It would be cool to have some code to play around with to test how changes in the setup change the results. I guess it isn't that difficult to write, but it's peculiar to have the benchmark but no code to run it yourself
- transformi 2y agoInteresting idea of <secret>...maybe extend it to several <secret_i>....to form a groups of secretes with different persons. In Addition it will be interesting to extend a variation of the game that the players can use tools and execute code to take their preparation one step further.
- wongarsu 2y agoMost models do pretty well with keeping state in XML if you ask them to. You could extend it to <secret><content>[...]</content><secret_from>P1</secret_from><shared_with>P2, P3</shared_with></secret>. Or tell the model that it can use <secret> tags with xml content and just let it develop a schema on the fly. At that point, I would love to also see sub-benchmarks how each models's score is affected by being given a schema vs having it make one up, and if the model does better with state in text vs xml vs json. Those don't tell you which model is best, but they are very useful to know for actually using them.
- eightysixfour 2y agoFor models that can call tools, just giving them a think tool where they can write their thoughts, can improve performance. Even reasoning models, surprisingly enough. https://www.anthropic.com/engineering/claude-think-tool https://www.anthropic.com/engineering/claude-think-tool
- jermaustin1 2y agoI did something similar for a "game engine" let the NPCs remember things from other NPCs' and the PC's interaction with them. It wasn't perfect, but the player could negotiate a cheaper price on a dagger for instance if they promised to owe the NPC a larger payout next time they returned to the shop. And it worked... most of the time, the shop owner remembered the debt and inquired about it on the next interaction - but not always, which I guess is kind of "human".
- rahimnathwani 2y agoThis is similar to this, right? https://github.com/modelcontextprotocol/servers/tree/main/src/sequentialthinking https://github.com/modelcontextprotocol/servers/tree/main/sr...
- eightysixfour 2y agoNo, the approach in the Anthropic blog has no structure. It is literally just a place for the model to “write its thoughts.” If you took the server above and stripped everything except the “thought” tool. With reasoning models you don’t carry forward the reasoning tokens in the context for the next call, so my hypothesis is that calling a tool/responding lets the model ditch a lot of built up context and “refocus.”
- rahimnathwani 2y agoAlthough the sequential thinking MCP asks for structure, I don't think it persists anything.
- 2y ago