4 ms·

While I understand the hesitancy for LLM fiction, I do look forward to LLM powered games, IF DONE RIGHT. I imagine a game like Skyrim, where each character ha

by km3r 2mo ago

While I understand the hesitancy for LLM fiction, I do look forward to LLM powered games, *IF DONE RIGHT*. I imagine a game like Skyrim, where each character has handwritten backstory and dialogue, with an LLM extending that. Having the city guard dynamically react to what you do, stay in character, and even point you to new POI from those conversations seems like a very cool new dimension in storytelling. I took a creative writing class back in the day, and we talked about how fiction writers often have a massive "canon" backstory to many characters and worlds, and draw from that to create the story. That same backstory details can be use to flesh out a very immersive world. You still have guidance and handwritten essential lines, just use LLMs to enhance the world. Now, will there be lots of projects that use LLMs to take shortcuts and be far from this idealized narrative, yes. But not all of them. Some will stand out for getting it right.

dgellow 2mo ago
Sounds like a fun gimmick for maybe a month. And something that will be getting really tiring really fast. And imagine the PR horror when people start jail breaking the NPC LLMs
- km3r 2mo ago
  I dont think it should be a PR horror anymore than someone using photoshop to make a disturbing image. I get that is not quite the current reaction, but i dont see why it should be that way.
mxmlnkn 2mo ago
I have been imagining this since GPT-3. I still only know of very few games making use of LLMs. There are add-ons for Skyrim, and web search points me to NVIDIA's ACE autonomous game characters framework and games such as InZOI, PUBG's ACE-powered AI teammate, Dead Meat (unreleased interrogation game), AI Dungeon, Suck Up!, Vaudeville, and more. It seems there are a few more than I would have thought, but the smaller ones seem to have some mixed reviews. It would be interesting to check out how these games resolve the reasons against LLMs in games I can think of: Cost: LLMs fitting into local GPUs, in addition to the memory required for game textures, are too constrained for the quality, latency, and concurrency required. Connecting to the cloud would be too expensive, and even if it would work, it would indeed be the death of local offline gaming, and it would give a valid reason for a subscription to keep the servers running as opposed to a buy-once play-forever game business model. Unreliability: You don't want to miss a major quest because the LLM did not decide to hand it out. It might be workable with some proper "agentic harness", with some hardcoded world lore, long-term memories, goals, midterm memories, some deterministic actions, and a short-term free-form loop that gets cleared every now and then. For OpenClaw instances, it would be the SOUL.md, but it would have to be read-only. Some logic might have to be hardcoded outside of LLMs, e.g., to start a new narrative branch powered by a new character memory when a keyword has been mentioned or a quest item has been given. Similar to agentic frameworks, it would have tool calls such as `offer_quest`, `give_reward`, etc. These ideas seem to converge onto the Westworld mechanics of backstories, cornerstone memories, reveries, narratives, and loops, at least as far as I remember them from the series. Thinking about it, given enough investment, I feel like the amusement park of Westworld should already be somewhat possible to implement with the recent advances in LLMs and robotics. In the end, I am not sure how much sense it makes for games. Maybe it would suffice to only make companion characters LLM-enhanced while NPCs stay pre-programmed in order to let the game stay deterministic. Games should simplify real life in order to increase enjoyability. Making all characters backed by LLMs might decrease enjoyability because of forgetfulness or confabulations inconsistent with the game world and actions. It would need to be heavily constrained and trained or prompted to say "I don't know about that.". It might also be questionable how much needs to be dynamic. E.g., when thinking about the NPCs in The Witcher, which are talking about random stuff, maybe you only need to pregenerate much more random stuff for them to talk about so that you don't notice the repetition, but there might be no need to have this completely dynamic and burn through GPU power for every action. The question would be what to cache and what really needs to be dynamic.
- km3r 2mo ago
  A harness can definitely still make it "deterministic": if (responses > 3) prompt.md += "Divert the conversation to bring up QUEST4.md" I look at it a lot like procedural generation. It can be done well if used in the right spots and enough handwritten/tweaked stuff to make sure it feels right.

While I understand the hesitancy for LLM fiction, I do look forward to LLM powered games, *IF DONE RIGHT*. I imagine a game like Skyrim, where each character ha

by km3r 2mo ago

dgellow 2mo ago

Sounds like a fun gimmick for maybe a month. And something that will be getting really tiring really fast. And imagine the PR horror when people start jail breaking the NPC LLMs

km3r 2mo ago
I dont think it should be a PR horror anymore than someone using photoshop to make a disturbing image. I get that is not quite the current reaction, but i dont see why it should be that way.

mxmlnkn 2mo ago

I have been imagining this since GPT-3. I still only know of very few games making use of LLMs. There are add-ons for Skyrim, and web search points me to NVIDIA's ACE autonomous game characters framework and games such as InZOI, PUBG's ACE-powered AI teammate, Dead Meat (unreleased interrogation game), AI Dungeon, Suck Up!, Vaudeville, and more. It seems there are a few more than I would have thought, but the smaller ones seem to have some mixed reviews. It would be interesting to check out how these games resolve the reasons against LLMs in games I can think of: Cost: LLMs fitting into local GPUs, in addition to the memory required for game textures, are too constrained for the quality, latency, and concurrency required. Connecting to the cloud would be too expensive, and even if it would work, it would indeed be the death of local offline gaming, and it would give a valid reason for a subscription to keep the servers running as opposed to a buy-once play-forever game business model. Unreliability: You don't want to miss a major quest because the LLM did not decide to hand it out. It might be workable with some proper "agentic harness", with some hardcoded world lore, long-term memories, goals, midterm memories, some deterministic actions, and a short-term free-form loop that gets cleared every now and then. For OpenClaw instances, it would be the SOUL.md, but it would have to be read-only. Some logic might have to be hardcoded outside of LLMs, e.g., to start a new narrative branch powered by a new character memory when a keyword has been mentioned or a quest item has been given. Similar to agentic frameworks, it would have tool calls such as `offer_quest`, `give_reward`, etc. These ideas seem to converge onto the Westworld mechanics of backstories, cornerstone memories, reveries, narratives, and loops, at least as far as I remember them from the series. Thinking about it, given enough investment, I feel like the amusement park of Westworld should already be somewhat possible to implement with the recent advances in LLMs and robotics. In the end, I am not sure how much sense it makes for games. Maybe it would suffice to only make companion characters LLM-enhanced while NPCs stay pre-programmed in order to let the game stay deterministic. Games should simplify real life in order to increase enjoyability. Making all characters backed by LLMs might decrease enjoyability because of forgetfulness or confabulations inconsistent with the game world and actions. It would need to be heavily constrained and trained or prompted to say "I don't know about that.". It might also be questionable how much needs to be dynamic. E.g., when thinking about the NPCs in The Witcher, which are talking about random stuff, maybe you only need to pregenerate much more random stuff for them to talk about so that you don't notice the repetition, but there might be no need to have this completely dynamic and burn through GPU power for every action. The question would be what to cache and what really needs to be dynamic.

km3r 2mo ago
A harness can definitely still make it "deterministic": if (responses > 3) prompt.md += "Divert the conversation to bring up QUEST4.md" I look at it a lot like procedural generation. It can be done well if used in the right spots and enough handwritten/tweaked stuff to make sure it feels right.