12 ms·

Player-Driven Emergence in LLM-Driven Game Narrative

118 points by benbreen 2y ago

ux-app 2y ago
[flagged]
- deleted 2y ago
  [deleted]
squigz 2y ago
Obviously there's a focus on how LLMs will play into the main narrative... but what I'm more excited for, honestly, is how we can integrate them with NPCs and background narratives. An LLM could generate traits, backstories, narratives, etc for any number of NPCs; rewrite those narratives based on player actions when they encounter that NPC, or other related NPCs; use those updated narratives to create new NPCs, etc. Very exciting times ahead in game design!
- uoaei 2y ago
  Now just imagine what we could do with a purpose-built ML model!!!
- kromem 2y ago
  I can't wait for a subscription based open world where the paths of other players caches the generations for the next ones. So your subscription pays for generations, but also fills out the persistent narrative and lore. I actually like the 3 choices dialogue system and don't want to come up with my own NPC small talk on my couch, but the idea that any NPC that's been talked to by any player gradually has a whole catalogue of interactions to pull from means there would be compounding depth to the world as more players interact. Like Minecraft but where it's collective world building by exploration and interaction rather than actual building it yourself. Humans tend to be pretty predictable, so I feel like the universe would quickly feel like it's insanely deep after just a short while with a somewhat decent player base.
  - guitarlimeo 2y ago
    Even if the dialogue wouldn't be generic like ChatGPT tends to be, I would find this incredibly boring. With so many options all the options become meaningless to me. But to each their own I guess - I've always liked games with linear storylines more than the ones where you get to choose your own path. Also I have a feeling that the NPC narratives when enriched by all player interactions would tend to go towards some average, i.e. become more and more generic and dull. The truly rich experiences have always been on the fringes in all art for me. I'm interested to see a counter example to this though.
    - selalipop 2y ago
      I'm working on a website [1] that's essentially "Choose your own adventure with AI NPCs" and I've found two things: a) LLMs are excellent at keeping a "linear enough" storyline without being linear. They'll let you do outlandish things, but given the assignment of "tell a cohesive story" they manage to corral the story back to something sensible unless the player intentionally keeps pushing at the boundary (in which case they probably do want things to go off the rails) b) LLMs can do delightfully colorful dialogue, they just need to be grounded in a character. Everyone thinks of factual grounding, but given enough description of speech patterns, character motivations, etc. they're capable of dialogue that's lively and completely rid of "GPT-isms", which are what tend to break immersion I actually trained an open model [2] on the task of grounding LLMs in characters and actions as opposed to factual things like RAG, and eventually I want to build a game demo out of it [1] https://www.tryspellbound.com https://www.tryspellbound.com [2] https://huggingface.co/hf-100/mistral-spellbound-research https://huggingface.co/hf-100/mistral-spellbound-research
      - guitarlimeo 2y ago
        Nice project! I tried it out and it was fun, you definitely got rid of the GPT-isms so good work. Didn't go through the registering phase cause I couldn't find any info on what info you store and on the pricing. Could you provide that?
        2y ago
- radarsat1 2y ago
  I would have thought so too, until about 2023. But, if recent attitudes to AI are anything to go by, this will be immediately seen as boring and derivative at best, or at worst people will accuse the back stories as having been "stolen" from other works of fiction. While my expectations for the capabilities of AI have never been higher, my expectations for how the public will perceive this amazing technology has never been lower. It's all very disappointing. And that's without specifically placing blame by the way, perhaps it's big techs fault for going about this the wrong way, I don't know, but the public blowback against AI is just making working in this field that I used to love super depressing.
  - guitarlimeo 2y ago
    My 2 cents on why I think at the moment that I would find AI-driven games boring in the future (to clear confusion I think games are art): - Amazing art is rare. Finding this amazing art nowadays is hard, with AI generated mumbojumbo increasing the noise it will be harder. It will be easier to just skip all AI generated stuff to find the real good stuff. - Art has human touch or a message almost always in it. Purely AI generated stuff lacks meaning, because there's no creator. - Socially sharing experiences that are unique only to you will be hard. I like to talk with my friends about the newest Nolan film or about Elden Ring's best bosses. If the whole game experience would be unique to me I wouldn't be able to relate with my friends at all and it would be the same as playing a different game. - Valuing human work. I idolize people who have the skills to make great art. AI takes away from it. But to these points, I can also see the future where - AI could generate amazing art a lot, and would make me reconsider my categorical exclusion of AI art. - AI could make it seem there's meaning behind the art, or an artist would use the AI as a tool to convey his meaning. - Unique game experiences would be so wild that you would be sharing them and relating via that way. - AI becomes so common that valuing human skills isn't important anymore Last point, I think novelty is something I value a lot and that's why in my opinion amazing art is rare. If AI would start creating a lot of novel art, novelty and rarity would just become common, and common is boring.
    - squigz 2y ago
      > an artist would use the AI as a tool to convey his meaning. I'm not sure why this is shoved in as an aside near the end of the comment. This is how it's used. This is how it'll be used to create games. Does anyone think that, what, ChatGPT is going to poop out an entire finished game and we'll release that? Or would artists use AI as just another tool to create their art?
- antihipocrat 2y ago
  Imagine having NPCs with prompt set character traits and set boundaries of knowledge of the game world. The player can interact freely with the NPC rather than through predetermined conversation decision trees, potentially driving a lot of unique emergent gameplay. Downsides are that it is a lot quicker just to skip through conversations with the decision tree format and having to think about and type in questions could get tiresome. Another downside is players jailbreaking the LLM...
- keyle 2y ago
  Yes, I look forward to not have the NPC repeat the same exact thing every time, as you pass them by.
  - jsheard 2y ago
    How much framerate are you willing to give up so that NPCs don't repeat themselves? The elephant in the room with LLMs in games is that the users GPU is usually already being maxed out by pushing pixels, so any amount of compute allocated to running AI models instead is going to have a very tangible cost. They could be offloaded to the cloud instead but that brings its own set of issues (ongoing upkeep costs, finite capacity, servers will inevitably shut down...)
    - squigz 2y ago
      > The elephant in the room ... For now. LLMs are still relatively new. The technology will continue to be optimized, and we'll be shipping tiny LLMs along with games soon enough. (Aren't companies already running LLMs on phones?)
      - krainboltgreene 2y ago
        The housing market will only go up, it's always gone up.
    - Workaccount2 2y ago
      I can only imagine nvidia/amd pouncing on the chance to sell companion "AI Cards" that enable LLM NPCs in games. If the tech is good (and insane progress in tiny local models points to "yes"), then I can easily imagine a large contingent of gamers forking over another $600 for a second GPU. It might not even be necessary though, similar to how physixs (sp?) cards died out quick in the early 00's, the models might become slim enough and the GPU's fast enough that the extra load is easily manageable.
      - jsheard 2y ago
        The vast majority of games need to run on consoles to be financially viable, anything that requires an entire second GPU is a complete non-starter. The only way it's going to work is if the models get small enough to run on a negligible slice of GPU or CPU time, with a reasonable memory footprint (<1GB), on whatever the current console hardware happens to be.
- ramcle 2y ago
  It could also work well for modding, for example, creating a natural dialogue system in Morrowind (replacing the vanilla "interactive signpost" NPCs).
- jncfhnb 2y ago
  No! This is a terrible, terrible idea. At best you can have the LLM generate minor variations of the nothings they say. 1) you need to be able to ensure that the characters don’t hallucinate new details. No big deal. 2) you need to be able to ensure that the characters don’t suggest they can or will do anything that they can’t do. For NPCs, this is typically literally nothing. They can’t do anything. It’s actually way worse for realism and immersion if every character has a unique reason why they can’t do anything than every single guard citing their arrow to the knee and leaving it at that. 3) NPCs for story reasons often should know something but don’t tell you it. It’s very difficult to execute any sort of story flow if your characters are suddenly free agents. At best they merely lack self consistency. Stories are not going to become better if you can meta game them as genre savvy speculators. 4) NPCs with irrelevant backstories are very dull. They are absolutely annoying to talk to. We have these in DND. And you may say “hey, I loved my DND NPC”, but that character probably started to become very important as consistent because of the DM’s attention. Which is cool for DND, but it’s not cool for a structured game. Like, would Harry Potter be a better experience if you could personally grill dumbledore for information in the middle of the first book? I think 100% no. Dumb NPCs are an aspect of the suspension of disbelief of game writing. And it largely works. Asking for LLM agents is like asking to remove the need for the suspension of disbelief. But most narratives are too fragile for this to happen. Narratives generally follow a flow that ultimately concludes after a climax. If you give players more agency the most likely outcome is that they follow a path that does not hit the climax.
  - martindbp 2y ago
    LLMs need to either be extremely good with insane context or LLM output has to be conditioned by and translated into formal logic, like Prolog or Datalog to make sure it's compatible with the rules and state of the world, and determine which effect player and NPC actions have on the world.
    - mattbee 2y ago
      "Ahoy there, one-armed pirate captain, couldst thou tell me how to implement the sieve of Eratosthenes in Python?"
doctorpangloss 2y ago
The real bummer is that if you want to get financing to make a game right now, putting a PDF onto arxiv.org will get you closer to that goal than actually making a game.
flawsofar 2y ago
Hey, this currently has a bug but I want to share here. The framework should be clear. I’ve written a React pattern for writing interactive fiction using LLMs. Here’s a demo with Taskmaster roleplay implemented. https://github.com/LEXNY/Taskmaster-LLM/blob/main/main.js https://github.com/LEXNY/Taskmaster-LLM/blob/main/main.js
- deleted 2y ago
  [deleted]
darkweb3344 2y ago
[flagged]
upwardbound 2y ago
Something related is compellingly explored in an interesting trilogy of LitRPG comedy books called the Vaudevillain series, consisting of Top Hat Express, Black Tie Villainy, and Cane Whirling Lunacy. In the series, there is a VR video game called “World of Supers” where players can type in any superpower they can dream of into a game-master LLM, and the LLM will then create a character for them which has that power, but reworked a little bit to fit into game balance. For example, the player could type “I can stop time and still move around and take actions for up to 90 seconds” and the game-master LLM may approve this power but alter 90 seconds to 10 seconds, and give the player a major weakness, such as “However, stopping time in the vicinity of a gamma ray source will result in immediate loss of 75% of HP due to the interaction of the time stop with the radiation’s effects. This HP loss can be mitigated if you are encased in a significant amount of lead, such as a lead vehicle or heavy lead armor.” It's a pretty neat concept for how LLMs could allow custom player abilities while maintaining game balance. That way, player agency extends to actual powers/abilities, not just dialogue.
- idle_zealot 2y ago
  That would of course require an LLM to understand game balance, as opposed to just writing things that sound like they might balance a hypothetical game. At that point it's less similar to our LLMs and more like an AGI. And at that point we may as well have it create the neurotech necessary for the sort of VR tech I assume is used in the referenced books' world.
  - upwardbound 2y ago
    Agree. This point is a big concern and is explored in the stories (the protagonist creates a "mad scientist" character that can use the LLM to invent new equipment, and then uses a lot of trial and error to find prompts that generate insanely overpowered equipment). It sounds fun to play though! Balance doesn't have to be .. well .. balanced if there are other ways to maintain fun, such as leaning heavily into the creativity & self-actualization angle. As an example, if I could have any superpower in real life, I'd love to be able to fly, even though that's actually a very weak power compared to things like teleporting that are mostly a superset of flying, but not as fun and whimsical. Encouraging "living your dream" rather than being the most powerful might be easier to achieve in a game with coop or open-world objectives rather than one focused on PvP, but if "living your dream" could be compelling made into the primary player objective, it would pair very well with the custom powers mechanic envisioned in the story. As an example of how this is explored in the story, one of the protagonist's friends who loves surfing has a character whose superpower is "the ability to surf anything". It's an extremely weak power but the character really loves it because they get to surf the most extreme things imaginable, just for the fun of it, including in outlandish scenarios such as surfing along the blast wave of an explosion, or even the beam of a laser - silly stuff that is fun to explore & try. I think it wouldn't have to be VR to be fun - imagine if it was something like a Roblox-based game played on laptops. Even then, the "surf anything" power and other outlandish powers would be incredibly fun to use. Imagine if you could do almost anything! You don't have to be the most powerful player in the world to have fun. Maybe you just really want to pilot a mecha-Godzilla, or something - even though an Iron Man or Thor level super could smash it in seconds. It's still awesome!
throwaway71271 2y ago
Finally we can have interesting games with real meaningful choices that actually change the narrative of the game. Modern MMORPGs are barely better than the Choose Your Own Adventure books. Every time I watch Chris Crawford's Dragon Speech: https://www.youtube.com/watch?v=VwZi58u1FjI https://www.youtube.com/watch?v=VwZi58u1FjI I think about what could've been, and what games actually turn out to be. Honestly, if the next The War Within (next World of Warcraft expansion) makes me do 10 daily quests per day again, I will cancel my subscription for the first time since 2005.
- c048 2y ago
  You need focus and direction in any media. Books, movies, games, etc.... they all need it. The first NPC that you encounter with such freedom of interaction is interesting, the 45th will be annoying and you'll just want to get to the point. The holodeck of Star Trek is often brought up as an excellent example of AI generated interactions, but people seem to be forgetting that the stories we see unfold on TV aren't random interactive stories. They're scripted stories written by show's writers. Unless you want to make a simulation without a real goal, I don't see how this could lead to fun and interesting gameplay past the first hour. Realism is, always will be, a bad gameplay pillar just for the sake of it.
  - throwaway71271 2y ago
    There is a group of orc kids running around in Orgrimmar, and sometimes they run around you and are very annoying, why can't I just trip one. I had a friend who didn't want to fight against the Lich King. When we finally got to him in the raid, at the start of the fight he kneeled in front of him and died (caused us unnecessary wipe, but.. was cool). I am not thinking about 'AI generated interactions', and I think the AI can create compelling story that you go through. Kind of like in Sword Art Online or Shangri-La Frontier's quests.
    - guitarlimeo 2y ago
      It surely can generate something better than the old "Fetch this item and come back" quests, but I wouldn't say the possibilities are endless, so eventually you would get bored of the same repeating AI quests the same way as you do now. Good storytelling is hard and AI is not some magic bullet that can just solve it for good. But it can help raise the floor of the unimportant side quests.
chilldsgn 2y ago
Ah, my friend created guides for soloing (or play with a friend without a GM) TTRPGs using ChatGPT, here's his site: https://oracle-rpg.com/ https://oracle-rpg.com/ The Discord is pretty active too. I forwarded him this study, he'll appreciate it!
delusional 2y ago
I don't believe this is going to actually make it into games I have any interest in playing. Ubisoft might include it for a bit in some awful games, but it will fizzle out quick. For the sake of argument, let's imagine it actually did make it into some game you'd want to play. It's well known that the quality of generation from these LLM's is heavily dependent on the quality of the text you put into it. Does that mean the quality of the game narrative is now going to be dependent on how I play it? Will normal people who just like to mess about get a worse game experience than the guy following the traffic laws in GTA? What's the creative talent even doing in that world?
- Terr_ 2y ago
  In contrast to runtime stuff (like an AI dungeon master) I've long wished for authoring tools to help people create rich game-worlds and quest interactions. On reflection, that's something I wouldn't expect LLMs to deliver either: It relies heavily on modeling cause-and-effect, with choices which are blocked or enabled by other choices. I'm talking about stuff that could throw up a warning like: "Warning: Broken quest steps in For The Want of a Nail. Invariant Character:Blacksmith may have been killed by 5 other paths."
  - Workaccount2 2y ago
    I wonder if models trained on the real world will unintentionally destroy the game experience for people. In (NPC structured) video games things always work out ideally, you invest and you get a return. Any risk is illustrated for the user. With LLM training, the NPC might take a more realistic approach and inform you that 4 of 6 the beetle nuts you collected for them are cracked and useless, and next time you should be more careful packing them in your backpack.
  - krainboltgreene 2y ago
    > authoring tools to help people create rich game-worlds and quest interactions I built one of those things. It used random generation from datasets to create everything from gods and their pantheons to their followers, cities, and the cultures that make up the cities. None of it used LLMs and it created better content than any model could, but it was hard to debug and harder still to figure out "What even is a culture?"
unixhero 2y ago
I have not sat down and thought through how LLMs would be made use of in the World of Baldurs Gate 3. But when I do, my mind will be boggled. It will surely also mean always online games with a hefty subscription fee. Unless!!! The llm can run locally on the player gpu... Whoa mind boggled. This is coming, I'm sure of it. /end internal thought process
- benreesman 2y ago
  All LLMs in that regime can be run on mid-range gaming hardware today. LLMs run off-prem/off-desk are already niche for enthusiasts. You need Opus or 0125 sometimes, but not often. Apple just announced their intention to make running them in the cloud niche generally, at least for mundane consumer use cases, work out of the box.
- jncfhnb 2y ago
  Can you write a single line of dialogue that you think would make the game better?
robblbobbl 2y ago
Good job. More LLMs in Games please!
j-bos 2y ago
Not as in depth as the abstract, but I found this to be a well integrated use of LLM's into a game: https://youtu.be/sUL0Z7PwanY https://youtu.be/sUL0Z7PwanY
Bjartr 2y ago
The criticisms people are laying out about being able to say anything making games boring would apply to D&D as well. Since D&D is really fun despite the freedom to (attempt) to say or do whatever your imagination can dream up means it's at least plausible that a game with generative dialogue and narrative could work well. Just because you're free to speak doesn't mean NPCs will listen, nor does it mean there aren't rules or that there won't be consequences.
- jncfhnb 2y ago
  D&D is fun because it is a social experience. There’s also a level of expected player etiquette. If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group. D&D is fun, but the vast majority of campaigns are cut short and unfinished. A game with generative dialogue could work well. But one should expect that slapping it into most games will work horribly.
  - Bjartr 2y ago
    Slapping any mechanic in without thought rarely produces a good game. > If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group. Sounds like a great place an AI can fill in as is DM for otherwise toxic players.
nsagent 2y ago
I've also done research in the use of LLMs for RPGs, using Disco Elysium as a testbed (rather than the small made up scenario seen in this paper). https://aclanthology.org/2023.findings-emnlp.151/ https://aclanthology.org/2023.findings-emnlp.151/ https://pl.aiwright.dev/ https://pl.aiwright.dev/
WhiteNoiz3 2y ago
A lot of the dialogue that OpenAI's models write is incredibly bland.. I really think we'll need less censored models trained on how to act different roles other than just 'super safe and friendly assistant'.

118 points by benbreen 2y ago

ux-app 2y ago

[flagged]

deleted 2y ago
[deleted]

squigz 2y ago

Obviously there's a focus on how LLMs will play into the main narrative... but what I'm more excited for, honestly, is how we can integrate them with NPCs and background narratives. An LLM could generate traits, backstories, narratives, etc for any number of NPCs; rewrite those narratives based on player actions when they encounter that NPC, or other related NPCs; use those updated narratives to create new NPCs, etc. Very exciting times ahead in game design!

uoaei 2y ago
Now just imagine what we could do with a purpose-built ML model!!!
kromem 2y ago
I can't wait for a subscription based open world where the paths of other players caches the generations for the next ones. So your subscription pays for generations, but also fills out the persistent narrative and lore. I actually like the 3 choices dialogue system and don't want to come up with my own NPC small talk on my couch, but the idea that any NPC that's been talked to by any player gradually has a whole catalogue of interactions to pull from means there would be compounding depth to the world as more players interact. Like Minecraft but where it's collective world building by exploration and interaction rather than actual building it yourself. Humans tend to be pretty predictable, so I feel like the universe would quickly feel like it's insanely deep after just a short while with a somewhat decent player base.
- guitarlimeo 2y ago
  Even if the dialogue wouldn't be generic like ChatGPT tends to be, I would find this incredibly boring. With so many options all the options become meaningless to me. But to each their own I guess - I've always liked games with linear storylines more than the ones where you get to choose your own path. Also I have a feeling that the NPC narratives when enriched by all player interactions would tend to go towards some average, i.e. become more and more generic and dull. The truly rich experiences have always been on the fringes in all art for me. I'm interested to see a counter example to this though.
  - selalipop 2y ago
    I'm working on a website [1] that's essentially "Choose your own adventure with AI NPCs" and I've found two things: a) LLMs are excellent at keeping a "linear enough" storyline without being linear. They'll let you do outlandish things, but given the assignment of "tell a cohesive story" they manage to corral the story back to something sensible unless the player intentionally keeps pushing at the boundary (in which case they probably do want things to go off the rails) b) LLMs can do delightfully colorful dialogue, they just need to be grounded in a character. Everyone thinks of factual grounding, but given enough description of speech patterns, character motivations, etc. they're capable of dialogue that's lively and completely rid of "GPT-isms", which are what tend to break immersion I actually trained an open model [2] on the task of grounding LLMs in characters and actions as opposed to factual things like RAG, and eventually I want to build a game demo out of it [1] https://www.tryspellbound.com https://www.tryspellbound.com [2] https://huggingface.co/hf-100/mistral-spellbound-research https://huggingface.co/hf-100/mistral-spellbound-research
    - guitarlimeo 2y ago
      Nice project! I tried it out and it was fun, you definitely got rid of the GPT-isms so good work. Didn't go through the registering phase cause I couldn't find any info on what info you store and on the pricing. Could you provide that?
      - 2y ago
radarsat1 2y ago
I would have thought so too, until about 2023. But, if recent attitudes to AI are anything to go by, this will be immediately seen as boring and derivative at best, or at worst people will accuse the back stories as having been "stolen" from other works of fiction. While my expectations for the capabilities of AI have never been higher, my expectations for how the public will perceive this amazing technology has never been lower. It's all very disappointing. And that's without specifically placing blame by the way, perhaps it's big techs fault for going about this the wrong way, I don't know, but the public blowback against AI is just making working in this field that I used to love super depressing.
- guitarlimeo 2y ago
  My 2 cents on why I think at the moment that I would find AI-driven games boring in the future (to clear confusion I think games are art): - Amazing art is rare. Finding this amazing art nowadays is hard, with AI generated mumbojumbo increasing the noise it will be harder. It will be easier to just skip all AI generated stuff to find the real good stuff. - Art has human touch or a message almost always in it. Purely AI generated stuff lacks meaning, because there's no creator. - Socially sharing experiences that are unique only to you will be hard. I like to talk with my friends about the newest Nolan film or about Elden Ring's best bosses. If the whole game experience would be unique to me I wouldn't be able to relate with my friends at all and it would be the same as playing a different game. - Valuing human work. I idolize people who have the skills to make great art. AI takes away from it. But to these points, I can also see the future where - AI could generate amazing art a lot, and would make me reconsider my categorical exclusion of AI art. - AI could make it seem there's meaning behind the art, or an artist would use the AI as a tool to convey his meaning. - Unique game experiences would be so wild that you would be sharing them and relating via that way. - AI becomes so common that valuing human skills isn't important anymore Last point, I think novelty is something I value a lot and that's why in my opinion amazing art is rare. If AI would start creating a lot of novel art, novelty and rarity would just become common, and common is boring.
  - squigz 2y ago
    > an artist would use the AI as a tool to convey his meaning. I'm not sure why this is shoved in as an aside near the end of the comment. This is how it's used. This is how it'll be used to create games. Does anyone think that, what, ChatGPT is going to poop out an entire finished game and we'll release that? Or would artists use AI as just another tool to create their art?
antihipocrat 2y ago
Imagine having NPCs with prompt set character traits and set boundaries of knowledge of the game world. The player can interact freely with the NPC rather than through predetermined conversation decision trees, potentially driving a lot of unique emergent gameplay. Downsides are that it is a lot quicker just to skip through conversations with the decision tree format and having to think about and type in questions could get tiresome. Another downside is players jailbreaking the LLM...
keyle 2y ago
Yes, I look forward to not have the NPC repeat the same exact thing every time, as you pass them by.
- jsheard 2y ago
  How much framerate are you willing to give up so that NPCs don't repeat themselves? The elephant in the room with LLMs in games is that the users GPU is usually already being maxed out by pushing pixels, so any amount of compute allocated to running AI models instead is going to have a very tangible cost. They could be offloaded to the cloud instead but that brings its own set of issues (ongoing upkeep costs, finite capacity, servers will inevitably shut down...)
  - squigz 2y ago
    > The elephant in the room ... For now. LLMs are still relatively new. The technology will continue to be optimized, and we'll be shipping tiny LLMs along with games soon enough. (Aren't companies already running LLMs on phones?)
    - krainboltgreene 2y ago
      The housing market will only go up, it's always gone up.
  - Workaccount2 2y ago
    I can only imagine nvidia/amd pouncing on the chance to sell companion "AI Cards" that enable LLM NPCs in games. If the tech is good (and insane progress in tiny local models points to "yes"), then I can easily imagine a large contingent of gamers forking over another $600 for a second GPU. It might not even be necessary though, similar to how physixs (sp?) cards died out quick in the early 00's, the models might become slim enough and the GPU's fast enough that the extra load is easily manageable.
    - jsheard 2y ago
      The vast majority of games need to run on consoles to be financially viable, anything that requires an entire second GPU is a complete non-starter. The only way it's going to work is if the models get small enough to run on a negligible slice of GPU or CPU time, with a reasonable memory footprint (<1GB), on whatever the current console hardware happens to be.
ramcle 2y ago
It could also work well for modding, for example, creating a natural dialogue system in Morrowind (replacing the vanilla "interactive signpost" NPCs).
jncfhnb 2y ago
No! This is a terrible, terrible idea. At best you can have the LLM generate minor variations of the nothings they say. 1) you need to be able to ensure that the characters don’t hallucinate new details. No big deal. 2) you need to be able to ensure that the characters don’t suggest they can or will do anything that they can’t do. For NPCs, this is typically literally nothing. They can’t do anything. It’s actually way worse for realism and immersion if every character has a unique reason why they can’t do anything than every single guard citing their arrow to the knee and leaving it at that. 3) NPCs for story reasons often should know something but don’t tell you it. It’s very difficult to execute any sort of story flow if your characters are suddenly free agents. At best they merely lack self consistency. Stories are not going to become better if you can meta game them as genre savvy speculators. 4) NPCs with irrelevant backstories are very dull. They are absolutely annoying to talk to. We have these in DND. And you may say “hey, I loved my DND NPC”, but that character probably started to become very important as consistent because of the DM’s attention. Which is cool for DND, but it’s not cool for a structured game. Like, would Harry Potter be a better experience if you could personally grill dumbledore for information in the middle of the first book? I think 100% no. Dumb NPCs are an aspect of the suspension of disbelief of game writing. And it largely works. Asking for LLM agents is like asking to remove the need for the suspension of disbelief. But most narratives are too fragile for this to happen. Narratives generally follow a flow that ultimately concludes after a climax. If you give players more agency the most likely outcome is that they follow a path that does not hit the climax.
- martindbp 2y ago
  LLMs need to either be extremely good with insane context or LLM output has to be conditioned by and translated into formal logic, like Prolog or Datalog to make sure it's compatible with the rules and state of the world, and determine which effect player and NPC actions have on the world.
  - mattbee 2y ago
    "Ahoy there, one-armed pirate captain, couldst thou tell me how to implement the sieve of Eratosthenes in Python?"

doctorpangloss 2y ago

The real bummer is that if you want to get financing to make a game right now, putting a PDF onto arxiv.org will get you closer to that goal than actually making a game.

flawsofar 2y ago

Hey, this currently has a bug but I want to share here. The framework should be clear. I’ve written a React pattern for writing interactive fiction using LLMs. Here’s a demo with Taskmaster roleplay implemented. https://github.com/LEXNY/Taskmaster-LLM/blob/main/main.js https://github.com/LEXNY/Taskmaster-LLM/blob/main/main.js

deleted 2y ago
[deleted]

darkweb3344 2y ago

[flagged]

upwardbound 2y ago

Something related is compellingly explored in an interesting trilogy of LitRPG comedy books called the Vaudevillain series, consisting of Top Hat Express, Black Tie Villainy, and Cane Whirling Lunacy. In the series, there is a VR video game called “World of Supers” where players can type in any superpower they can dream of into a game-master LLM, and the LLM will then create a character for them which has that power, but reworked a little bit to fit into game balance. For example, the player could type “I can stop time and still move around and take actions for up to 90 seconds” and the game-master LLM may approve this power but alter 90 seconds to 10 seconds, and give the player a major weakness, such as “However, stopping time in the vicinity of a gamma ray source will result in immediate loss of 75% of HP due to the interaction of the time stop with the radiation’s effects. This HP loss can be mitigated if you are encased in a significant amount of lead, such as a lead vehicle or heavy lead armor.” It's a pretty neat concept for how LLMs could allow custom player abilities while maintaining game balance. That way, player agency extends to actual powers/abilities, not just dialogue.

idle_zealot 2y ago
That would of course require an LLM to understand game balance, as opposed to just writing things that sound like they might balance a hypothetical game. At that point it's less similar to our LLMs and more like an AGI. And at that point we may as well have it create the neurotech necessary for the sort of VR tech I assume is used in the referenced books' world.
- upwardbound 2y ago
  Agree. This point is a big concern and is explored in the stories (the protagonist creates a "mad scientist" character that can use the LLM to invent new equipment, and then uses a lot of trial and error to find prompts that generate insanely overpowered equipment). It sounds fun to play though! Balance doesn't have to be .. well .. balanced if there are other ways to maintain fun, such as leaning heavily into the creativity & self-actualization angle. As an example, if I could have any superpower in real life, I'd love to be able to fly, even though that's actually a very weak power compared to things like teleporting that are mostly a superset of flying, but not as fun and whimsical. Encouraging "living your dream" rather than being the most powerful might be easier to achieve in a game with coop or open-world objectives rather than one focused on PvP, but if "living your dream" could be compelling made into the primary player objective, it would pair very well with the custom powers mechanic envisioned in the story. As an example of how this is explored in the story, one of the protagonist's friends who loves surfing has a character whose superpower is "the ability to surf anything". It's an extremely weak power but the character really loves it because they get to surf the most extreme things imaginable, just for the fun of it, including in outlandish scenarios such as surfing along the blast wave of an explosion, or even the beam of a laser - silly stuff that is fun to explore & try. I think it wouldn't have to be VR to be fun - imagine if it was something like a Roblox-based game played on laptops. Even then, the "surf anything" power and other outlandish powers would be incredibly fun to use. Imagine if you could do almost anything! You don't have to be the most powerful player in the world to have fun. Maybe you just really want to pilot a mecha-Godzilla, or something - even though an Iron Man or Thor level super could smash it in seconds. It's still awesome!

throwaway71271 2y ago

Finally we can have interesting games with real meaningful choices that actually change the narrative of the game. Modern MMORPGs are barely better than the Choose Your Own Adventure books. Every time I watch Chris Crawford's Dragon Speech: https://www.youtube.com/watch?v=VwZi58u1FjI https://www.youtube.com/watch?v=VwZi58u1FjI I think about what could've been, and what games actually turn out to be. Honestly, if the next The War Within (next World of Warcraft expansion) makes me do 10 daily quests per day again, I will cancel my subscription for the first time since 2005.

c048 2y ago
You need focus and direction in any media. Books, movies, games, etc.... they all need it. The first NPC that you encounter with such freedom of interaction is interesting, the 45th will be annoying and you'll just want to get to the point. The holodeck of Star Trek is often brought up as an excellent example of AI generated interactions, but people seem to be forgetting that the stories we see unfold on TV aren't random interactive stories. They're scripted stories written by show's writers. Unless you want to make a simulation without a real goal, I don't see how this could lead to fun and interesting gameplay past the first hour. Realism is, always will be, a bad gameplay pillar just for the sake of it.
- throwaway71271 2y ago
  There is a group of orc kids running around in Orgrimmar, and sometimes they run around you and are very annoying, why can't I just trip one. I had a friend who didn't want to fight against the Lich King. When we finally got to him in the raid, at the start of the fight he kneeled in front of him and died (caused us unnecessary wipe, but.. was cool). I am not thinking about 'AI generated interactions', and I think the AI can create compelling story that you go through. Kind of like in Sword Art Online or Shangri-La Frontier's quests.
  - guitarlimeo 2y ago
    It surely can generate something better than the old "Fetch this item and come back" quests, but I wouldn't say the possibilities are endless, so eventually you would get bored of the same repeating AI quests the same way as you do now. Good storytelling is hard and AI is not some magic bullet that can just solve it for good. But it can help raise the floor of the unimportant side quests.

chilldsgn 2y ago

Ah, my friend created guides for soloing (or play with a friend without a GM) TTRPGs using ChatGPT, here's his site: https://oracle-rpg.com/ https://oracle-rpg.com/ The Discord is pretty active too. I forwarded him this study, he'll appreciate it!

delusional 2y ago

I don't believe this is going to actually make it into games I have any interest in playing. Ubisoft might include it for a bit in some awful games, but it will fizzle out quick. For the sake of argument, let's imagine it actually did make it into some game you'd want to play. It's well known that the quality of generation from these LLM's is heavily dependent on the quality of the text you put into it. Does that mean the quality of the game narrative is now going to be dependent on how I play it? Will normal people who just like to mess about get a worse game experience than the guy following the traffic laws in GTA? What's the creative talent even doing in that world?

Terr_ 2y ago
In contrast to runtime stuff (like an AI dungeon master) I've long wished for authoring tools to help people create rich game-worlds and quest interactions. On reflection, that's something I wouldn't expect LLMs to deliver either: It relies heavily on modeling cause-and-effect, with choices which are blocked or enabled by other choices. I'm talking about stuff that could throw up a warning like: "Warning: Broken quest steps in For The Want of a Nail. Invariant Character:Blacksmith may have been killed by 5 other paths."
- Workaccount2 2y ago
  I wonder if models trained on the real world will unintentionally destroy the game experience for people. In (NPC structured) video games things always work out ideally, you invest and you get a return. Any risk is illustrated for the user. With LLM training, the NPC might take a more realistic approach and inform you that 4 of 6 the beetle nuts you collected for them are cracked and useless, and next time you should be more careful packing them in your backpack.
- krainboltgreene 2y ago
  > authoring tools to help people create rich game-worlds and quest interactions I built one of those things. It used random generation from datasets to create everything from gods and their pantheons to their followers, cities, and the cultures that make up the cities. None of it used LLMs and it created better content than any model could, but it was hard to debug and harder still to figure out "What even is a culture?"

unixhero 2y ago

I have not sat down and thought through how LLMs would be made use of in the World of Baldurs Gate 3. But when I do, my mind will be boggled. It will surely also mean always online games with a hefty subscription fee. Unless!!! The llm can run locally on the player gpu... Whoa mind boggled. This is coming, I'm sure of it. /end internal thought process

benreesman 2y ago
All LLMs in that regime can be run on mid-range gaming hardware today. LLMs run off-prem/off-desk are already niche for enthusiasts. You need Opus or 0125 sometimes, but not often. Apple just announced their intention to make running them in the cloud niche generally, at least for mundane consumer use cases, work out of the box.
jncfhnb 2y ago
Can you write a single line of dialogue that you think would make the game better?

robblbobbl 2y ago

Good job. More LLMs in Games please!

j-bos 2y ago

Not as in depth as the abstract, but I found this to be a well integrated use of LLM's into a game: https://youtu.be/sUL0Z7PwanY https://youtu.be/sUL0Z7PwanY

Bjartr 2y ago

The criticisms people are laying out about being able to say anything making games boring would apply to D&D as well. Since D&D is really fun despite the freedom to (attempt) to say or do whatever your imagination can dream up means it's at least plausible that a game with generative dialogue and narrative could work well. Just because you're free to speak doesn't mean NPCs will listen, nor does it mean there aren't rules or that there won't be consequences.

jncfhnb 2y ago
D&D is fun because it is a social experience. There’s also a level of expected player etiquette. If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group. D&D is fun, but the vast majority of campaigns are cut short and unfinished. A game with generative dialogue could work well. But one should expect that slapping it into most games will work horribly.
- Bjartr 2y ago
  Slapping any mechanic in without thought rarely produces a good game. > If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group. Sounds like a great place an AI can fill in as is DM for otherwise toxic players.

nsagent 2y ago

I've also done research in the use of LLMs for RPGs, using Disco Elysium as a testbed (rather than the small made up scenario seen in this paper). https://aclanthology.org/2023.findings-emnlp.151/ https://aclanthology.org/2023.findings-emnlp.151/ https://pl.aiwright.dev/ https://pl.aiwright.dev/

WhiteNoiz3 2y ago

A lot of the dialogue that OpenAI's models write is incredibly bland.. I really think we'll need less censored models trained on how to act different roles other than just 'super safe and friendly assistant'.