5 ms·

This is awesome. LLM-powered NPCs is one thing I'm most excited about in the future of gaming. Characters repeating the same scripted dialog over and over again

This is awesome. LLM-powered NPCs is one thing I'm most excited about in the future of gaming. Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.

malfist 1y ago
I used to care about repeat scripted dialogue like you, until I took an arrow to the knee
- Arisaka1 1y ago
  Yep. I can see how relying on non-repetitive dialogue generated by LLM's will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. A way to bypass this problem is to keep the important dialogue fixed, but this only one of the problems. Another issue would be emphasizing the meaninglessness of the dialogue. For example, playing Trails in the Sky has lots of NPC dialogue that's repetitive, but at least the dialogue is relevant with how the NPC's life progresses in the grander scheme of things, such as having difficulty with her entrance exams, or having an argument with his fiancé. It's not main dialogue but adds flavor for anyone who cares about the world enough to interact with the citizens. I don't think I'd like to interact with characters that I know whatever it is they have to say is generated on the fly and adds nothing other than random tidbits. The novelty would quickly wear off.
  - cubefox 1y ago
    A better application of LLMs could be merely using them as a text parser in text adventures. NPCs could continue to have some hidden, hard coded information or abilities, here stored in the context window, and the LLM is used to provide that information to the the player he (or she) puts in roughly the right words. Rather than exactly the right words.
  - maplethorpe 1y ago
    Instead of generating truly random dialogue, you could choose to only generate random seeds within a certain number range, so that the probability that two players seeing the same line of dialogue increases, while still having enough variety for it to feel "random" to a solo user.
  - ares623 1y ago
    I agree. An important aspect would be lost. It would be as interesting and relatable as sharing that dream you had last night with your friends. “Uh huh. Cool story bro.”
  - vidarh 1y ago
    It doesn't need to be random tidbits. You can add "life events" for the character to the prompt, and steer the responses heavily toward the character wanting to talk about those. With an LLM in the picture, you can even have the character remember past interactions with you, and expand on it different ways depending on how you interacted with them.
  - latexr 1y ago
    > will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. GLaDOS from Portal would offer one player pudding and another one a steak. You get to a wall which says “the ravioli is a fraud” and become utterly confused.
  - malfist 1y ago
    It's the whole debate about carefully created vs procedurely generated. LLM created dialog with be a vapid, vacuous and sterile as starfield was, or No Mans Sky
ehnto 1y ago
I would say local LLMs are already good enough for this. It might be an issue that the GPU is already quite busy at the time it's needed for LLM inference, but not an insurmountable issue. I wouldn't ever want a game to use it for the core story writing, because it's pretty important that it is consistent and unable to be derailed. But for less serious NPC interactions or like an RPG scenario it is such a great fit. I also wouldn't want a single player game to rely on remote inference, because that will get turned off eventually and then your game doesn't work.
- mungoman2 1y ago
  For story writing you generate into the framework of https://news.ycombinator.com/item?id=45134144 https://news.ycombinator.com/item?id=45134144
- azernik 1y ago
  I think generally, games should move to using the GPU less for graphics and more for computation. Not just AI computation - those fancy GPUs are a big resource that simulation games could be taking advantage of and just... aren't. (Yes, this is a Paradox callout. Give me less fancy particle effects in Vic3 and use the GPU for computing pop updates faster!) (Probably the biggest barrier to this is the lack of a convenient C++/C#-level cross-manufacturer compute API. Vulkan is a bit too low-level for game devs to work with, OpenCL kind of sucks, and CUDA is NVIDIA-only.)
serp002 1y ago
Agreed, this AI-ran DnD campaign is an example of how good/funny it can be. https://www.youtube.com/watch?v=TpYVyJBmH0g&ab_channel=DougDoug https://www.youtube.com/watch?v=TpYVyJBmH0g&ab_channel=DougD...
beckthompson 1y ago
I'm unsure how useful they'll really be honestly! In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc... There could probably be cool uses but I don't think it will be a pure "upgrade" as the repeating dialog is kind of a feature honestly. We'll have to see how it pans out xD
- heckelson 1y ago
  I guess you could just introduce a Symbol that marks the NPC as "I said everything I have" or gray out their text, or some other visual marker
  - anticrymactic 1y ago
    Or they could just tell you. Imagine talking to someone over and over again. They would tell u to get on with wherever you promised them.
    - heckelson 1y ago
      oh yeah that's much smarter than what I suggested
  - Terr_ 1y ago
    > a Symbol that marks the NPC That seems a bit like deck-chairs on the Titanic. The hard part isn't icon design, the hard part is (A) ensuring a clear list exists of what the NPC is supposed to ensure the user knows and (B) determining whether those goals were received successfully. For example, imagine a mystery/puzzle game where the NPC needs to inform the user of a clue for the next puzzle, but the LLM-layer botches it, either by generating dialogue that phrases it wrong, or by failing to fit it into the first response, so that the user must always do a few "extra" interactions anyway "just in case." I suppose you could... Feed the output into another document of "Did this NPC answer correctly" and feed it to another LLM... but down that path lies [more] madness.
    - 1y ago
hoskdoug 1y ago
Imagine finally hearing further dialogue from the Bucket Mouse after all these years.
arkh 1y ago
> Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers. The real immersion breaker and the holy grail of RPG is the fact NPC have no life or goal outside of what the player does. Imagine a game where NPCs have wants and goals and do things to get those done. Where you could leave it running for 10 years and things would have happened without you.
- latexr 1y ago
  That’s a nice experiment, but it’s a life simulator and not really a game. There’s little fun in working your way through the first two dungeons and then getting out and being told “no need to go on your epic quest anymore, someone else from the last town beat God and now we’re at peace. Oh, and they got all the powerful artefacts along the way, so there’s literally nothing of interest for you to do anymore. Unless you want to help us plant our crops. That’s what you wanted when you picked Final Fantasy, right? To play a poor version of Harvest Moon instead.”
  - wffurr 1y ago
    Simulation games are their own whole genre. Amazing stories have come out of games like Dwarf Fortress.
MacNCheese23 1y ago
There are plenty proof of concepts already done, for example in Skyrim https://www.youtube.com/watch?v=xNPF9VKmzxw https://www.youtube.com/watch?v=xNPF9VKmzxw Mods Name is https://art-from-the-machine.github.io/Mantella/ https://art-from-the-machine.github.io/Mantella/
nerdjon 1y ago
I find myself somewhere in the middle. I think for the vast majority of games this would largely be a negative. Anything heavily story driven this would just cause issues and I think would largely ruin the experience. I think sandbox games like animal crossing are the exception, if it ever becomes reliable enough to put in anything other than an M rated game. Animal Crossing, the Sims, Cult of the Lamb, and similar games would infinitely extend the life of the game. But I am sure we can all already imagine the headlines when these "family" games start saying things that they really should not be... especially given recent issues. For the record, not dismissing the person who did this work. But doing this commercially has its risks.
A_Duck 1y ago
You should check out https://www.meaningmachine.games/ https://www.meaningmachine.games/ then — they're building games with weird dynamics centred around this new possibility of LLM-powered NPCs
dbish 1y ago
I spent a lot of time on ML driven dialogue and natural language understanding for games in the pre-llm days working with some folks building Xbox, PC, and VR/AR games, and I'm excited about the potential here. The major barriers to shipping in a production game though are still the same imho: 1. (predictability) Games like to have a clear arc and tend to use at least some of their NPCs to move that forward, it is harder to do this with a model that could make a choice you don't predict. They tend to have a set of items, quests, what-have-you that you need boundaries around. 2. (testing) Games like to test like crazy before launch, at least the AAA ones, so their QA folks just don't like a model that can have infinite responses/variants. Many then drop to a skeletal crew for maintenance and improvements after launch, where with ML models you actually need to keep improving the model, finding long tail bugs as more players interact with the system, etc. 3. (cost) Games are usually very cost aware, it's far cheaper to just have a set human-written dialogue path, then to run a model, even an offline one. Cheaper in both actual dollar costs if you're talking about a high end LLM service call, and CPU/GPU/memory costs if you're talking an on-box system. 4. (internationalization/localization) AAA games need to launch fast to many languages and locales, using a model for NLP and dialogue management/natural language generation adds added testing costs for each new language, that is just a very cheap translation normally that can be outsourced. There have been some fun experiments in this space, and I expect to see this improve and become common use in the future, but it will take time and more work on how best to integrate a model into the flow of a game. I do love it for "presence" so talking to NPCs feels more human-like. Nick and the crew at AI Dungeon (and related) have always done some interesting work in this space, trying out games where AI can be used in interesting ways.
- vuciv 1y ago
  I would love to work on this. I have worked on this a ton and have really strong ideas for some. 1. I think people assume you have one LLM per character, but I think if you had specialized ones for each quest, item, etc.., this would actually work quite well. 3. I actually think if you cached responses under certain conditions, costs can be saved significantly. This would require quite a robust context, though, to still feel dynamic.
YesBox 1y ago
IMO this isn't going to be fun for most once the novelty wears off. Games are goal oriented at the end of the day and the great games are masterfully curated multi-disciplinary experiences. I'd argue throwing a game wrapper around an LLM is a new LLM experience, not a new game experience.
- BizarroLand 1y ago
  I think the people that are for this development are thinking more along the line of a game designed from the ground up to include AI LLM interaction. Once someone decides that will be a critical and fundamental part of a AAA game, the rest can be worked out despite what will be I am sure many unintended emergent behaviors.
d_burfoot 1y ago
> LLM-powered NPCs is one thing I'm most excited about in the future of gaming Incoming new type of health crisis: video game addiction coupled with LLM-induced psychosis. Dudes spending 12 hours a day farming gold in a MMORPG while flirting with their AI girlfriend sidekick
jerf 1y ago
There's a number of little games and a major mod for Skyrim that implements this. I've spent a bit of time playing with the Skyrim mod. There are some significant issues with it at the moment. One is that you have to train on vast swathes of text to get an LLM, and it's difficult after the fact to remove things after the fact. If you cooperate with the AI and stay "in Skyrim" with what you say to them it works out OK, but if you don't cooperate it becomes clear that Skyrim NPCs know something about Taylor Swift and Fox News, just to name two examples. LLMs in their current form basically can't solve this. The LLMs are also prone to writing checks the game can't cash. It's neat that the NPCs started talking about a perfectly plausible dungeon adventure they went on in a location that doesn't exist, but "felt" perfectly Skyrim-esque, but there's clearly some non-optimal aspects about that too. And again, this is basically not solvable with LLMs as they are currently constituted. Really slick experiences with this I think will require a generational change in AI technology. The Mantella mod is fun and all but it would be hard to sell that at scale right now as a gaming experience.
- testdelacc1 1y ago
  Feel like they could probably be used in text based adventures. Doki Doki Literature Club, despite the silly name, is a very unusual game that might really benefit from this.
- jhatemyjob 1y ago
  It's solvable it's just a lot of work. You need to have guardrails, i.e. multiple sanity-check passes (maybe there are other techniques, but I found this to be most effective) One concrete example I'm sure these Skyrim mods aren't using is: enums in structured outputs [1] with a finite list of locations/characters/topics/etc that are allowed to be discussed. The AI is not allowed to respond with anything that is not in the enum. So you can give it a list of all the locations in the game in a huge array and it would be forced to pick one. [1] https://platform.openai.com/docs/guides/structured-outputs#all-fields-must-be-required https://platform.openai.com/docs/guides/structured-outputs#a...
- faxmeyourcode 1y ago
  This is something fine tuning should be able to improve. The main caveats are the same as always: dataset collection, labeling, and training/experimentation time.

malfist 1y ago

I used to care about repeat scripted dialogue like you, until I took an arrow to the knee

Arisaka1 1y ago
Yep. I can see how relying on non-repetitive dialogue generated by LLM's will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. A way to bypass this problem is to keep the important dialogue fixed, but this only one of the problems. Another issue would be emphasizing the meaninglessness of the dialogue. For example, playing Trails in the Sky has lots of NPC dialogue that's repetitive, but at least the dialogue is relevant with how the NPC's life progresses in the grander scheme of things, such as having difficulty with her entrance exams, or having an argument with his fiancé. It's not main dialogue but adds flavor for anyone who cares about the world enough to interact with the citizens. I don't think I'd like to interact with characters that I know whatever it is they have to say is generated on the fly and adds nothing other than random tidbits. The novelty would quickly wear off.
- cubefox 1y ago
  A better application of LLMs could be merely using them as a text parser in text adventures. NPCs could continue to have some hidden, hard coded information or abilities, here stored in the context window, and the LLM is used to provide that information to the the player he (or she) puts in roughly the right words. Rather than exactly the right words.
- maplethorpe 1y ago
  Instead of generating truly random dialogue, you could choose to only generate random seeds within a certain number range, so that the probability that two players seeing the same line of dialogue increases, while still having enough variety for it to feel "random" to a solo user.
- ares623 1y ago
  I agree. An important aspect would be lost. It would be as interesting and relatable as sharing that dream you had last night with your friends. “Uh huh. Cool story bro.”
- vidarh 1y ago
  It doesn't need to be random tidbits. You can add "life events" for the character to the prompt, and steer the responses heavily toward the character wanting to talk about those. With an LLM in the picture, you can even have the character remember past interactions with you, and expand on it different ways depending on how you interacted with them.
- latexr 1y ago
  > will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. GLaDOS from Portal would offer one player pudding and another one a steak. You get to a wall which says “the ravioli is a fraud” and become utterly confused.
- malfist 1y ago
  It's the whole debate about carefully created vs procedurely generated. LLM created dialog with be a vapid, vacuous and sterile as starfield was, or No Mans Sky

ehnto 1y ago

I would say local LLMs are already good enough for this. It might be an issue that the GPU is already quite busy at the time it's needed for LLM inference, but not an insurmountable issue. I wouldn't ever want a game to use it for the core story writing, because it's pretty important that it is consistent and unable to be derailed. But for less serious NPC interactions or like an RPG scenario it is such a great fit. I also wouldn't want a single player game to rely on remote inference, because that will get turned off eventually and then your game doesn't work.

mungoman2 1y ago
For story writing you generate into the framework of https://news.ycombinator.com/item?id=45134144 https://news.ycombinator.com/item?id=45134144
azernik 1y ago
I think generally, games should move to using the GPU less for graphics and more for computation. Not just AI computation - those fancy GPUs are a big resource that simulation games could be taking advantage of and just... aren't. (Yes, this is a Paradox callout. Give me less fancy particle effects in Vic3 and use the GPU for computing pop updates faster!) (Probably the biggest barrier to this is the lack of a convenient C++/C#-level cross-manufacturer compute API. Vulkan is a bit too low-level for game devs to work with, OpenCL kind of sucks, and CUDA is NVIDIA-only.)

serp002 1y ago

Agreed, this AI-ran DnD campaign is an example of how good/funny it can be. https://www.youtube.com/watch?v=TpYVyJBmH0g&ab_channel=DougDoug https://www.youtube.com/watch?v=TpYVyJBmH0g&ab_channel=DougD...

beckthompson 1y ago

I'm unsure how useful they'll really be honestly! In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc... There could probably be cool uses but I don't think it will be a pure "upgrade" as the repeating dialog is kind of a feature honestly. We'll have to see how it pans out xD

heckelson 1y ago
I guess you could just introduce a Symbol that marks the NPC as "I said everything I have" or gray out their text, or some other visual marker
- anticrymactic 1y ago
  Or they could just tell you. Imagine talking to someone over and over again. They would tell u to get on with wherever you promised them.
  - heckelson 1y ago
    oh yeah that's much smarter than what I suggested
- Terr_ 1y ago
  > a Symbol that marks the NPC That seems a bit like deck-chairs on the Titanic. The hard part isn't icon design, the hard part is (A) ensuring a clear list exists of what the NPC is supposed to ensure the user knows and (B) determining whether those goals were received successfully. For example, imagine a mystery/puzzle game where the NPC needs to inform the user of a clue for the next puzzle, but the LLM-layer botches it, either by generating dialogue that phrases it wrong, or by failing to fit it into the first response, so that the user must always do a few "extra" interactions anyway "just in case." I suppose you could... Feed the output into another document of "Did this NPC answer correctly" and feed it to another LLM... but down that path lies [more] madness.
  - 1y ago

hoskdoug 1y ago

Imagine finally hearing further dialogue from the Bucket Mouse after all these years.

arkh 1y ago

> Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers. The real immersion breaker and the holy grail of RPG is the fact NPC have no life or goal outside of what the player does. Imagine a game where NPCs have wants and goals and do things to get those done. Where you could leave it running for 10 years and things would have happened without you.

latexr 1y ago
That’s a nice experiment, but it’s a life simulator and not really a game. There’s little fun in working your way through the first two dungeons and then getting out and being told “no need to go on your epic quest anymore, someone else from the last town beat God and now we’re at peace. Oh, and they got all the powerful artefacts along the way, so there’s literally nothing of interest for you to do anymore. Unless you want to help us plant our crops. That’s what you wanted when you picked Final Fantasy, right? To play a poor version of Harvest Moon instead.”
- wffurr 1y ago
  Simulation games are their own whole genre. Amazing stories have come out of games like Dwarf Fortress.

MacNCheese23 1y ago

There are plenty proof of concepts already done, for example in Skyrim https://www.youtube.com/watch?v=xNPF9VKmzxw https://www.youtube.com/watch?v=xNPF9VKmzxw Mods Name is https://art-from-the-machine.github.io/Mantella/ https://art-from-the-machine.github.io/Mantella/

nerdjon 1y ago

I find myself somewhere in the middle. I think for the vast majority of games this would largely be a negative. Anything heavily story driven this would just cause issues and I think would largely ruin the experience. I think sandbox games like animal crossing are the exception, if it ever becomes reliable enough to put in anything other than an M rated game. Animal Crossing, the Sims, Cult of the Lamb, and similar games would infinitely extend the life of the game. But I am sure we can all already imagine the headlines when these "family" games start saying things that they really should not be... especially given recent issues. For the record, not dismissing the person who did this work. But doing this commercially has its risks.

A_Duck 1y ago

You should check out https://www.meaningmachine.games/ https://www.meaningmachine.games/ then — they're building games with weird dynamics centred around this new possibility of LLM-powered NPCs

dbish 1y ago

I spent a lot of time on ML driven dialogue and natural language understanding for games in the pre-llm days working with some folks building Xbox, PC, and VR/AR games, and I'm excited about the potential here. The major barriers to shipping in a production game though are still the same imho: 1. (predictability) Games like to have a clear arc and tend to use at least some of their NPCs to move that forward, it is harder to do this with a model that could make a choice you don't predict. They tend to have a set of items, quests, what-have-you that you need boundaries around. 2. (testing) Games like to test like crazy before launch, at least the AAA ones, so their QA folks just don't like a model that can have infinite responses/variants. Many then drop to a skeletal crew for maintenance and improvements after launch, where with ML models you actually need to keep improving the model, finding long tail bugs as more players interact with the system, etc. 3. (cost) Games are usually very cost aware, it's far cheaper to just have a set human-written dialogue path, then to run a model, even an offline one. Cheaper in both actual dollar costs if you're talking about a high end LLM service call, and CPU/GPU/memory costs if you're talking an on-box system. 4. (internationalization/localization) AAA games need to launch fast to many languages and locales, using a model for NLP and dialogue management/natural language generation adds added testing costs for each new language, that is just a very cheap translation normally that can be outsourced. There have been some fun experiments in this space, and I expect to see this improve and become common use in the future, but it will take time and more work on how best to integrate a model into the flow of a game. I do love it for "presence" so talking to NPCs feels more human-like. Nick and the crew at AI Dungeon (and related) have always done some interesting work in this space, trying out games where AI can be used in interesting ways.

vuciv 1y ago
I would love to work on this. I have worked on this a ton and have really strong ideas for some. 1. I think people assume you have one LLM per character, but I think if you had specialized ones for each quest, item, etc.., this would actually work quite well. 3. I actually think if you cached responses under certain conditions, costs can be saved significantly. This would require quite a robust context, though, to still feel dynamic.

YesBox 1y ago

IMO this isn't going to be fun for most once the novelty wears off. Games are goal oriented at the end of the day and the great games are masterfully curated multi-disciplinary experiences. I'd argue throwing a game wrapper around an LLM is a new LLM experience, not a new game experience.

BizarroLand 1y ago
I think the people that are for this development are thinking more along the line of a game designed from the ground up to include AI LLM interaction. Once someone decides that will be a critical and fundamental part of a AAA game, the rest can be worked out despite what will be I am sure many unintended emergent behaviors.

d_burfoot 1y ago

> LLM-powered NPCs is one thing I'm most excited about in the future of gaming Incoming new type of health crisis: video game addiction coupled with LLM-induced psychosis. Dudes spending 12 hours a day farming gold in a MMORPG while flirting with their AI girlfriend sidekick

jerf 1y ago

There's a number of little games and a major mod for Skyrim that implements this. I've spent a bit of time playing with the Skyrim mod. There are some significant issues with it at the moment. One is that you have to train on vast swathes of text to get an LLM, and it's difficult after the fact to remove things after the fact. If you cooperate with the AI and stay "in Skyrim" with what you say to them it works out OK, but if you don't cooperate it becomes clear that Skyrim NPCs know something about Taylor Swift and Fox News, just to name two examples. LLMs in their current form basically can't solve this. The LLMs are also prone to writing checks the game can't cash. It's neat that the NPCs started talking about a perfectly plausible dungeon adventure they went on in a location that doesn't exist, but "felt" perfectly Skyrim-esque, but there's clearly some non-optimal aspects about that too. And again, this is basically not solvable with LLMs as they are currently constituted. Really slick experiences with this I think will require a generational change in AI technology. The Mantella mod is fun and all but it would be hard to sell that at scale right now as a gaming experience.

testdelacc1 1y ago
Feel like they could probably be used in text based adventures. Doki Doki Literature Club, despite the silly name, is a very unusual game that might really benefit from this.
jhatemyjob 1y ago
It's solvable it's just a lot of work. You need to have guardrails, i.e. multiple sanity-check passes (maybe there are other techniques, but I found this to be most effective) One concrete example I'm sure these Skyrim mods aren't using is: enums in structured outputs [1] with a finite list of locations/characters/topics/etc that are allowed to be discussed. The AI is not allowed to respond with anything that is not in the enum. So you can give it a list of all the locations in the game in a huge array and it would be forced to pick one. [1] https://platform.openai.com/docs/guides/structured-outputs#all-fields-must-be-required https://platform.openai.com/docs/guides/structured-outputs#a...
faxmeyourcode 1y ago
This is something fine tuning should be able to improve. The main caveats are the same as always: dataset collection, labeling, and training/experimentation time.