7 ms·

Okay, this is probably the right place to say this: You can't play D&D with GPT-3/3.5, but you absolutely can with GPT-4. GPT-4 is a solid DM. GPT-3 couldn't

Okay, this is probably the right place to say this: You can't play D&D with GPT-3/3.5, but you absolutely can with GPT-4. GPT-4 is a solid DM. GPT-3 couldn't remember what it said 30 minutes ago, and was basically incapable of planning, so unless you want a simple dungeon rougelike-esque adventure, it'll lose track of things and start hallucinating. It will also sometimes forget the rules. My initial impressions are that GPT-4 is nearly as competent as a human DM, and much faster and more flexible. Also, unlike GPT-3, it will roll dice for you, and it knows the rules just as well as most players. These maps + GPT-4 could = infinite campaigns. Perhaps it's just a matter of time -- and just a few years, at that -- before generative text and graphic AIs merge, and games like Baldur's Gate can quite literally be made on the fly.

Jensson 4y ago
> Also, unlike GPT-3, it will roll dice for you I'd question the fairness of those rolls. But yeah, DMing seems like an excellent fit for these models. But you probably need to track hitpoints and stats separately, GPT-4 can't do math either. Or I guess it not doing math might be a feature, that way it will tell you what reasonably happens with fuzzy logic, might be a more immersive experience.
- hhjinks 4y ago
  It would be interesting to see the dsitribution of dice rolls when asking GPT-3 and 4 to roll a dice repeatedly.
  - simonh 4y ago
    I hope they didn't train it on transcripts of XKCD. https://xkcd.com/221/ https://xkcd.com/221/
- BiteCode_dev 4y ago
  You don't need it to be fair, you need it to be fun. Most video games play with randomness so that the distribution serves the gameplay, sometimes bypassing completely the RNG to balance frustration/satisfaction. GPT can learn do to that: not providing true randomness, but the illusion of it to serve the game.
  - pessimizer 4y ago
    Players will detect drama being injected into their randomness (unless it is very gently and rarely done), and over time they will start to be able to predict dice rolls [edit: or even learn to shape their responses to trick the model into generating the rolls they want.] That will ruin the game.
    - BiteCode_dev 4y ago
      Xcom 2 make hits more likely the more you miss because fair proba was frustrating the players when they missed a shot with 95% of success. So I'm not sure you are right.
      - Jensson 4y ago
        A 95% shot will never miss in Xcom 2 unless you play on harder difficulties. It multiplies hit chance by 1.1 so 91% is 100%.
        deleted 4y ago
        [deleted]
        pessimizer 4y ago
        So that's not removing randomness, it's just grade inflation. I suppose it's also secret range narrowing, since an 3% improvement from 92% to 95% is actually no improvement at all at 1.1 * (random) capped at 100%.
jagermo 4y ago
It is also fantastic to create on the fly scores for Blades in the Dark, surprisingly detailed and able to flesh them out. Runs for Shadowrun are also possible, really nice. Great for systems where you don't have a lot of published adventures.
globalnode 4y ago
This has to be the most inspiring comment on HN ive ever seen. I must live to see this.
mhd 4y ago
> My initial impressions are that GPT-4 is nearly as competent as a human DM I wonder what kind of campaigns we're talking about here. I mean, if it's just mindless modern encounter-based design, sure, I can believe that. Kicking down doors, rolling for initiative, GOTO 10. But even mildly more moderate DM work like re-jiggering set up monsters to react to the actions of the players (in e.g. a Caves of Chaos type of setup)?
- A_D_E_P_T 4y ago
  See, that's the thing. No matter how much effort you put in to prompting and coaxing it, GPT-3 really couldn't manage anything but the simplest encounter-based game design. There were things that GPT-3 did really well -- it could write you a 600-word essay on the characteristics and atmosphere of each chamber or passage you find yourself in -- but it couldn't stick to a narrative or engage in anything resembling long-term planning. So, ultimately, you're left with something quite strange and dreamlike: A campaign that's vividly -- even hyper-realistically -- described, but where nothing makes sense and there's no logical continuity over time. GPT-4 knows what it's doing, though. It's very different. As far as I can tell right now, it combines the outstanding descriptive capabilities of GPT-3 with the ability to (a) remember things it has previously said and establish logical relations over time, and (b) make sure that the campaign appropriately mirrors a "normal" D&D campaign. GPT-3 would frequently mess up and have you fight dragons at Level 1; GPT-4 more appropriately sticks to goblins and bandits. And, when it comes to tactics, pacing, etc., it's really quite good. I suppose it's a little bit conventionally-minded, but that's to be expected. Almost scary to think that it'll only get better from here on out, because GPT-4 is damn near "good human DM" tier, and already much better than a bad DM.
- thom 4y ago
  I've been extremely impressed with GPT-4 so far, generating a campaign for my kids. The encounters it created were very well fleshed out (monsters, minions, stats and abilities for all, including very flavourful lair actions for bosses). It's given me NPCs and sample dialogue for each, locations, and a central storyline that is well-threaded throughout. I was genuinely astonished even at its first attempt. I don't know if it could stand up on its own as an interactive dungeon master, but as a tool I am loving it.
poulpy123 4y ago
in my eyes the big problem is that these model are purely statistics and are unable to apply logic
JimWestergren 4y ago
I am working on something like this, text-based with images. But I don't generate on the fly, I will have big lists (many thousands) of curated text outputs from GPT that get selected with randomization and nested if statements. For those interested, the game (work in progress) is Advinsula: https://advinsula.com/ https://advinsula.com/
paulluuk 4y ago
I spent some weeks trying to make a GPT-3 D&D GM, but indeed it just won't work. I haven't tried GPT-4 yet but this post makes me want to test it for sure. The problem I had with GPT-3, most of all, was that it tries to "finish the story". E.g. if the players are on a grand campaign to defeat some big bad boss, then this might happen: GM (GPT): You enter the tavern, and you see a halfling bartender and a few gnome patrons. Player: I ask the bartender for information about <BIG BAD EVIL GUY> GM (GPT): The bartender tells you <BIG BAD EVIL GUY> lives in a castle. You and your party storm the castle and manage to defeat <BIG BAD EVIL GUY>.
- rafabulsing 4y ago
  I managed to get a bit better than that story-wise, but I really felt this problem with battles, it just refused to do an actual battle. When a battle scene started, it always rushed to "after a long and tiring fight, you eventually come out victorious" or something to that effect.
BiteCode_dev 4y ago
When you think of the content of the books in Baldur's Gate and the sheer amount of human hours poured into something just to give background to the game, this is going to save indeed a lot of time. But on the other hand, just like open world was a terrible curse with hundred of games feeling empty, I'm afraid we will see a wave of games with a lot of generated but eventually uninteresting content because this is so easy to do. After all, good content is coherent, and filtered. It's not just about producing a lot, it's what you don't put in the game, and what you do to glue things together. I sure GPT will eventually be better at it than humans, but for now, having a human veto is necessary to have a good noise/signal ratio and because GPT is "good enough", being lazy is tempting. But yeah, I can confirm GPT4 is already quite good at it and will only improve from this point. This is going to be very interesting years. Edit: I just made it write a full video game scenario, using BG as a competitor for basis. At first it was average, but with directions, it refined it and became quite good. Some kind of steam punk guild called the Navigators reading the wind, living in floating islands and a girl rediscovering the earth bellow the clouds and rebuilding old aliances with lost civilizations, plus a politician leader trying to prevent that and even a mentor to help her. Not bad. I would play this, and even read it. You can see the inspiration from "The Pillars of the Earth", "Gunnm", "Baldur's Gate" and "nausica" proving again it's mostly assembling known stuff. But a cake is mostly assembling known stuff and it's still good. The "last of us" is just a good take on something that has been done a thousand times. So why not?
- Asooka 4y ago
  What's missing is overt structure. More or less the same reason as to why AI can't draw hands - there is no "understanding" of the fact that a hand is a rectangle with five sticks coming out of it in a specific arrangement. For a human, that is all that's needed to recognise a hand, i.e. | | | | | | | | | | | | | | | | | | | | | +---+---+---+ | | | | | | | | | . | | / | | / +-----------+-+ This is a (very crude) hand. This registers as a hand for you, because it has the structure of a hand. This is less similar to a photorealistic hand than a 6-fingered AI hand, objectively, however it fits the human understanding of what a hand is supposed to be, because it has the right kind of shapes in the right relations to one another. In the same vein, if we could get the AI to understand the structure of a story, it should be able to DM session by session, starting from a high level abstract idea and refining it based on player actions, incorporating player input into the game. I'm not sure if hierarchical abstract reasoning should be forcibly added to the model, or if it can be emergent in a specific configuration, but if we figure out how to build an AI with that, then it should be able to produce something a lot more sensible.
andrepd 4y ago
That's exactly what everyone said about GPT3, two or three years ago > Before, you couldn't have a chatbot run a D&D campaign. But GPT3 is a solid DM and then when all the hype died down to acceptable levels, turns out it was not. So forgive me if I stay skeptical until I see convincing evidence.
zug_zug 4y ago
Played a solo-campgain with GPT4 for 1-2 hours yesterday, summary: - Created character sheet for me, at desired level with spells and abilities - Was able to add a cat to the game upon request - Took hints. - Was an outstanding writer, far better than any DM in terms of sensory descriptions and articulation - Had to ask it to only tell 5 paragraphs at a time - Performed combat rolls and math, all apparently correctly for a long while - The first 20 rolls were all greater than 10, asked it give more random rolls and it seemingly did for a while - Added elements on the fly consistently without losing track mostly - Used puzzles and challenges and created a sense of danger. Adapted well on the fly to unusual character behavior (guessing an unreasonable answer for a puzzle solution). - After a few hours it started confusing classes in my party (I think there is some context window). It forgot I had agonizing blast after a few hours. - Sometimes fudged distances Overall for a DM with infinite patience and an entirely customizable campaign on the fly with no need to plan/recover it seems promising at first. But some sort of prompt engineering will eventually prove necessary i think.
- simonh 4y ago
  It might be possible to 'extend' the context window by summarising important information from time to time, to refresh it's memory.
- elevaet 4y ago
  Did it feel fun?
  - zug_zug 4y ago
    Definitely did for me, but I also wonder if I learned the limitations/weaknesses of the AI too much that it would break the immersion.
    - elevaet 4y ago
      It's dizzying to think about how immersive these things will become after the next however-many iterations that are foreseeable.
recuter 4y ago
Can confirm, saw somebody post this here the other day: https://www.youtube.com/watch?v=H-89vnqxkFg https://www.youtube.com/watch?v=H-89vnqxkFg I don't think it is a matter of years, it is a matter of months.
keithalewis 4y ago
But can GPT-4 suggest tunes for Nero to play while Rome burns?
- meh8881 4y ago
  Yes
  - keithalewis 4y ago
    People dream every night for an hour or two. If you can remember your dream and type it into ChatGPT can it interpret it? "Yes" is not an answer.
    - meh8881 4y ago
      Yes, it can. Why is Yes not an answer?
airstrike 4y ago
> Perhaps it's just a matter of time -- and just a few years, at that -- before generative text and graphic AIs merge, and games like Baldur's Gate can quite literally be made on the fly. All we need after that is an AI generated musical theme / background ambience / sound effects. This came to mind while listening to https://open.spotify.com/episode/1xs8fxI5ZdaIA1i386IHH2 https://open.spotify.com/episode/1xs8fxI5ZdaIA1i386IHH2 yesterday
JaDogg 4y ago
Yes! with enough generative AI capabilities (Star Trek) Holodeck can be achieved.
- red-iron-pine 4y ago
  Yes but will it be able to make the holograms come to life and take over the bridge?
mungoman2 4y ago
How does GPT-4 remember things this long? The memory of it is just 8k / 32k tokens long, right? Seems you would run out of tokens quickly.
katabasis 4y ago
I run a small D&D group with a couple of friends. I don't know that I'd want to play in a campaign helmed by an AI dungeon master, but the idea of having some help generating plot, assets, etc. when I don't have a lot of time before a game session sounds amazing. I tend to be an AI pessimist, but this seems like a good example of a way that AI could be used to supplement human creativity instead of just replacing it entirely.
- StrictDabbler 4y ago
  This whole thread has me re-evaluating my prior assessments of "Ender's Game" and the Mind Game it described. When the book came out I was a child programmer and I had total contempt for the idea that a player could just decide to burrow into the eye of the giant and that the engine would generate a result. I was used to Sierra games where you'd click an inventory item on something you saw and the game would tell you "there's nothing useful you can do with that."
- lkschubert8 4y ago
  I actually currently do this ( for cyberpunk red not D&D). What I will say is it has let me be way more open in letting the game go where the players want it. If they decide chasing after some thread that I wasn't even thinking about is what they want to do, I can stall for a bit and let chatgpt come up with some characters, locations, and interesting skill checks.

Jensson 4y ago

> Also, unlike GPT-3, it will roll dice for you I'd question the fairness of those rolls. But yeah, DMing seems like an excellent fit for these models. But you probably need to track hitpoints and stats separately, GPT-4 can't do math either. Or I guess it not doing math might be a feature, that way it will tell you what reasonably happens with fuzzy logic, might be a more immersive experience.

hhjinks 4y ago
It would be interesting to see the dsitribution of dice rolls when asking GPT-3 and 4 to roll a dice repeatedly.
- simonh 4y ago
  I hope they didn't train it on transcripts of XKCD. https://xkcd.com/221/ https://xkcd.com/221/
BiteCode_dev 4y ago
You don't need it to be fair, you need it to be fun. Most video games play with randomness so that the distribution serves the gameplay, sometimes bypassing completely the RNG to balance frustration/satisfaction. GPT can learn do to that: not providing true randomness, but the illusion of it to serve the game.
- pessimizer 4y ago
  Players will detect drama being injected into their randomness (unless it is very gently and rarely done), and over time they will start to be able to predict dice rolls [edit: or even learn to shape their responses to trick the model into generating the rolls they want.] That will ruin the game.
  - BiteCode_dev 4y ago
    Xcom 2 make hits more likely the more you miss because fair proba was frustrating the players when they missed a shot with 95% of success. So I'm not sure you are right.
    - Jensson 4y ago
      A 95% shot will never miss in Xcom 2 unless you play on harder difficulties. It multiplies hit chance by 1.1 so 91% is 100%.
      - deleted 4y ago
        [deleted]
      - pessimizer 4y ago
        So that's not removing randomness, it's just grade inflation. I suppose it's also secret range narrowing, since an 3% improvement from 92% to 95% is actually no improvement at all at 1.1 * (random) capped at 100%.

jagermo 4y ago

It is also fantastic to create on the fly scores for Blades in the Dark, surprisingly detailed and able to flesh them out. Runs for Shadowrun are also possible, really nice. Great for systems where you don't have a lot of published adventures.

globalnode 4y ago

This has to be the most inspiring comment on HN ive ever seen. I must live to see this.

mhd 4y ago

> My initial impressions are that GPT-4 is nearly as competent as a human DM I wonder what kind of campaigns we're talking about here. I mean, if it's just mindless modern encounter-based design, sure, I can believe that. Kicking down doors, rolling for initiative, GOTO 10. But even mildly more moderate DM work like re-jiggering set up monsters to react to the actions of the players (in e.g. a Caves of Chaos type of setup)?

A_D_E_P_T 4y ago
See, that's the thing. No matter how much effort you put in to prompting and coaxing it, GPT-3 really couldn't manage anything but the simplest encounter-based game design. There were things that GPT-3 did really well -- it could write you a 600-word essay on the characteristics and atmosphere of each chamber or passage you find yourself in -- but it couldn't stick to a narrative or engage in anything resembling long-term planning. So, ultimately, you're left with something quite strange and dreamlike: A campaign that's vividly -- even hyper-realistically -- described, but where nothing makes sense and there's no logical continuity over time. GPT-4 knows what it's doing, though. It's very different. As far as I can tell right now, it combines the outstanding descriptive capabilities of GPT-3 with the ability to (a) remember things it has previously said and establish logical relations over time, and (b) make sure that the campaign appropriately mirrors a "normal" D&D campaign. GPT-3 would frequently mess up and have you fight dragons at Level 1; GPT-4 more appropriately sticks to goblins and bandits. And, when it comes to tactics, pacing, etc., it's really quite good. I suppose it's a little bit conventionally-minded, but that's to be expected. Almost scary to think that it'll only get better from here on out, because GPT-4 is damn near "good human DM" tier, and already much better than a bad DM.
thom 4y ago
I've been extremely impressed with GPT-4 so far, generating a campaign for my kids. The encounters it created were very well fleshed out (monsters, minions, stats and abilities for all, including very flavourful lair actions for bosses). It's given me NPCs and sample dialogue for each, locations, and a central storyline that is well-threaded throughout. I was genuinely astonished even at its first attempt. I don't know if it could stand up on its own as an interactive dungeon master, but as a tool I am loving it.

poulpy123 4y ago

in my eyes the big problem is that these model are purely statistics and are unable to apply logic

JimWestergren 4y ago

I am working on something like this, text-based with images. But I don't generate on the fly, I will have big lists (many thousands) of curated text outputs from GPT that get selected with randomization and nested if statements. For those interested, the game (work in progress) is Advinsula: https://advinsula.com/ https://advinsula.com/

paulluuk 4y ago

I spent some weeks trying to make a GPT-3 D&D GM, but indeed it just won't work. I haven't tried GPT-4 yet but this post makes me want to test it for sure. The problem I had with GPT-3, most of all, was that it tries to "finish the story". E.g. if the players are on a grand campaign to defeat some big bad boss, then this might happen: GM (GPT): You enter the tavern, and you see a halfling bartender and a few gnome patrons. Player: I ask the bartender for information about <BIG BAD EVIL GUY> GM (GPT): The bartender tells you <BIG BAD EVIL GUY> lives in a castle. You and your party storm the castle and manage to defeat <BIG BAD EVIL GUY>.

rafabulsing 4y ago
I managed to get a bit better than that story-wise, but I really felt this problem with battles, it just refused to do an actual battle. When a battle scene started, it always rushed to "after a long and tiring fight, you eventually come out victorious" or something to that effect.

BiteCode_dev 4y ago

When you think of the content of the books in Baldur's Gate and the sheer amount of human hours poured into something just to give background to the game, this is going to save indeed a lot of time. But on the other hand, just like open world was a terrible curse with hundred of games feeling empty, I'm afraid we will see a wave of games with a lot of generated but eventually uninteresting content because this is so easy to do. After all, good content is coherent, and filtered. It's not just about producing a lot, it's what you don't put in the game, and what you do to glue things together. I sure GPT will eventually be better at it than humans, but for now, having a human veto is necessary to have a good noise/signal ratio and because GPT is "good enough", being lazy is tempting. But yeah, I can confirm GPT4 is already quite good at it and will only improve from this point. This is going to be very interesting years. Edit: I just made it write a full video game scenario, using BG as a competitor for basis. At first it was average, but with directions, it refined it and became quite good. Some kind of steam punk guild called the Navigators reading the wind, living in floating islands and a girl rediscovering the earth bellow the clouds and rebuilding old aliances with lost civilizations, plus a politician leader trying to prevent that and even a mentor to help her. Not bad. I would play this, and even read it. You can see the inspiration from "The Pillars of the Earth", "Gunnm", "Baldur's Gate" and "nausica" proving again it's mostly assembling known stuff. But a cake is mostly assembling known stuff and it's still good. The "last of us" is just a good take on something that has been done a thousand times. So why not?

Asooka 4y ago
What's missing is overt structure. More or less the same reason as to why AI can't draw hands - there is no "understanding" of the fact that a hand is a rectangle with five sticks coming out of it in a specific arrangement. For a human, that is all that's needed to recognise a hand, i.e. | | | | | | | | | | | | | | | | | | | | | +---+---+---+ | | | | | | | | | . | | / | | / +-----------+-+ This is a (very crude) hand. This registers as a hand for you, because it has the structure of a hand. This is less similar to a photorealistic hand than a 6-fingered AI hand, objectively, however it fits the human understanding of what a hand is supposed to be, because it has the right kind of shapes in the right relations to one another. In the same vein, if we could get the AI to understand the structure of a story, it should be able to DM session by session, starting from a high level abstract idea and refining it based on player actions, incorporating player input into the game. I'm not sure if hierarchical abstract reasoning should be forcibly added to the model, or if it can be emergent in a specific configuration, but if we figure out how to build an AI with that, then it should be able to produce something a lot more sensible.

andrepd 4y ago

That's exactly what everyone said about GPT3, two or three years ago > Before, you couldn't have a chatbot run a D&D campaign. But GPT3 is a solid DM and then when all the hype died down to acceptable levels, turns out it was not. So forgive me if I stay skeptical until I see convincing evidence.

zug_zug 4y ago

Played a solo-campgain with GPT4 for 1-2 hours yesterday, summary: - Created character sheet for me, at desired level with spells and abilities - Was able to add a cat to the game upon request - Took hints. - Was an outstanding writer, far better than any DM in terms of sensory descriptions and articulation - Had to ask it to only tell 5 paragraphs at a time - Performed combat rolls and math, all apparently correctly for a long while - The first 20 rolls were all greater than 10, asked it give more random rolls and it seemingly did for a while - Added elements on the fly consistently without losing track mostly - Used puzzles and challenges and created a sense of danger. Adapted well on the fly to unusual character behavior (guessing an unreasonable answer for a puzzle solution). - After a few hours it started confusing classes in my party (I think there is some context window). It forgot I had agonizing blast after a few hours. - Sometimes fudged distances Overall for a DM with infinite patience and an entirely customizable campaign on the fly with no need to plan/recover it seems promising at first. But some sort of prompt engineering will eventually prove necessary i think.

simonh 4y ago
It might be possible to 'extend' the context window by summarising important information from time to time, to refresh it's memory.
elevaet 4y ago
Did it feel fun?
- zug_zug 4y ago
  Definitely did for me, but I also wonder if I learned the limitations/weaknesses of the AI too much that it would break the immersion.
  - elevaet 4y ago
    It's dizzying to think about how immersive these things will become after the next however-many iterations that are foreseeable.

recuter 4y ago

Can confirm, saw somebody post this here the other day: https://www.youtube.com/watch?v=H-89vnqxkFg https://www.youtube.com/watch?v=H-89vnqxkFg I don't think it is a matter of years, it is a matter of months.

keithalewis 4y ago

But can GPT-4 suggest tunes for Nero to play while Rome burns?

meh8881 4y ago
Yes
- keithalewis 4y ago
  People dream every night for an hour or two. If you can remember your dream and type it into ChatGPT can it interpret it? "Yes" is not an answer.
  - meh8881 4y ago
    Yes, it can. Why is Yes not an answer?

airstrike 4y ago

> Perhaps it's just a matter of time -- and just a few years, at that -- before generative text and graphic AIs merge, and games like Baldur's Gate can quite literally be made on the fly. All we need after that is an AI generated musical theme / background ambience / sound effects. This came to mind while listening to https://open.spotify.com/episode/1xs8fxI5ZdaIA1i386IHH2 https://open.spotify.com/episode/1xs8fxI5ZdaIA1i386IHH2 yesterday

JaDogg 4y ago

Yes! with enough generative AI capabilities (Star Trek) Holodeck can be achieved.

red-iron-pine 4y ago
Yes but will it be able to make the holograms come to life and take over the bridge?

mungoman2 4y ago

How does GPT-4 remember things this long? The memory of it is just 8k / 32k tokens long, right? Seems you would run out of tokens quickly.

katabasis 4y ago

I run a small D&D group with a couple of friends. I don't know that I'd want to play in a campaign helmed by an AI dungeon master, but the idea of having some help generating plot, assets, etc. when I don't have a lot of time before a game session sounds amazing. I tend to be an AI pessimist, but this seems like a good example of a way that AI could be used to supplement human creativity instead of just replacing it entirely.

StrictDabbler 4y ago
This whole thread has me re-evaluating my prior assessments of "Ender's Game" and the Mind Game it described. When the book came out I was a child programmer and I had total contempt for the idea that a player could just decide to burrow into the eye of the giant and that the engine would generate a result. I was used to Sierra games where you'd click an inventory item on something you saw and the game would tell you "there's nothing useful you can do with that."
lkschubert8 4y ago
I actually currently do this ( for cyberpunk red not D&D). What I will say is it has let me be way more open in letting the game go where the players want it. If they decide chasing after some thread that I wasn't even thinking about is what they want to do, I can stall for a bit and let chatgpt come up with some characters, locations, and interesting skill checks.