6 ms·

I really hope this tech will be miniaturized at some point. I hate to be that guy, but when I saw the article the other day that ended with "plot twist, this a

I really hope this tech will be miniaturized at some point. I hate to be that guy, but when I saw the article the other day that ended with "plot twist, this article was autogenerated with GPT-3!", I was not impressed, mainly because it looked like content farm's content to me, conveying no information. It basically looked like an incredibly costly spam tool. But then I thought of possible applications. Give humans a new tool, and they will amaze you, I'm sure we'll see cool applications of this tech in the future (and probably horrible ones too). One thing that I could think of where it would totally rock : games. If games were able to use such tech, their game world could be filled with casual discussions rather than the developers needing to fill everything. This means that NPCs in a game could have ever changing discussions, and even answer to the player about the subject they're discussing. Their discussion could even change based on what is happening in the world or what the players are doing, even to things with the smallest impact, or what is happening directly around them at the moment, without devs needing to script it all. That would be awesome. But yeah, this won't happen easily unless the model can be embedded and shipped on local computers.

puranjay 6y ago
That would be a great use. It really takes me out of a game when I talk to an NPC and they run out of dialog options. After that, every time you approach them, they repeat the last thing they said.
- xamuel 6y ago
  I predict it wouldn't be as fun as it might initially sound. NPC dialog is part of level design just like all the rest of the level it occurs on. With few exceptions (such as roguelikes), random level generation tends to produce bland cookie-cutter levels; I don't see why dialog would be any different. The reason it's fun to talk to NPCs in a game like Chrono Trigger or FF7 is because someone put the work in to make their dialog interesting and relevant and fun. Where I could see it working better would be for e.g. newspaper headlines in a grand-strategy or SimCity-like game. When you do something crazy like have Lichtenstein conquer Western Europe, it could be funny to have some auto-generated commentary on said crazy state of affairs.
  - devalgo 6y ago
    It offers you procedurally generated dialogue that can be relevant to any potential player actions. You decide to kill the town chicken and now all the townspeople are talking about it. You've freed up devs from having to hard code all these possible interactions, obviously not perfect but it allows for some interesting possibilities.
skoll43 6y ago
Just imagine a caves of qud weirdly coherent storytelling be the base of new games, I LOVE THE FUTURE
- deleted 6y ago
  [deleted]
yyyk 6y ago
Hm. GPT-3 is trained on internet data from the real world. Your NPCs are in the gameworld. I guess you wouldn't want NPCs to ever reference anything 'real' for fear of breaking immersion (not to mention political backlash if the model grabs the wrong thing!). However, if you limit your corpus to the gameworld that's nowhere near comparable to the real dataset, and someone would have to retrain it. There needs to be a way of absolutely limiting it, from what I've seen the prompt can't do that with 100% success. A blacklist would be useful in some applications (people tolerate their search engine screwing up once in a while), but not in games. Still, maybe we can accept limited performance by limiting to the gameworld + some manual texts. It's not NPC dialog is literature-level anyway...
- labelbias 6y ago
  The main thing about GPT-3 is that they wanted to demonstrate one-shot fine-tuning and succeeded at it. So the model can be transformed to output part-of-speech words, dependency grammar trees or named entities in input even if training data is sparse. Similarily, you could fine tune it to produce game lore and then see how it works for that. The model easily switches to different modes of operation and achieves state-of-the-art or close to state-of-the-art performance. It's quite funny how NLP folks tried to solve low level tasks (POS tagging, NER, Named entity relationship extraction, dependency parsing, sentiment classification etc.) to get to higher level tasks (good summarization, machine translation, text generation, question & answering) and now a single model captures all the low level stuff for free and does high level stuff so good that finetuning it to do low level stuff is unnecessary.
  - fredliu 6y ago
    This, the difference between one-shot fine-tuning, vs fine tuning for GPT-2, is one of the major breakthroughs. Since GPT-3 is so hot in the past few days, people seem to forgot or not realize lots of the GPT-3 examples shown off today were possible with GPT-2, with the catch that you had to fine-tune your own GPT-2 model to fit your problem domain (game plots, poems, music, bots that chats like certain characters, etc). GPT-3 makes that fine tuning process unnecessary (although practically you probably can't/can't afford to fine-tune your GPT-3 model)
mcbits 6y ago
Another potential application/adaptation that would be useful is lossy text compression, but I'm not excited about using 300 GB of RAM or a web service to compress and decompress text.
- eru 6y ago
  Especially since text is so small anyway, compared to pictures and videos or sound.
amelius 6y ago
It would be great as a "lorem ipsum" generator ...
phepranto 6y ago
In case you didnt know, aidungeon.io uses GPT-3 in their "Dragon" model. It's only available for pro accounts though. I think this might be the way to go forward regarding language models in games: Offer them through cloud computing for GaaS.
- ricksharp 6y ago
  This is an excellent example of the potential. It feels like what I imagined playing in a Star Trek Holodeck would be (well the dialog anyway).
nutanc 6y ago
"Give humans a new tool and they will amaze you." You are right. Already some special use cases of the API have come up. I have been exploring the API since yesterday and it is a big deal. But not for the reason of AI and AGI. That may happen later. But right now, the API is the game changer. Single API to apply for lots of different tasks. I have been updating my thread of demos which I am building at https://twitter.com/nutanc/status/1285128265519083520 https://twitter.com/nutanc/status/1285128265519083520
- testopenai 6y ago
  How long did it take you to get access to the API? I filled out the form almost a week ago and haven't received anything back.
  - nutanc 6y ago
    Around 2 weeks I think
devalgo 6y ago
>But then I thought of possible applications Indeed, some pretty impressive demos have been sprouting up on Twitter. Things like generating properly running code from english descriptions of the desired functionality are potentially game changing.
- not2b 6y ago
  GPT-3 doesn't know how to code. However, it has so many parameters that it was almost able to memorize its training data, which included people asking how to code something and other people answering.
  - plutonorm 6y ago
    I see people saying this over and over again, with nothing to back it up. I could say the same about the human mind.
    - kordlessagain 6y ago
      And at least those can be run on prem. This thing we are talking about is already in walled gardens.
  - devalgo 6y ago
    Even if it was completely rote memorization it is still extremely valuable to be able to give a plain text prompt and get relevant answers back. As SWEs most of us probably have very high end google-fu so we can find potentially obscure answers to our questions but this could make that kind of skill redundant. Why try to search stackoverflow when GPT could just generate exactly the code snippet you need?

puranjay 6y ago

That would be a great use. It really takes me out of a game when I talk to an NPC and they run out of dialog options. After that, every time you approach them, they repeat the last thing they said.

xamuel 6y ago
I predict it wouldn't be as fun as it might initially sound. NPC dialog is part of level design just like all the rest of the level it occurs on. With few exceptions (such as roguelikes), random level generation tends to produce bland cookie-cutter levels; I don't see why dialog would be any different. The reason it's fun to talk to NPCs in a game like Chrono Trigger or FF7 is because someone put the work in to make their dialog interesting and relevant and fun. Where I could see it working better would be for e.g. newspaper headlines in a grand-strategy or SimCity-like game. When you do something crazy like have Lichtenstein conquer Western Europe, it could be funny to have some auto-generated commentary on said crazy state of affairs.
- devalgo 6y ago
  It offers you procedurally generated dialogue that can be relevant to any potential player actions. You decide to kill the town chicken and now all the townspeople are talking about it. You've freed up devs from having to hard code all these possible interactions, obviously not perfect but it allows for some interesting possibilities.

skoll43 6y ago

Just imagine a caves of qud weirdly coherent storytelling be the base of new games, I LOVE THE FUTURE

deleted 6y ago
[deleted]

yyyk 6y ago

Hm. GPT-3 is trained on internet data from the real world. Your NPCs are in the gameworld. I guess you wouldn't want NPCs to ever reference anything 'real' for fear of breaking immersion (not to mention political backlash if the model grabs the wrong thing!). However, if you limit your corpus to the gameworld that's nowhere near comparable to the real dataset, and someone would have to retrain it. There needs to be a way of absolutely limiting it, from what I've seen the prompt can't do that with 100% success. A blacklist would be useful in some applications (people tolerate their search engine screwing up once in a while), but not in games. Still, maybe we can accept limited performance by limiting to the gameworld + some manual texts. It's not NPC dialog is literature-level anyway...

labelbias 6y ago
The main thing about GPT-3 is that they wanted to demonstrate one-shot fine-tuning and succeeded at it. So the model can be transformed to output part-of-speech words, dependency grammar trees or named entities in input even if training data is sparse. Similarily, you could fine tune it to produce game lore and then see how it works for that. The model easily switches to different modes of operation and achieves state-of-the-art or close to state-of-the-art performance. It's quite funny how NLP folks tried to solve low level tasks (POS tagging, NER, Named entity relationship extraction, dependency parsing, sentiment classification etc.) to get to higher level tasks (good summarization, machine translation, text generation, question & answering) and now a single model captures all the low level stuff for free and does high level stuff so good that finetuning it to do low level stuff is unnecessary.
- fredliu 6y ago
  This, the difference between one-shot fine-tuning, vs fine tuning for GPT-2, is one of the major breakthroughs. Since GPT-3 is so hot in the past few days, people seem to forgot or not realize lots of the GPT-3 examples shown off today were possible with GPT-2, with the catch that you had to fine-tune your own GPT-2 model to fit your problem domain (game plots, poems, music, bots that chats like certain characters, etc). GPT-3 makes that fine tuning process unnecessary (although practically you probably can't/can't afford to fine-tune your GPT-3 model)

mcbits 6y ago

Another potential application/adaptation that would be useful is lossy text compression, but I'm not excited about using 300 GB of RAM or a web service to compress and decompress text.

eru 6y ago
Especially since text is so small anyway, compared to pictures and videos or sound.

amelius 6y ago

It would be great as a "lorem ipsum" generator ...

phepranto 6y ago

In case you didnt know, aidungeon.io uses GPT-3 in their "Dragon" model. It's only available for pro accounts though. I think this might be the way to go forward regarding language models in games: Offer them through cloud computing for GaaS.

ricksharp 6y ago
This is an excellent example of the potential. It feels like what I imagined playing in a Star Trek Holodeck would be (well the dialog anyway).

nutanc 6y ago

"Give humans a new tool and they will amaze you." You are right. Already some special use cases of the API have come up. I have been exploring the API since yesterday and it is a big deal. But not for the reason of AI and AGI. That may happen later. But right now, the API is the game changer. Single API to apply for lots of different tasks. I have been updating my thread of demos which I am building at https://twitter.com/nutanc/status/1285128265519083520 https://twitter.com/nutanc/status/1285128265519083520

testopenai 6y ago
How long did it take you to get access to the API? I filled out the form almost a week ago and haven't received anything back.
- nutanc 6y ago
  Around 2 weeks I think

devalgo 6y ago

>But then I thought of possible applications Indeed, some pretty impressive demos have been sprouting up on Twitter. Things like generating properly running code from english descriptions of the desired functionality are potentially game changing.

not2b 6y ago
GPT-3 doesn't know how to code. However, it has so many parameters that it was almost able to memorize its training data, which included people asking how to code something and other people answering.
- plutonorm 6y ago
  I see people saying this over and over again, with nothing to back it up. I could say the same about the human mind.
  - kordlessagain 6y ago
    And at least those can be run on prem. This thing we are talking about is already in walled gardens.
- devalgo 6y ago
  Even if it was completely rote memorization it is still extremely valuable to be able to give a plain text prompt and get relevant answers back. As SWEs most of us probably have very high end google-fu so we can find potentially obscure answers to our questions but this could make that kind of skill redundant. Why try to search stackoverflow when GPT could just generate exactly the code snippet you need?