4 ms·

See also my experiments with GPT-3 on sane prompts, which have wildly varying quality even after generating them in bulk: https://github.com/minimaxir/gpt-3-exp

See also my experiments with GPT-3 on sane prompts, which have wildly varying quality even after generating them in bulk: https://github.com/minimaxir/gpt-3-experiments https://github.com/minimaxir/gpt-3-experiments Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees). I'm more bullish on small bespoke models for a given use case, which is what I spend my time researching. (example: a ~1M parameter GPT-2 Magic: the Gathering card generator: https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZmU5vwhi4X-6E_a?usp=sharing https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZm... )

YeGoblynQueenne 6y ago
Generating M:tG cards is interesting to me. Can we talk about this a bit? :) My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3]. I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project. A couple of examples (you probably know already): −2: Put a +1/+1 counter on each other creature you control. "Each other creature" should refer to a card with the (super) type "creature" but this card is a "planeswaker". +1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles. Which creature was that creature a copy of? −10: Target opponent sacrifices a creature. −9: Each opponent sacrifices a creature. Representing costs in a meaningful manner is a constant problem in every M:tG generator I've seen. The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet. My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above). Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar. __________________ [1] https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over. [2] https://github.com/stassa/THELEMA https://github.com/stassa/THELEMA - should work with older versions of Swi-Prolog, unfortunately not documented in the README. [3] https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w - see Section 3.3 "Experiment 3: M:tG fragment". Cringe-inducing to read just a few months after writing, sorry. [4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_arenas_gre_the_groundwork_for_future/do0rvfe/ https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren... [5] https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadispersal_bugged/e73pr9u/ https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadisper...
- minimaxir 6y ago
  The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models. As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
  - YeGoblynQueenne 6y ago
    >> The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models. Are you sure? I reloaded the page once, before commenting above, but the output didn't change. I keep reloading now and the same set of cards is displayed. I reloaded with Shift-F5 to be sure ish. >> As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win. I don't know what an "interesting" card is, when we're talking about automatically generated cards. If a card looks too much like an existing card (e.g. if it has abilities that you can find on cards in a real-world set) then it's not very interesting. If it's an ability that hasn't been seen before but is only a variant of an existing ability ("Destroy target donkey") it's still not very interesting. To generate really "interesting" cards a generator must go beyond what's in the M:tG corpus, but not so far out that the abilities don't make sense anymore because that's not "interesting", just "random". The ones generated by your project are not far out enough to be what I'd call "interesting" but I think if you tried to make them more interesting they'd also become less coherent than they are currently (looking at the same few results I keep getting anyway). (This is not very harsh criticism I hope.)
    - minimaxir 6y ago
      You have to run the notebook itself by following the instructions at the top, then you can customize the output (lower temperatures will follow the rules better). Here's some examples of curated cards generated that probably fit your description of "interesting": https://www.reddit.com/r/magicTCG/comments/n396g8/i_trained_an_ai_on_all_magic_cards_and_now_you/gwpp8tf/?context=3 https://www.reddit.com/r/magicTCG/comments/n396g8/i_trained_...

YeGoblynQueenne 6y ago

Generating M:tG cards is interesting to me. Can we talk about this a bit? :) My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3]. I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project. A couple of examples (you probably know already): −2: Put a +1/+1 counter on each other creature you control. "Each other creature" should refer to a card with the (super) type "creature" but this card is a "planeswaker". +1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles. Which creature was that creature a copy of? −10: Target opponent sacrifices a creature. −9: Each opponent sacrifices a creature. Representing costs in a meaningful manner is a constant problem in every M:tG generator I've seen. The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet. My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above). Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar. __________________ [1] https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over. [2] https://github.com/stassa/THELEMA https://github.com/stassa/THELEMA - should work with older versions of Swi-Prolog, unfortunately not documented in the README. [3] https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w - see Section 3.3 "Experiment 3: M:tG fragment". Cringe-inducing to read just a few months after writing, sorry. [4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_arenas_gre_the_groundwork_for_future/do0rvfe/ https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren... [5] https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadispersal_bugged/e73pr9u/ https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadisper...

minimaxir 6y ago
The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models. As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
- YeGoblynQueenne 6y ago
  >> The outputs on pageload are just a random uncurated set I ran. I never claimed smaller models like the MtG models were immune to the same curation issues as larger text generation models. Are you sure? I reloaded the page once, before commenting above, but the output didn't change. I keep reloading now and the same set of cards is displayed. I reloaded with Shift-F5 to be sure ish. >> As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win. I don't know what an "interesting" card is, when we're talking about automatically generated cards. If a card looks too much like an existing card (e.g. if it has abilities that you can find on cards in a real-world set) then it's not very interesting. If it's an ability that hasn't been seen before but is only a variant of an existing ability ("Destroy target donkey") it's still not very interesting. To generate really "interesting" cards a generator must go beyond what's in the M:tG corpus, but not so far out that the abilities don't make sense anymore because that's not "interesting", just "random". The ones generated by your project are not far out enough to be what I'd call "interesting" but I think if you tried to make them more interesting they'd also become less coherent than they are currently (looking at the same few results I keep getting anyway). (This is not very harsh criticism I hope.)
  - minimaxir 6y ago
    You have to run the notebook itself by following the instructions at the top, then you can customize the output (lower temperatures will follow the rules better). Here's some examples of curated cards generated that probably fit your description of "interesting": https://www.reddit.com/r/magicTCG/comments/n396g8/i_trained_an_ai_on_all_magic_cards_and_now_you/gwpp8tf/?context=3 https://www.reddit.com/r/magicTCG/comments/n396g8/i_trained_...