7 ms·

Show HN: RNN that generates Game of Thrones text

143 points by zthoutt 9y ago

yablak 9y ago
If you're using TF 1.2.1 or above, you want to replace this line: cell = tf.contrib.rnn.MultiRNNCell([drop_cell] * num_layers) with: cell = tf.contrib.rnn.MultiRNNCell([create_cell() for _ in range(num_layers)]) otherwise you'll be using the same weights for each LSTM layer and your model is less expressive.
- zthoutt 9y ago
  Yep, I was using TF 1.0.0. I will probably upgrade in the future and will keep this in mind.
notamy 9y ago
This is amazing. I wonder if it'd be possible to actually get a coherent plot out of it while retaining the humorous aspects of it.
- zthoutt 9y ago
  Ya, I'm going to continue to tune it and try to get a more coherent plot. I think that I will have to (at a minimum) make the sequence length really long, which will make training time even slower.
SwellJoe 9y ago
The RNN seems to agree with the most popular fan theory about what will happen with Jaime and Cersei. So, it's not entirely crazy. Nonetheless, I don't think GRR Martin has to worry about competition from RNN Martin anytime soon, but this is still quite fun, and surprisingly almost cogent at times.
rlglwx 9y ago
That data folder seems ripe for a DMCA takedown request.
jaz46 9y ago
This is hilarious! We did a similar project last year using the scripts from the show. http://docs.pachyderm.io/en/latest/examples/tensor_flow/readme.html http://docs.pachyderm.io/en/latest/examples/tensor_flow/read... Ours was not nearly as comprehensible, but still funny at points. Some example output: {{ Stannis runs away and Melisandre walks through the lift door , no smiles) . }} [ TYRION ]: I wanted him to be a spoiled Greyjoy again , that 's your niece . And you 're a terrible man . My little Give me A_VOICE . . {{ The HIGH SPARROW sits down on the bucket , DORAN sentences to have waiting to the shoulder , wearing this building . She wargs off the woods and approaches The sound of the riders . BLACK WALDER reaches over his horse beside MARGAERY . }} . [ EDMURE ]: We 're among The Khaleesi . . [ JORAH ]: For far soon . . [ SANSA_STARK ]: He 's travelling a true vow to command until his father 's death. [ DAVOS ]: I don 't mind , Your Grace . I got to build your place and see rolled women in all of their fanatics And so I getting another conversation , watcher ships in the matters of the dead as gods of Westeros .
- zthoutt 9y ago
  Haha, that's awesome! I was thinking about trying to combine the scripts and book text so that I can train on more data.
minimaxir 9y ago
It should be noted that per the included Jupyter Notebook, the resulting network took 24 hours to train, and the resulting network (of a 3-layer 512 cell LSTM) would be hundreds of MB. (too much to share on GitHub anyways). The approach in the repo also uses a word-based approach instead of a character-based approach. Although a word-approach avoids typoes/nonsense, it makes prediction much more difficult since the selection space is harder. (from notebook, error is high as well). There are lots of trade offs in this field. Training these networks is not for the faint of heart and one of the reasons I developed a tool to train text generators using a pretrained RNN (https://github.com/minimaxir/textgenrnn https://github.com/minimaxir/textgenrnn) on a much more simple network architecture that trains quickly, although it probably would not hit the depth of all of Game of Thrones.
- Semiapies 9y ago
  It does seem like a lot of work to get output that looks like something from a simple Markov chain.
  - visarga 9y ago
    Phrase generation doesn't work well without any constraints. For example, in translation a similar system can generate decent text.
dfee 9y ago
I would like to see a TV comedy - perhaps in the vain of Who’s Line Is It Anyways - where actors act out these NN scripts.
- thegeomaster 9y ago
  I guess this is close: https://www.youtube.com/watch?v=LY7x2Ihqjmc https://www.youtube.com/watch?v=LY7x2Ihqjmc. It's a script written by an RNN acted out as a short sci-fi film, and Thomas Middleditch is in it. Don't expect too much, though. I was reading an article on it and was excited to watch it, but it's too incoherent, in the usual fashion of today's RNNs.
  - xirdstl 9y ago
    I wonder what an RNN could generate for an episode of Twin Peaks.
yakult 9y ago
So, what are the odds that we'll have an AI that writes at GoT quality before George finishes the last book?
- gpm 9y ago
  Lower than the odds that George never finishes his last book.
grambrose 9y ago
God I'm so sick of this shit. So what you can train an RNN on text. The results are awful. You aren't special
wbertelsen 9y ago
Nice! I did a similar thing with a character-based rnn a few years ago, but its output was less coherent. I liked when it started spitting out character descriptions though: "NOTHAN, a boy of nine, cupped and dismal," https://twitter.com/asoiaf_ebooks https://twitter.com/asoiaf_ebooks
itschekkers 9y ago
I wonder if there are any forums/fan fiction websites where you could find big chunks of prose to add to the training data. Might help bring the loss down a bit if you mix some in.
- gpm 9y ago
  fanfiction.net undoubtedly has a ton, but I wouldn't rely on the quality or style of writing being similar. Maybe not yet a concern for this project.
DoctorNick 9y ago
You need to take down the data folder now and rebuild the repository. You're going to get hit with a DMCA notice. It's not a matter of IF, it's a matter of WHEN. Publishers of popular book series don't take these matters lightly.
- marssaxman 9y ago
  Why do the copyright mafia's work for them? Let them pay their lawyers first, THEN take the data down. Money is the only language these people understand, so let's force them to spend as much of it as we can.
  - kbenson 9y ago
    > Why do the copyright mafia's work for them? Because it's the right thing to do in this case? If you define what you want to defend purely based on what a law you don't agree with can be used to do, as opposed to the merits of the actual instance, you're bound to pick a bunch of really bad battles.
  - EpicEng 9y ago
    This guy is distributing entire books. You think that's ok?
    - marssaxman 9y ago
      Sure. What harm is it causing?
      - EpicEng 9y ago
        Really? You don't think theft is an appreciable and measurable societal harm? You honestly have no problem whatsoever with outright ripping off these works? I'd love to hear your defense of that position. It's not like we're talking about situations like cable where I have to pay $100 / month to watch one or maybe two shows I enjoy and have no other way to get access at a reasonable price (and even that is debatable). You can pick up a copy of GRRM's work for < $10 without hassle and contribute back to someone whose work gives you joy and the publishers who got that work into your hands.
ryanlol 9y ago
Game of Thrones is the TV series, A Song of Ice and Fire is the book series.
mrhigat4 9y ago
You really should not upload other people's complete works while under copyright. Edit: Nice License > Copyright (c) 2017 Zack Thoutt
edco 9y ago
woah spoilers!
xiphias 9y ago
I'd like to see a trailer video generated for the next series
adtac 9y ago
Hah, I built something similar a while ago (with a pretty front-end): https://asoiaf.now.sh/ https://asoiaf.now.sh/
JustFinishedBSG 9y ago
You definitely need to add an attention mechanism if you want better-than-markov-chain results. Found a project ;)
donquichotte 9y ago
With Markov Chains: "Raved on fire, he clambered up her pallet in a swatch of blood and crept toward him shudder. She was too well Ghost back behind their crews could not lack for the warg could stand guard. The singers was a fire with our beds they could have won that make yo, tell Joffrey awarded to make repairs and threatening to my sons had entered the flesh. A life and a dim his promises of Eddark Stark, he announced the foundation and set sail cracking their wounded and bucket swayed back room to go go. Har Tormund Well the Bastard s home When the Others followed him a fine rule the better look stupid. She could see the same only empty sockets weeping child he ran with that morning. His nose Wife pleased to the man with winter s notion. My brother will never lifted a scare me wise where."
zthoutt 9y ago
A message to all of the vigilantes on HN: I'm sorry that the GOT text ended up in the repo. I actually worked on this project about a month ago and after showing it to a few friends, they convinced me to share it. I didn't think about what I had committed to the repository when I made it public and shared it. This is the first project I have ever shared on HN, so I didn't expect so many people to see it. I actually tried to remove the text within the first few hours of posting (I was not even home or with my computer), but I removed the text in a separate branch and when I merged it into master it created duplicates of every commit instead of removing the text from the git history. Someone put up an issue on Github later that night and I fixed my mistake early the next morning. I own all of the ASOIAF books both in paperback and ebook and pay for an HBO subscription for the sole purpose of watching GOT for 6-10 weeks per year.

143 points by zthoutt 9y ago

If you're using TF 1.2.1 or above, you want to replace this line: cell = tf.contrib.rnn.MultiRNNCell([drop_cell] * num_layers) with: cell = tf.contrib.rnn.MultiRNNCell([create_cell() for _ in range(num_layers)]) otherwise you'll be using the same weights for each LSTM layer and your model is less expressive.

zthoutt 9y ago
Yep, I was using TF 1.0.0. I will probably upgrade in the future and will keep this in mind.

notamy 9y ago

This is amazing. I wonder if it'd be possible to actually get a coherent plot out of it while retaining the humorous aspects of it.

zthoutt 9y ago
Ya, I'm going to continue to tune it and try to get a more coherent plot. I think that I will have to (at a minimum) make the sequence length really long, which will make training time even slower.

SwellJoe 9y ago

The RNN seems to agree with the most popular fan theory about what will happen with Jaime and Cersei. So, it's not entirely crazy. Nonetheless, I don't think GRR Martin has to worry about competition from RNN Martin anytime soon, but this is still quite fun, and surprisingly almost cogent at times.

rlglwx 9y ago

That data folder seems ripe for a DMCA takedown request.

jaz46 9y ago

This is hilarious! We did a similar project last year using the scripts from the show. http://docs.pachyderm.io/en/latest/examples/tensor_flow/readme.html http://docs.pachyderm.io/en/latest/examples/tensor_flow/read... Ours was not nearly as comprehensible, but still funny at points. Some example output: {{ Stannis runs away and Melisandre walks through the lift door , no smiles) . }} [ TYRION ]: I wanted him to be a spoiled Greyjoy again , that 's your niece . And you 're a terrible man . My little Give me A_VOICE . . {{ The HIGH SPARROW sits down on the bucket , DORAN sentences to have waiting to the shoulder , wearing this building . She wargs off the woods and approaches The sound of the riders . BLACK WALDER reaches over his horse beside MARGAERY . }} . [ EDMURE ]: We 're among The Khaleesi . . [ JORAH ]: For far soon . . [ SANSA_STARK ]: He 's travelling a true vow to command until his father 's death. [ DAVOS ]: I don 't mind , Your Grace . I got to build your place and see rolled women in all of their fanatics And so I getting another conversation , watcher ships in the matters of the dead as gods of Westeros .

zthoutt 9y ago
Haha, that's awesome! I was thinking about trying to combine the scripts and book text so that I can train on more data.

minimaxir 9y ago

It should be noted that per the included Jupyter Notebook, the resulting network took 24 hours to train, and the resulting network (of a 3-layer 512 cell LSTM) would be hundreds of MB. (too much to share on GitHub anyways). The approach in the repo also uses a word-based approach instead of a character-based approach. Although a word-approach avoids typoes/nonsense, it makes prediction much more difficult since the selection space is harder. (from notebook, error is high as well). There are lots of trade offs in this field. Training these networks is not for the faint of heart and one of the reasons I developed a tool to train text generators using a pretrained RNN (https://github.com/minimaxir/textgenrnn https://github.com/minimaxir/textgenrnn) on a much more simple network architecture that trains quickly, although it probably would not hit the depth of all of Game of Thrones.

Semiapies 9y ago
It does seem like a lot of work to get output that looks like something from a simple Markov chain.
- visarga 9y ago
  Phrase generation doesn't work well without any constraints. For example, in translation a similar system can generate decent text.

dfee 9y ago

I would like to see a TV comedy - perhaps in the vain of Who’s Line Is It Anyways - where actors act out these NN scripts.

thegeomaster 9y ago
I guess this is close: https://www.youtube.com/watch?v=LY7x2Ihqjmc https://www.youtube.com/watch?v=LY7x2Ihqjmc. It's a script written by an RNN acted out as a short sci-fi film, and Thomas Middleditch is in it. Don't expect too much, though. I was reading an article on it and was excited to watch it, but it's too incoherent, in the usual fashion of today's RNNs.
- xirdstl 9y ago
  I wonder what an RNN could generate for an episode of Twin Peaks.

yakult 9y ago

So, what are the odds that we'll have an AI that writes at GoT quality before George finishes the last book?

gpm 9y ago
Lower than the odds that George never finishes his last book.

grambrose 9y ago

God I'm so sick of this shit. So what you can train an RNN on text. The results are awful. You aren't special

wbertelsen 9y ago

Nice! I did a similar thing with a character-based rnn a few years ago, but its output was less coherent. I liked when it started spitting out character descriptions though: "NOTHAN, a boy of nine, cupped and dismal," https://twitter.com/asoiaf_ebooks https://twitter.com/asoiaf_ebooks

itschekkers 9y ago

I wonder if there are any forums/fan fiction websites where you could find big chunks of prose to add to the training data. Might help bring the loss down a bit if you mix some in.

gpm 9y ago
fanfiction.net undoubtedly has a ton, but I wouldn't rely on the quality or style of writing being similar. Maybe not yet a concern for this project.

DoctorNick 9y ago

You need to take down the data folder now and rebuild the repository. You're going to get hit with a DMCA notice. It's not a matter of IF, it's a matter of WHEN. Publishers of popular book series don't take these matters lightly.

marssaxman 9y ago
Why do the copyright mafia's work for them? Let them pay their lawyers first, THEN take the data down. Money is the only language these people understand, so let's force them to spend as much of it as we can.
- kbenson 9y ago
  > Why do the copyright mafia's work for them? Because it's the right thing to do in this case? If you define what you want to defend purely based on what a law you don't agree with can be used to do, as opposed to the merits of the actual instance, you're bound to pick a bunch of really bad battles.
- EpicEng 9y ago
  This guy is distributing entire books. You think that's ok?
  - marssaxman 9y ago
    Sure. What harm is it causing?
    - EpicEng 9y ago
      Really? You don't think theft is an appreciable and measurable societal harm? You honestly have no problem whatsoever with outright ripping off these works? I'd love to hear your defense of that position. It's not like we're talking about situations like cable where I have to pay $100 / month to watch one or maybe two shows I enjoy and have no other way to get access at a reasonable price (and even that is debatable). You can pick up a copy of GRRM's work for < $10 without hassle and contribute back to someone whose work gives you joy and the publishers who got that work into your hands.

ryanlol 9y ago

Game of Thrones is the TV series, A Song of Ice and Fire is the book series.

mrhigat4 9y ago

edco 9y ago

woah spoilers!

xiphias 9y ago

I'd like to see a trailer video generated for the next series

adtac 9y ago

Hah, I built something similar a while ago (with a pretty front-end): https://asoiaf.now.sh/ https://asoiaf.now.sh/

JustFinishedBSG 9y ago

You definitely need to add an attention mechanism if you want better-than-markov-chain results. Found a project ;)

donquichotte 9y ago

With Markov Chains: "Raved on fire, he clambered up her pallet in a swatch of blood and crept toward him shudder. She was too well Ghost back behind their crews could not lack for the warg could stand guard. The singers was a fire with our beds they could have won that make yo, tell Joffrey awarded to make repairs and threatening to my sons had entered the flesh. A life and a dim his promises of Eddark Stark, he announced the foundation and set sail cracking their wounded and bucket swayed back room to go go. Har Tormund Well the Bastard s home When the Others followed him a fine rule the better look stupid. She could see the same only empty sockets weeping child he ran with that morning. His nose Wife pleased to the man with winter s notion. My brother will never lifted a scare me wise where."

zthoutt 9y ago

A message to all of the vigilantes on HN: I'm sorry that the GOT text ended up in the repo. I actually worked on this project about a month ago and after showing it to a few friends, they convinced me to share it. I didn't think about what I had committed to the repository when I made it public and shared it. This is the first project I have ever shared on HN, so I didn't expect so many people to see it. I actually tried to remove the text within the first few hours of posting (I was not even home or with my computer), but I removed the text in a separate branch and when I merged it into master it created duplicates of every commit instead of removing the text from the git history. Someone put up an issue on Github later that night and I fixed my mistake early the next morning. I own all of the ASOIAF books both in paperback and ebook and pay for an HBO subscription for the sole purpose of watching GOT for 6-10 weeks per year.