5 ms·

The generated English (and a number of other languages) doesn't make a lick of sense (certainly not comparable to decent NN language models), and there's HTML a

The generated English (and a number of other languages) doesn't make a lick of sense (certainly not comparable to decent NN language models), and there's HTML and JS visible all over the place. But they certainly look a lot like typical geocities pages. How do you generate HTML and JS with NNs? I wish there were an explanation somewhere. Maybe that's the reason that there's so much invalid HTML.

jxdjsks 10y ago
It looks a lot like the stuff I produced a while back with an rnn. I just fed the rnn raw web pages for a giggle. It does surprisingly well, often managing to close braces correctly. It really goes to show how good the browsers are at dealing with junk html. I was thinking about trying to jointly train the html and an image of the webpage, in the vague hope that you might be able to go from image to webpage. One day. Probably quite soon...
jnpatel 10y ago
Agreed about wishing there were an explanation. It's not clear to me that NN are the best way to get interesting Franken-pages. Maybe a Markov chain instead?
- drdeca 10y ago
  Markov chains can't keep track of nested stuff though I thought? I suppose that NNs have some sort of limit to how many layers they handle, but I thought they tended to learn nesting fairly well, unlike markov chains? I could be wrong about this.
- yolesaber 10y ago
  The best way is to have the NN generate the HTML / CSS / JS code and then use a markov chain to create content to populate the page with. It also has to do with quality of input. The texts on geocities pages aren't exactly pinnacles of clear and legible language. Same with the HTML.
gwern 10y ago
It could just be char-rnn output inside a very simple HTML wrapper. I found that a char-rnn can produce a lot of syntactically valid CSS with lots of URLs, so CSS+some HTML+image URLs is entirely possible: http://www.gwern.net/AB%20testing#training-a-neural-net-to-generate-css http://www.gwern.net/AB%20testing#training-a-neural-net-to-g... The template here looks like it would go "<script src="jquery.js"></script> <script src="sha1.js"></script> <script src="links.js"></script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-4209567-5', 'auto'); ga('send', 'pageview'); </script> <html> ...[until generation limit, ~1600 characters; note the absence of any valid closing tags] The low quality of the text might reflect the demands of simultaneously modeling text+HTML+CSS in a too-small RNN or insufficient training (training a high quality char-rnn/torch-rnn to convergence on the entire Geocities corpus, or even just a few hundred megabytes, could take weeks).
- gwern 10y ago
  Yes, it turns out to be char-rnn/torch-rnn: 'RNN was 512-node, 3-layer' https://twitter.com/aanand/status/728738421758414849 https://twitter.com/aanand/status/728738421758414849 He wasn't training on his own computer so aside from being much too small (for modeling at least 3 languages, text+HTML+CSS, I would start with 1k neurons and go up from there), it probably wasn't trained to convergence either. The 50MB size is maybe a bit on the small size but probably not the root cause of the low text quality.

jxdjsks 10y ago

It looks a lot like the stuff I produced a while back with an rnn. I just fed the rnn raw web pages for a giggle. It does surprisingly well, often managing to close braces correctly. It really goes to show how good the browsers are at dealing with junk html. I was thinking about trying to jointly train the html and an image of the webpage, in the vague hope that you might be able to go from image to webpage. One day. Probably quite soon...

jnpatel 10y ago

Agreed about wishing there were an explanation. It's not clear to me that NN are the best way to get interesting Franken-pages. Maybe a Markov chain instead?

drdeca 10y ago
Markov chains can't keep track of nested stuff though I thought? I suppose that NNs have some sort of limit to how many layers they handle, but I thought they tended to learn nesting fairly well, unlike markov chains? I could be wrong about this.
yolesaber 10y ago
The best way is to have the NN generate the HTML / CSS / JS code and then use a markov chain to create content to populate the page with. It also has to do with quality of input. The texts on geocities pages aren't exactly pinnacles of clear and legible language. Same with the HTML.

gwern 10y ago

It could just be char-rnn output inside a very simple HTML wrapper. I found that a char-rnn can produce a lot of syntactically valid CSS with lots of URLs, so CSS+some HTML+image URLs is entirely possible: http://www.gwern.net/AB%20testing#training-a-neural-net-to-generate-css http://www.gwern.net/AB%20testing#training-a-neural-net-to-g... The template here looks like it would go "<script src="jquery.js"></script> <script src="sha1.js"></script> <script src="links.js"></script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-4209567-5', 'auto'); ga('send', 'pageview'); </script> <html> ...[until generation limit, ~1600 characters; note the absence of any valid closing tags] The low quality of the text might reflect the demands of simultaneously modeling text+HTML+CSS in a too-small RNN or insufficient training (training a high quality char-rnn/torch-rnn to convergence on the entire Geocities corpus, or even just a few hundred megabytes, could take weeks).

gwern 10y ago
Yes, it turns out to be char-rnn/torch-rnn: 'RNN was 512-node, 3-layer' https://twitter.com/aanand/status/728738421758414849 https://twitter.com/aanand/status/728738421758414849 He wasn't training on his own computer so aside from being much too small (for modeling at least 3 languages, text+HTML+CSS, I would start with 1k neurons and go up from there), it probably wasn't trained to convergence either. The 50MB size is maybe a bit on the small size but probably not the root cause of the low text quality.