4 ms·

it could be a nice experiment to have a basic html structure and a picture, and then attempt to generate the style until the rendered html match the image. gen

• ChrisClark 11y ago
  That would be really interesting to see. I bet there would be convoluted CSS hacks we never even dreamed were possible and they won't be practical at all. :)
  • nacs 11y ago
    Or maybe it would just generate a <table> for layout with a bunch of 1px spacer cells/gifs..
    • chc 11y ago
      It's generating CSS, so it can't add (non-pseudo-)elements.
• gwern 11y ago
  Yes, a generative model of HTML+CSS is definitely a direction I'm nodding towards. (I discussed it briefly in the previous section about reinforcement learning in general.) I'm still hazy on the full architecture: you need an RNN to generate the CSS, you probably want to feed in a particular HTML page to target the CSS onto, the users' browsers generate the reward signal, and you can create images of the HTML+CSS combo as rendered in a web browser and bring in a convolutional network somehow... There doesn't seem to be anything quite like this in the literature that I've come across. (Actually, there's a surprising dearth of reinforcement learning in general. Very few blog posts or demos or introductory materials. It makes it hard to understand what is new about DQN or how the whole system works on a concrete coding level.)
  • gburt 11y ago
    The "reward signal" aka objective function seems to be the challenging part here. In the parent post's suggestion, all you'd end up with is a neural network that could /maybe/ reproduce a picture (assuming CSS was capable and the network has the approximately learnable properties necessary). It'd be more interesting to have some "quality" measure that actually meant something to evaluate outputs.
    • stefs 11y ago
      of course you need a fitness function; why not pixel by pixel comparison of the rendered output?
      • biot 11y ago
        If the original image were created in a vector-based program -- in other words, something where the x,y,width,height parameters are known and stored -- you can load the generated HTML+CSS in a known reference browser and enumerate the x,y,width,height of the matching elements in the DOM. If your fitness function is a golf score, then something like: sum( abs(X_expected - X_actual) * abs(Width_expected - Width_actual) + abs(Y_expected - Y_actual) * abs(Height_expected - Height_actual) ) mapped over all elements ought to do the trick. When you hit 0, it's a perfect reproduction.
        • dsfsdfd 11y ago
          yeah, but you need to render in a headless browser. This might take 0.1 seconds per webpage which is extremely slow when trying to use in the context of reinforcement learning. I thought about trying to do MCMC over a beam search through the rnn output, but ran out of time and patience.
      • gburt 11y ago
        My point was that this is boring. A fitness function that evaluated some notion of how "pretty" a page was would be cooler than being able to regenerate a screenshot's CSS (in a likely very complex form).