4 ms·

It would be really interesting/fun to run this on a frame by frame output of classic 8 bit video games and see what it does. I know it wouldn't make the game lo

by j4yav 5y ago

It would be really interesting/fun to run this on a frame by frame output of classic 8 bit video games and see what it does. I know it wouldn't make the game look real within it's own concept, but the ordered/familiar inputs to the AI might generate some interesting video outputs.

toxik 5y ago
You’d need some kind of inter frame consistency also, would probably jiggle quite a lot.
jffry 5y ago
I'm not entirely sure how this would be achiveved. When you draw in the app, you use a brush and have to pick materials from a palette (like "sky" or "ground" or "stone wall" etc). It doesn't seem to have any sort of "import an image" feature because what would that even mean in their model. The approach I would probably consider is to use a modified ROM so that the different sprites in the game are different solid colors. Then I'd write some kind of mouse automation to use those captured images and draw the frame in the app, clicking on the various palette options based on color. The next challenge is that the Canvas app doesn't let you set individual pixels, the smallest brush is ~10px across on its ~550px canvas. Maybe I'd have to settle for picking a Z-ordering and just drawing everything approximately, or maybe you could do some sort of attempt at a path routing algorithm to draw along the edges of the shapes and fill in the centers.
- yeldarb 5y ago
  An example is here (using the same NVIDIA GauGAN model that backs this Canvas app): https://twitter.com/jonathanfly/status/1144735290591981568?lang=en https://twitter.com/jonathanfly/status/1144735290591981568?l... Jonathan has played with GauGAN quite a bit (search twitter for "from:@jonathanfly gaugan" to see more).
  - jffry 5y ago
    Cool! For anybody looking for Mario specifically, here it is: https://twitter.com/jonathanfly/status/1158846045285232641 https://twitter.com/jonathanfly/status/1158846045285232641 The one with the Pole Position racing game looks pretty cool, with a surprising amount of stability between frames: https://twitter.com/jonathanfly/status/1146569133376573440 https://twitter.com/jonathanfly/status/1146569133376573440
  - j4yav 5y ago
    Nice, this is exactly what I imagined.
- nitrogen 5y ago
  Tile-based 2D games should be fairly easily converted to provide semantic labeling. They already have a low-res grid that says "sky, stone, etc."

by j4yav 5y ago

toxik 5y ago

You’d need some kind of inter frame consistency also, would probably jiggle quite a lot.

jffry 5y ago

I'm not entirely sure how this would be achiveved. When you draw in the app, you use a brush and have to pick materials from a palette (like "sky" or "ground" or "stone wall" etc). It doesn't seem to have any sort of "import an image" feature because what would that even mean in their model. The approach I would probably consider is to use a modified ROM so that the different sprites in the game are different solid colors. Then I'd write some kind of mouse automation to use those captured images and draw the frame in the app, clicking on the various palette options based on color. The next challenge is that the Canvas app doesn't let you set individual pixels, the smallest brush is ~10px across on its ~550px canvas. Maybe I'd have to settle for picking a Z-ordering and just drawing everything approximately, or maybe you could do some sort of attempt at a path routing algorithm to draw along the edges of the shapes and fill in the centers.

yeldarb 5y ago
An example is here (using the same NVIDIA GauGAN model that backs this Canvas app): https://twitter.com/jonathanfly/status/1144735290591981568?lang=en https://twitter.com/jonathanfly/status/1144735290591981568?l... Jonathan has played with GauGAN quite a bit (search twitter for "from:@jonathanfly gaugan" to see more).
- jffry 5y ago
  Cool! For anybody looking for Mario specifically, here it is: https://twitter.com/jonathanfly/status/1158846045285232641 https://twitter.com/jonathanfly/status/1158846045285232641 The one with the Pole Position racing game looks pretty cool, with a surprising amount of stability between frames: https://twitter.com/jonathanfly/status/1146569133376573440 https://twitter.com/jonathanfly/status/1146569133376573440
- j4yav 5y ago
  Nice, this is exactly what I imagined.
nitrogen 5y ago
Tile-based 2D games should be fairly easily converted to provide semantic labeling. They already have a low-res grid that says "sky, stone, etc."