This isn't remotely near the state of the art for raw image generation; that would be something like ProGAN or pixel-CNN, which don't involve any reinforcement

This isn't remotely near the state of the art for raw image generation; that would be something like ProGAN or pixel-CNN, which don't involve any reinforcement learning or paintbrushes, and already do photorealistic synthesis. Horse, barn. The point of OP is that you're learning to generate little images in a much more difficult setup: you have to control some complex blackbox system (like a paintbrush robot) to try to generate an image with only crude success/failure feedback at the end of the sequence of actions. The hope is that by going through this intermediate environment, instead of generating an entire image in a single shot via convolutions, it'll be learning more abstract structure about what makes up a face etc, so hypothetically it could do things like rotate faces in 3D (whereas something like ProGAN only sees faces as 2D blobs so it can do things like add/subtract sunglasses or change hair color, but 3D transformations are beyond it). And, with this more abstract deeper understanding, it should be able to speed up learning in settings like robotics etc (instead of paintings, imagine they are videos of humans controlling pick-and-place robot arms); you can see this as one way of approaching unsupervised learning and providing primitives which a higher-level agent can learn faster from (somewhat like the GAIL architecture uses GANs for semi-supervised learning).