That is too indirect a problem. Neural networks will find answers to problems where there is a relatively direct hierarchical connection between input and outpu

That is too indirect a problem. Neural networks will find answers to problems where there is a relatively direct hierarchical connection between input and output. Symbolic information would require different and very big architectures. Also doing that is still state-of-the-art and you don't try to solve 2 state-of-the-art problems at the same time. Firstly, using the better rendering approach has been tried and failed. These networks exploit things that perfect rendering has but reality doesn't. For instance, in a rendered simulated environment it is perfectly reasonable to estimate exactly where an edge is and then follow that edge with zero margin with the robot. In reality you need a minimum margin if you expect it to work (or your object will be damaged), and a decent one. Photorealistic rendering tries to emulate optics that are incredibly good, with no distortion, and the robots have cheap webcams with plastic lenses, with both consistent distortion near the edges and random distortions everywhere because they're webcams. Not supercheap ones, but this isn't a RED camera either. You could fix those lenses, but why would you ? It would be more valuable if it works with cheap lenses, and human and animal eyes don't really try to be distortion free. Rather they have the "software" behind it compensate. Second, photorealistic rendering requires modeling accuracy, and material modeling on a scale we just can't do for real datasets. These datasets have millions of objects, and to make the robot more tolerant of the general weirdness of the real world, they are randomly perturbed, to turn millions of examples to billions. But those perturbed objects are not photorealistic anymore. Nobody has a toothbrush with a triangle in the bottom sticking out to one side. Besides, keep in mind that the resolutions used here are quite small. It's not like they're trying to run a renderer at 4k resolution. Think 128x128, or 256x256. Not exactly a big issue.