Also as a starving artist who isn't starving because I have a non-artist job, I agree wholeheartedly. I've also done some research into this and capsule networ

Also as a starving artist who isn't starving because I have a non-artist job, I agree wholeheartedly. I've also done some research into this and capsule networks are the closest SOTA we have to solving this problem. There isn't any 2D projections of 3D space work with these networks AFAIK, but the purpose and intuition points in this direction. Also, quality data is always the hard part here, so we might be able to bootstrap some of this structure learning with 3D models created for games, projected as 2D, and use that as transfer learning somehow.