5 ms·
This work is a natural progression from a lot of other prior work in the literature... but that doesn't make the results any less impressive. The examples show
by cs702 8y ago
This work is a natural progression from a lot of other prior work in the literature... but that doesn't make the results any less impressive. The examples shown are amazingly, unbelievably good! Really GREAT WORK.
Based on a quick skim of the paper, here is my oversimplified description of how this works:
During training, an agent navigates an artificial 3D scene, observing multiple 2D snapshots of the scene, each snapshot from a different vantage point. The agent passes these snapshots to a deep net composed of two main parts: a representation-learning net and a scene-generation net. The representation-learning net takes as input the agent's observations and produces a scene representation (i.e., a lower-dimensional embedding which encodes information about the underlying scene). The scene-generation network then predicts the scene from three inputs: (1) an arbitrary query viewpoint, (2) the scene representation, and (3) stochastic latent variables. The two networks are trained jointly, end-to-end, to maximize the likelihood of generating the ground-truth image that would be observed from the query viewpoint. See Figure 1 on Page 15 of the Open Access version of the paper. Obviously I'm playing loose with language and leaving out numerous important details, but this is essentially how training works, as I understand it based on a first skim.
EDIT: I replaced "somewhat obvious" with "natural," which better conveys what I actually meant to write the first time around.
- fizx 8y ago> This work is a somewhat obvious progression from a lot of other prior work in the literature... but that doesn't make the results any less impressive. Everything takes figuratively forever to train, and the field is moving incredibly fast, so everything you see is both directly adjacent to previous work and also impressive. To me, this represents the best of science--that we can collectively make rapid progress without having to invoke an Einstein figure to make some magical leap!
- cs702 8y agoI replaced "somewhat obvious" with "natural," which better conveys what I actually meant to write the first time around This is a natural progression from prior work. It's a compliment :-)
- Rainymood 8y agoI, literally just 15 minutes ago, had a chat with a friend of mine exactly about how what we are doing right now with computer vision is all based on a flawed premise (supervised 2D training set). The human brain works in 3D space (or 3D+time) and then projects all this knowledge in a 2D image. Here I was, thinking I finally had thought of a nice PhD project and then Deepmind comes along and gets the scoop! Haha.
- SubiculumCode 8y agoThe trick is to then to have one agent try to take on the perspective of another agent, then change its behavior accordingly (depending on task goal). I.e. theory of mind.
- tlarkworthy 8y agoHide and seek
- SubiculumCode 8y agoThere you go. A good task.
- Retra 8y agoWould probably make for more interesting game AI and pathfinding behavior for sure.
- bitL 8y agoMaybe in trivial examples, but bleeding edge has for quite some time adopted 3D convolutions (either 3D space or 2D + time), and combined them with RNNs for more power.
- taejavu 8y agoHere is an excellent talk by Geoffrey Hinton on exactly that topic https://www.youtube.com/watch?v=rTawFwUvnLE https://www.youtube.com/watch?v=rTawFwUvnLE
- thomasahle 8y agoWhat is a stochastic latent variable? Is it just random noise inputted to the network?
- blt 8y agoYes