4 ms·

If any of the authors are on this thread, am wondering if there are any to plans to release source code this? This can potentially make generating environments

If any of the authors are on this thread, am wondering if there are any to plans to release source code this? This can potentially make generating environments for games or VR trivial by just taking photos IRL and then importing them into some game engine to generate a scene

modeless 8y ago
The important part is the dataset, which in this case is generated by DeepMind Lab, which is already open source: https://github.com/deepmind/lab https://github.com/deepmind/lab Reimplementing the rest of the paper shouldn't be tremendously difficult. These techniques tend to be fairly simple at their core. But training it could be expensive. In any case, it is a long, long way from here to "make generating environments for games or VR trivial by just taking photos IRL and then importing them into some game engine". Many years of research remain.
- formalsystem 8y ago
  Hey James, since you seem to know a lot about graphics and ML. Am wondering which specific problems remain open before what I mentioned becomes more feasible product wise?
  - modeless 8y ago
    The domain of images used in this research is extremely limited. These are very low resolution artificially generated images of small scenes with simple lighting, simple textures, simple geometry, and very restricted camera positions and parameters that are known exactly (which is not the case for most natural photos). Each of those restrictions needs to be lifted before this will work on realistic natural scenes, and that will require many orders of magnitude more data. It's not clear that this approach will easily scale up to that amount of dataset variation. It's likely that a much fancier neural net architecture and training scheme will be required, and probably faster hardware too. This is not intended as a criticism of this research, which I think is really great.
- purplebear 8y ago
  Hi, you can find the actual datasets used for the paper here: https://github.com/deepmind/gqn-datasets https://github.com/deepmind/gqn-datasets

modeless 8y ago

The important part is the dataset, which in this case is generated by DeepMind Lab, which is already open source: https://github.com/deepmind/lab https://github.com/deepmind/lab Reimplementing the rest of the paper shouldn't be tremendously difficult. These techniques tend to be fairly simple at their core. But training it could be expensive. In any case, it is a long, long way from here to "make generating environments for games or VR trivial by just taking photos IRL and then importing them into some game engine". Many years of research remain.

formalsystem 8y ago
Hey James, since you seem to know a lot about graphics and ML. Am wondering which specific problems remain open before what I mentioned becomes more feasible product wise?
- modeless 8y ago
  The domain of images used in this research is extremely limited. These are very low resolution artificially generated images of small scenes with simple lighting, simple textures, simple geometry, and very restricted camera positions and parameters that are known exactly (which is not the case for most natural photos). Each of those restrictions needs to be lifted before this will work on realistic natural scenes, and that will require many orders of magnitude more data. It's not clear that this approach will easily scale up to that amount of dataset variation. It's likely that a much fancier neural net architecture and training scheme will be required, and probably faster hardware too. This is not intended as a criticism of this research, which I think is really great.
purplebear 8y ago
Hi, you can find the actual datasets used for the paper here: https://github.com/deepmind/gqn-datasets https://github.com/deepmind/gqn-datasets