3 ms·

>Because there's always another type of 'elephant' to be thrown in. Well, you could randomly generate and render 3D objects, and paste them into photos. This w

>Because there's always another type of 'elephant' to be thrown in. Well, you could randomly generate and render 3D objects, and paste them into photos. This will give you a huge space of 'elephants'.

QML 8y ago
That would not seem computationally efficient if for every class (i.e. object) you were to create a 3D modeling and paste it into a training set
YeGoblynQueenne 8y ago
If what you propose was possible to do, if it was possible to overcome the problem described in the above article by using 3d models of real-world objects, then it would also be possible to train on those models in the first place, i.e. we wouldn't need to train machine vision algorithms like CNNs on large datasets of digital stills or photographs- we'd just generate millions of 3d models of objects and scenes of interest, already fully annotated for supervised learning, and train on those. But, as you can probably tell, this is not what happens and instead we have to make do with "real" images, painstakingly collected and annotated with the classes of objets within them "by hand". What you propose is, essentially, training on a simulation. That doesn't work very well because statistical machine learning algorithms generalise very poorly from training data to unseen data and when your training data is a simulation (such as a 3d model of an object) and the unseen data is the real world (such as a video still, or a digitial photograph of that object) the resulting model will be very bad at predicting the unseen data, so identifying objects in images of the real world. It might seem to you (and to me!) that modern 3d simulations (as in high-end CGI and video games) is a very faithful simulation of the real world, but that's primarily because we look at modern 3d through human eyes and perceive it with our (presumably) human brains. We don't know how that works exactly, so we can't reproduce the process in computers, yet, therefore machine vision systems essentially "see" and "perceive" something completely different than what we do. In the end, what looks like a very close approximation to you, is pretty much useless for them- or in any case, they are not able to learn to generalise from that to real objects.
- weregiraffe 8y ago
  >It might seem to you (and to me!) that modern 3d simulations (as in high-end CGI and video games) is a very faithful simulation of the real world, but that's primarily because we look at modern 3d through human eyes and perceive it with our (presumably) human brains. We don't know how that works exactly, so we can't reproduce the process in computers, yet, therefore machine vision systems essentially "see" and "perceive" something completely different than what we do. In the end, what looks like a very close approximation to you, is pretty much useless for them- or in any case, they are not able to learn to generalise from that to real objects. This really is not what I expected, being a layman. Do you have any more data on this topic of training on a simulation?
  - YeGoblynQueenne 8y ago
    "Data", not so much, but you can explore the subject by searching for "reality gap in machine learning". To be honest there are not many references to it that I could find online, but this post by the Google AI blog starts with a very good summary (then proceeds to propose one way to overcome the difficulties of training on simulations, though of course the problem is still far from solved): https://ai.googleblog.com/2017/10/closing-simulation-to-reality-gap-for.html https://ai.googleblog.com/2017/10/closing-simulation-to-real... Simulating many years of robotic interaction is quite feasible with modern parallel computing, physics simulation, and rendering technology. Moreover, the resulting data comes with automatically-generated annotations, which is particularly important for tasks where success is hard to infer automatically. The challenge with simulated training is that even the best available simulators do not perfectly capture reality. Models trained purely on synthetic data fail to generalize to the real world, as there is a discrepancy between simulated and real environments, in terms of both visual and physical properties. In fact, the more we increase the fidelity of our simulations, the more effort we have to expend in order to build them, both in terms of implementing complex physical phenomena and in terms of creating the content (e.g., objects, backgrounds) to populate these simulations. This difficulty is compounded by the fact that powerful optimization methods based on deep learning are exceptionally proficient at exploiting simulator flaws: the more powerful the machine learning algorithm, the more likely it is to discover how to "cheat" the simulator to succeed in ways that are infeasible in the real world. The question then becomes: how can a robot utilize simulation to enable it to perform useful tasks in the real world? The difficulty of transferring simulated experience into the real world is often called the "reality gap." The reality gap is a subtle but important discrepancy between reality and simulation that prevents simulated robotic experience from directly enabling effective real-world performance. Visual perception often constitutes the widest part of the reality gap: while simulated images continue to improve in fidelity, the peculiar and pathological regularities of synthetic pictures, and the wide, unpredictable diversity of real-world images, makes bridging the reality gap particularly difficult when the robot must use vision to perceive the world, as is the case for example in many manipulation tasks. Note the bit about deep learning algorithms being very proficient "cheaters", which I missed in my comment above. Indeed, one way to fail to generalise from a simulation to the real world is to "overfit" to the defects in the simulation! Like the linked blog post, most material you are likely to find online focus on training robots with (deep) Reinforcement Learning, I think because that just happens to be one domain where it is even harder to collect training data than good old supervised learning for image recognition. I can find virtually no source referring to the "reality gap" in the context of purely machine vision research- it's just not the done thing to train vision algorithms on simulated data, for the reasons described above, consequently it's very difficult to find hard data on why it's not done. A good related source (with a bazillion references) is the following blog post, discussing the difficulties of deep RL, which is primarily trained on simulated environments: https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html The post makes only passing referenct to the "reality gap" but it should give a good idea about the ins and outs of training in simulated environments.

QML 8y ago

That would not seem computationally efficient if for every class (i.e. object) you were to create a 3D modeling and paste it into a training set

YeGoblynQueenne 8y ago

If what you propose was possible to do, if it was possible to overcome the problem described in the above article by using 3d models of real-world objects, then it would also be possible to train on those models in the first place, i.e. we wouldn't need to train machine vision algorithms like CNNs on large datasets of digital stills or photographs- we'd just generate millions of 3d models of objects and scenes of interest, already fully annotated for supervised learning, and train on those. But, as you can probably tell, this is not what happens and instead we have to make do with "real" images, painstakingly collected and annotated with the classes of objets within them "by hand". What you propose is, essentially, training on a simulation. That doesn't work very well because statistical machine learning algorithms generalise very poorly from training data to unseen data and when your training data is a simulation (such as a 3d model of an object) and the unseen data is the real world (such as a video still, or a digitial photograph of that object) the resulting model will be very bad at predicting the unseen data, so identifying objects in images of the real world. It might seem to you (and to me!) that modern 3d simulations (as in high-end CGI and video games) is a very faithful simulation of the real world, but that's primarily because we look at modern 3d through human eyes and perceive it with our (presumably) human brains. We don't know how that works exactly, so we can't reproduce the process in computers, yet, therefore machine vision systems essentially "see" and "perceive" something completely different than what we do. In the end, what looks like a very close approximation to you, is pretty much useless for them- or in any case, they are not able to learn to generalise from that to real objects.

weregiraffe 8y ago
>It might seem to you (and to me!) that modern 3d simulations (as in high-end CGI and video games) is a very faithful simulation of the real world, but that's primarily because we look at modern 3d through human eyes and perceive it with our (presumably) human brains. We don't know how that works exactly, so we can't reproduce the process in computers, yet, therefore machine vision systems essentially "see" and "perceive" something completely different than what we do. In the end, what looks like a very close approximation to you, is pretty much useless for them- or in any case, they are not able to learn to generalise from that to real objects. This really is not what I expected, being a layman. Do you have any more data on this topic of training on a simulation?
- YeGoblynQueenne 8y ago
  "Data", not so much, but you can explore the subject by searching for "reality gap in machine learning". To be honest there are not many references to it that I could find online, but this post by the Google AI blog starts with a very good summary (then proceeds to propose one way to overcome the difficulties of training on simulations, though of course the problem is still far from solved): https://ai.googleblog.com/2017/10/closing-simulation-to-reality-gap-for.html https://ai.googleblog.com/2017/10/closing-simulation-to-real... Simulating many years of robotic interaction is quite feasible with modern parallel computing, physics simulation, and rendering technology. Moreover, the resulting data comes with automatically-generated annotations, which is particularly important for tasks where success is hard to infer automatically. The challenge with simulated training is that even the best available simulators do not perfectly capture reality. Models trained purely on synthetic data fail to generalize to the real world, as there is a discrepancy between simulated and real environments, in terms of both visual and physical properties. In fact, the more we increase the fidelity of our simulations, the more effort we have to expend in order to build them, both in terms of implementing complex physical phenomena and in terms of creating the content (e.g., objects, backgrounds) to populate these simulations. This difficulty is compounded by the fact that powerful optimization methods based on deep learning are exceptionally proficient at exploiting simulator flaws: the more powerful the machine learning algorithm, the more likely it is to discover how to "cheat" the simulator to succeed in ways that are infeasible in the real world. The question then becomes: how can a robot utilize simulation to enable it to perform useful tasks in the real world? The difficulty of transferring simulated experience into the real world is often called the "reality gap." The reality gap is a subtle but important discrepancy between reality and simulation that prevents simulated robotic experience from directly enabling effective real-world performance. Visual perception often constitutes the widest part of the reality gap: while simulated images continue to improve in fidelity, the peculiar and pathological regularities of synthetic pictures, and the wide, unpredictable diversity of real-world images, makes bridging the reality gap particularly difficult when the robot must use vision to perceive the world, as is the case for example in many manipulation tasks. Note the bit about deep learning algorithms being very proficient "cheaters", which I missed in my comment above. Indeed, one way to fail to generalise from a simulation to the real world is to "overfit" to the defects in the simulation! Like the linked blog post, most material you are likely to find online focus on training robots with (deep) Reinforcement Learning, I think because that just happens to be one domain where it is even harder to collect training data than good old supervised learning for image recognition. I can find virtually no source referring to the "reality gap" in the context of purely machine vision research- it's just not the done thing to train vision algorithms on simulated data, for the reasons described above, consequently it's very difficult to find hard data on why it's not done. A good related source (with a bazillion references) is the following blog post, discussing the difficulties of deep RL, which is primarily trained on simulated environments: https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html The post makes only passing referenct to the "reality gap" but it should give a good idea about the ins and outs of training in simulated environments.