3 ms·
I find it odd that the article doesn't address the apparent success of training with transformer based models in virtual environments to build models that are t
by dbspin 2y ago
I find it odd that the article doesn't address the apparent success of training with transformer based models in virtual environments to build models that are then mapped onto the real world. This is being used in everything from building datasets for self driving cars, to navigation and task completion for humanoid robots. Nvidia have their omniverse project [1], but there are countless other examples [2][3][4]. Isn't this obviously the way to build the corpus of experience needed to train these kinds of cross modal models?
[1] https://www.nvidia.com/en-us/industries/robotics/#:~:text=NVIDIA%20Omniverse%E2%80%AC%20is%20a,the%20transfer%20of%20learned%20behavior https://www.nvidia.com/en-us/industries/robotics/#:~:text=NV....
[2] https://www.sciencedirect.com/science/article/abs/pii/S009784932100011X https://www.sciencedirect.com/science/article/abs/pii/S00978...
[3] https://techcrunch.com/2024/01/04/google-outlines-new-methods-for-training-robots-with-video-and-large-language-models/ https://techcrunch.com/2024/01/04/google-outlines-new-method...
[4] https://techxplore.com/news/2024-09-google-deepmind-unveils-ai-based.html https://techxplore.com/news/2024-09-google-deepmind-unveils-...
- cybernoodles 2y agoA common practice is to train a transformer model to control a given robot model in simulation by first teleoperating the simulated model with some controller (keyboard, joystick, etc.) to complete the task and create a dataset, and then setting up the simulator to permute the environment variables such as frictions, textures, etc (domain randomization) and run many epochs at faster than real time until a final policy converges. If the right things were randomized and your demonstration examples provided enough variation of information, it should generalize well to the actual hardware.