3 ms·

The biggest part usually missing is spatial information. I think that the text-to-video stuff is going in the direction of having a real understanding of the 3d

by ilaksh 4y ago

The biggest part usually missing is spatial information. I think that the text-to-video stuff is going in the direction of having a real understanding of the 3d world (or other non-textual structures) and how it relates to textual descriptions. I think that with things like transformers that probably means real language grounding. So it might be something like using the prompt to generate an embedding that captures spatial-temporal aspects of a prompt. Or really somehow the whole prompt encoding should terminate at some type of spatial-temporal tokens maybe. I don't know exactly but I think that text-to-video and video-to-text is going to lead in a more general purpose direction than just language alone.

darepublic 4y ago
I was thinking about this before and I thought of having a 3d physics engine that the AI could create objects in and simulate things to see their physical viability. Could also help with question answering that requires that spatial knowledge / real world simulation.
- zone411 4y ago
  This has been a thing for a while. For example, here are a couple random papers from 2017: https://openaccess.thecvf.com/content_cvpr_2017/html/Varol_Learning_From_Synthetic_CVPR_2017_paper.html https://openaccess.thecvf.com/content_cvpr_2017/html/Varol_L..., https://openaccess.thecvf.com/content_ICCV_2017_workshops/w23/html/Abbasnejad_Using_Synthetic_Data_ICCV_2017_paper.html https://openaccess.thecvf.com/content_ICCV_2017_workshops/w2... or a newer one about deformable objects: https://arxiv.org/abs/2107.08898 https://arxiv.org/abs/2107.08898. You can even use a robot to manipulate things in real life to create synthetic data for a neural net.
- ilaksh 4y ago
  Yes I think that the latest in ML everything else will help to create those traditional simulations. But also I think that what an AI that can for example really answer questions about a video would need to do to be really effective would be to basically do compressed versions of those simulations using the spatial-temporal-abstract latent space. Which should be a better model than just the textual space.

by ilaksh 4y ago

darepublic 4y ago

I was thinking about this before and I thought of having a 3d physics engine that the AI could create objects in and simulate things to see their physical viability. Could also help with question answering that requires that spatial knowledge / real world simulation.

zone411 4y ago
This has been a thing for a while. For example, here are a couple random papers from 2017: https://openaccess.thecvf.com/content_cvpr_2017/html/Varol_Learning_From_Synthetic_CVPR_2017_paper.html https://openaccess.thecvf.com/content_cvpr_2017/html/Varol_L..., https://openaccess.thecvf.com/content_ICCV_2017_workshops/w23/html/Abbasnejad_Using_Synthetic_Data_ICCV_2017_paper.html https://openaccess.thecvf.com/content_ICCV_2017_workshops/w2... or a newer one about deformable objects: https://arxiv.org/abs/2107.08898 https://arxiv.org/abs/2107.08898. You can even use a robot to manipulate things in real life to create synthetic data for a neural net.
ilaksh 4y ago
Yes I think that the latest in ML everything else will help to create those traditional simulations. But also I think that what an AI that can for example really answer questions about a video would need to do to be really effective would be to basically do compressed versions of those simulations using the spatial-temporal-abstract latent space. Which should be a better model than just the textual space.