2 ms·
I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical
by wolttam 1y ago
I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical household tasks
Even if it requires human evaluators at first, and even if the models completely suck at this task right now: it seems like the kind of task you'd want them to be good at, if you want these models to eventually carry out these tasks in embodied forms in the real world.
Just having the benchmark in the first place is what gives model makers something to optimize for.
- luckydata 1y agoGenerating footage wouldn't help with the opposite but navigating a simulation would which is a pretty standard type of evaluation for multimodal AIs designed to act in the real world.
- wolttam 1y agoDo you mean that it wouldn't help with ingesting footage and then determining how to act? I can imagine a robotics architecture where you have one model generating footage (next frames for what it is currently seeing) and another dumber model which takes in the generated footage and only knows how to generate the motor/servo control outputs needed to control whatever robot platform it is integrated with. I think that kind of architecture decoupling would be nice. It allows the model with all the world and task-specific knowledge to be agnostic from its underlying robot platform.