5 ms·
It feels like this issue have been (at least in academia) known for a long time. When I try to explain the problem I usually use the slides from ICML 2018 talk
by NalNezumi 3y ago
It feels like this issue have been (at least in academia) known for a long time. When I try to explain the problem I usually use the slides from ICML 2018 talk by Yisong Yue[1]. And there's already (hard to use in practice) methods such as DAgger to correct for that accumulating error.
Out of the more recent advancements, I think ACT[2]s approach seems like the more interesting direction. Similar to MPC that predict(calculate) not only the next action but predict sequences of actions, discard all the actions except for one, at each loop.
Mapping the input to only the current action seems to be prone to error, while mapping the multiple steps is more robust.
My hunch is that there's probably some pretty fundamental limitations to our current methods (Transformers etc) though, which makes applying it to things such as language when error accumulation is essentially neglible (relatively speaking) fine but, to cars and robots maybe not.
[1] https://youtu.be/WjFdD7PDGw0?si=auftqglrNNTxel-Y https://youtu.be/WjFdD7PDGw0?si=auftqglrNNTxel-Y
[2] https://arxiv.org/abs/2304.13705 https://arxiv.org/abs/2304.13705
- polygamous_bat 3y agoThe problem with ACT is very similar to the problem with MPC: if your forward model of the world is not great, it will come back to bite you. So much so that afaik ACT policies can have difficulty transferring between two instances of the same robot.
- NalNezumi 3y agoI don't think ACT (unlike MPC) have an explicit use case for different embodiment. Or what do you mean by two different instances? Two different tasks /arm/robots? Their recent work (Octo policy or something) wasn't even that convincing in that evaluation. But as you're a fellow Robot Learning (BC/LfD I'm assuming) practitioner, in the field of BC (Not IRL) what have been the more robust method for your case?
- polygamous_bat 3y ago> I don't think ACT (unlike MPC) have an explicit use case for different embodiment. Or what do you mean by two different instances? I mean two different Aloha setups (2 x 2x WidowX) in the same lab. This is the very minimal level of "generalization" you may expect, barely enough to be called generalization, but still, the learned policy seems to overfit to the individual robot's joint space quirks. > in the field of BC (Not IRL) what have been the more robust method for your case? Different things for different definitions of robust. What's your definition?
- alsodumb 3y agoYou're absolutely right about ACT. What are your thoughts on diffusion policy line of work? In my personal experience, I found it way more robust than ACT. Have you had a chance to try it?
- polygamous_bat 3y ago> What are your thoughts on diffusion policy line of work? It's been interesting, for sure! More robust than ACT as long as you are not using action chunking (has the same problem as ACT). Downside is, it can be too slow/require too much data to train. I have a sneaking suspicion that at that scale of data even visual nearest neighbor stuff would be similarly robust. Of course, no one has figured out an iota of useful generalization, which is sad across the field.