6 ms·

Egocentric means that the sensors frame of reference is the same as the ego (self). The person walking around is wearing the sensors/glasses themselves & their

Egocentric means that the sensors frame of reference is the same as the ego (self). The person walking around is wearing the sensors/glasses themselves & their pose is given as “ego pose”. This dataset is clearly targeted for research on AR/XR/VR applications.

AndrewKemendo 3y ago
+1 I’ll “yes and” here…beyond AR/VR a more powerful use case is multi-modal learning (with RL) which is what Meta is probably the leader in IMO. Example paper here: “ Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning” https://arxiv.org/abs/2301.10931 https://arxiv.org/abs/2301.10931 This IMO is the pathway to AGI, as it combines all sense-plan-do data into a time coordinated stream and mimics how humans transfer learning to children via demonstration recording and behavior authoring. If we can create robotics with locomotion and dexterous manipulation, egocentric exploration, and a behavior authoring loop that uses human behavior demonstration and trajectory reinforcement - well, we’ll have the AI we’ve been all talking about. Probably the most exciting area of research that most people don’t know or care about. That’s why head mounted all day ego centric AR is so important - it gives eyes ears and sense perception to our learning systems with human directed egocentric behaviors, guiding the whole thing. Just like pushing your kid down the street in the stroller.
- socratic1 3y ago
  +1 Applications of embodied AI very interesting. Additionally a lot of hard problems are increasingly being solved in simulation like this. See Wayve's GAIA world model
- dleeftink 3y ago
  Just to make sure I understand your excitement: we need guinea-pigs ahem people to wear 'head mounted all day ego centric AR' with who knows how many integrated sensors for long stretches on end, so we can finally get to our fabled A.G.i? That is some B.F. Skinner level future we're aiming for--only this time around, humans become the fully surveilled 'teaching machine'.
  - AndrewKemendo 3y ago
    Well no...not guinea pigs. But correct conceptually - if it's opt-in only and perfectly transparent to everyone what is happening, which in this specific case of Aria it absolutely is. If we want to make machines with equivalent or better capacity as humans we have to transfer the process for scientific discovery, including the sum of our cognitive capacity and knowledge to them. If you quantize human adult-infant interactions, then it boils down to Human adults introducing learning trajectories, labeling input data and biasing weights with reinforcing behaviors for new reinforcement agents. If we can re-build the infrastructure to do precisely that, where the agent is in the place of the infant and society is in the place of the "Human Adult" then we will have re-built at scale the process for human development. The best way we know how to do this today is implementing transfer learning approaches from the basic human developmental research. I started down this road back in 2010 trying to follow the work of Frank Guerin out of the University of Aberdeen [1] [2]. [1]https://www.surrey.ac.uk/people/frank-guerin https://www.surrey.ac.uk/people/frank-guerin [2] https://scholar.google.co.uk/citations?view_op=view_citation&hl=en&user=OK8EYAoAAAAJ&cstart=20&pagesize=80&sortby=pubdate&citation_for_view=OK8EYAoAAAAJ:qUcmZB5y_30C https://scholar.google.co.uk/citations?view_op=view_citation...
    - dleeftink 3y ago
      But what about observer effects? People act differently when recorded, and rarely do we catch humans acting natural when knowingly observed (some of the early 24h/day Twitch streamers come to mind). And what happens once trials are done? How would people feel about their actions becoming part of a technology potentially able to replace them? Even when this barrier can be overcome (i.e. people become accustomed to wearing these devices), I worry about the opt-in nature of it. We've yet to see a disruptive technology adhering to this principle through-and-through, and if current learning efforts are anything to go by, training data is not something companies want to willingly let go or lose out on. Taken both, this path has the potential to be quite coercive if no strong guarantees or safeties can be upheld, especially if early exciting trials generate an interest-boom similar to the one we're seeing right now in the LM-space.
      - 3y ago
- posterboy 3y ago
  > a behavior authoring loop A behaviour that authors loops?
  - germinalphrase 3y ago
    I would read it as “a loop that authors behaviors”.

AndrewKemendo 3y ago

+1 I’ll “yes and” here…beyond AR/VR a more powerful use case is multi-modal learning (with RL) which is what Meta is probably the leader in IMO. Example paper here: “ Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning” https://arxiv.org/abs/2301.10931 https://arxiv.org/abs/2301.10931 This IMO is the pathway to AGI, as it combines all sense-plan-do data into a time coordinated stream and mimics how humans transfer learning to children via demonstration recording and behavior authoring. If we can create robotics with locomotion and dexterous manipulation, egocentric exploration, and a behavior authoring loop that uses human behavior demonstration and trajectory reinforcement - well, we’ll have the AI we’ve been all talking about. Probably the most exciting area of research that most people don’t know or care about. That’s why head mounted all day ego centric AR is so important - it gives eyes ears and sense perception to our learning systems with human directed egocentric behaviors, guiding the whole thing. Just like pushing your kid down the street in the stroller.

socratic1 3y ago
+1 Applications of embodied AI very interesting. Additionally a lot of hard problems are increasingly being solved in simulation like this. See Wayve's GAIA world model
dleeftink 3y ago
Just to make sure I understand your excitement: we need guinea-pigs ahem people to wear 'head mounted all day ego centric AR' with who knows how many integrated sensors for long stretches on end, so we can finally get to our fabled A.G.i? That is some B.F. Skinner level future we're aiming for--only this time around, humans become the fully surveilled 'teaching machine'.
- AndrewKemendo 3y ago
  Well no...not guinea pigs. But correct conceptually - if it's opt-in only and perfectly transparent to everyone what is happening, which in this specific case of Aria it absolutely is. If we want to make machines with equivalent or better capacity as humans we have to transfer the process for scientific discovery, including the sum of our cognitive capacity and knowledge to them. If you quantize human adult-infant interactions, then it boils down to Human adults introducing learning trajectories, labeling input data and biasing weights with reinforcing behaviors for new reinforcement agents. If we can re-build the infrastructure to do precisely that, where the agent is in the place of the infant and society is in the place of the "Human Adult" then we will have re-built at scale the process for human development. The best way we know how to do this today is implementing transfer learning approaches from the basic human developmental research. I started down this road back in 2010 trying to follow the work of Frank Guerin out of the University of Aberdeen [1] [2]. [1]https://www.surrey.ac.uk/people/frank-guerin https://www.surrey.ac.uk/people/frank-guerin [2] https://scholar.google.co.uk/citations?view_op=view_citation&hl=en&user=OK8EYAoAAAAJ&cstart=20&pagesize=80&sortby=pubdate&citation_for_view=OK8EYAoAAAAJ:qUcmZB5y_30C https://scholar.google.co.uk/citations?view_op=view_citation...
  - dleeftink 3y ago
    But what about observer effects? People act differently when recorded, and rarely do we catch humans acting natural when knowingly observed (some of the early 24h/day Twitch streamers come to mind). And what happens once trials are done? How would people feel about their actions becoming part of a technology potentially able to replace them? Even when this barrier can be overcome (i.e. people become accustomed to wearing these devices), I worry about the opt-in nature of it. We've yet to see a disruptive technology adhering to this principle through-and-through, and if current learning efforts are anything to go by, training data is not something companies want to willingly let go or lose out on. Taken both, this path has the potential to be quite coercive if no strong guarantees or safeties can be upheld, especially if early exciting trials generate an interest-boom similar to the one we're seeing right now in the LM-space.
    - 3y ago
posterboy 3y ago
> a behavior authoring loop A behaviour that authors loops?
- germinalphrase 3y ago
  I would read it as “a loop that authors behaviors”.