3 ms·

V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

• dbish 3y ago
  Interesting, I wonder if there's a similar approach that can be taken for complex tasks in the "multimodal" virtual environment of a computer desktop or phone, working across/between multiple applications and web pages with various inputs and interactions. Taking a more general approach that might be closer to how humans work then ACT-1 and the like.
• kromem 3y ago
  I've actually been thinking recently of the embodiment approach here in the fact that foveated rendering in VR/AR and eye tracking for 3D laptops means there's going to be much more eye tracking data that could feed into ML systems around what humans pay attention to given a paired image. I'd imagine that data, combined with things like this, could really streamline visual processing, particularly for video.
  • htrp 3y ago
    Outside of facebook... who would be collecting that data in a systematic manner?
    • kromem 3y ago
      Mostly Meta, but it's not like they don't have AI ambitions. Apple will soon. Media companies will eventually have it too as they roll out apps for the hardware. I don't think it will take much data relative to overall visual training data. As a supplementary layer, a little bit goes a long way.