With a multimodal llm watching a stream of these objects, should be able answer the question - "where are my keys?" It should be able to respond - "Here on t

With a multimodal llm watching a stream of these objects, should be able answer the question - "where are my keys?" It should be able to respond - "Here on the dining room table" with an astar path drawn to them in the headset. It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).