20 ms·
Agent4 is going to store "stable selectors" that worked (when it performs a task first time most of the time is spent in identifying these css/xpath selectors).
by artpar 1y ago
Agent4 is going to store "stable selectors" that worked (when it performs a task first time most of the time is spent in identifying these css/xpath selectors). Memories are pretty straighforward at this point, they are stored locally in your browser's IndexedDB (you can inspect from chrome inspector).
- philo23 1y agoGood to hear, that’s what I was hoping that it was doing.
- erichocean 1y agoHow are you mapping from "click this element" (presumably obtained via a VLM) to the actual DOM locator that refers to it? I guess Playwright can do it in "record" mode; I'm curious how you do it from a Chrome extension. Spitballing here, you inject an event filter on the page and when the click happens, grab the element and run some code to synthesize a selector that just refers to that element? (Presumably you could just reuse Playwright's element-to-locator code at this point.)
- artpar 1y agoSo when you go into the "selector" mode, the plugin will add event listeners to all the DOM nodes. Based on your click it will try to generate a bunch of selectors statically first (multiple, css and xpath based), and then based on your guidance its the job of agent4 to make stable selectors.
- cjr 1y agodocument.elementFromPoint to get the elem at co-ordinates, then use npm package similar to optimal-select to come up with a unique css selector.