4 ms·
How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages
by davedx 2y ago
How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.
- KhoomeiK 2y agoTotally agreed—this is a design choice that basically comes from our agent architecture, and the codegen-based architecture that we think will likely proliferate for web agent tasks in the future. We provide Tarsier's text/screenshot to an LLM and have it write code with generically written selectors rather than the naive selectors that Tarsier assigns to each element. It's sort of like when you (as a human) write a web scraper and visually click on individual elements to look at the surrounding HTML structure / their selectors, but then end up writing code with more general selectors—not copypasting the selectors of the elements you clicked.
- davedx 2y agoOoh that's a very neat approach, great idea! Chains of thought across abstraction layers. Definitely worth a blog post I reckon. Good luck!
- KhoomeiK 2y agoThanks! We might put out a paper about it with some Carnegie Mellon collaborators this summer.
- bryanrasmussen 2y agoYou might like to look at https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing.
- ghxst 2y agoGreat question, also situations where you have multiple CTAs with similar names/contexts on a page is still something I see LLM based automation struggle with.
- KhoomeiK 2y agoHm, not sure I follow why those situations would be especially difficult? Regarding website changes, the nice thing about using LLMs is that we can simply provide the previous scraper as context and have it regenerate the scraper to "self-heal" when significant website changes are detected.