8 ms·
Show HN: Tarsier – Vision utilities for web interaction agents
Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be.
Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.
Here's the Tarsier pipeline in a nutshell:
1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot
2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)
3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict
Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai https://reworkd.ai).
By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!
https://reworkd.ai/careers https://reworkd.ai/careers
- pk19238 2y agothis is such a creative solution. reminds me of how a team rendered wolfenstein into ASCII characters and fine tuned mistral to successfully play it.
- KhoomeiK 2y agoThanks! Yeah, it seems like a lot can be done with just text while we wait for multimodal models to catch up. The recent Platonic Representation Hypothesis [1] also suggests that different models, regardless of modality, build the same internal representations of the world. [1] https://arxiv.org/abs/2405.07987 https://arxiv.org/abs/2405.07987
- dbish 2y agoVery cool. We do something similar by combining OCR along with accessiblity data and other data (speech reco et. al.) for desktop based screensharing understanding, but evaluation compared to multi-modal LLMs has not been easy. How are you evaluating to come up with this number "consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%,"? fwiw so far we've seen that Azure has the best OCR for screenshot type data across the proprietary and open source models, though we are far more focused on grabbing data from desktop based applications then web pages so ymmv
- KhoomeiK 2y agoYup, evals can definitely be tough. We basically have a suite of several hundred web data extraction evals in a tool we built called Bananalyzer [1]. It's made it pretty straightforward for us to benchmark how accurately our agent generates code when it uses Tarsier-text (+ GPT-4) for perception v.s. Tarsier-screenshot (+ GPT-4V/o). Will have to look into supporting Azure OCR in Tarsier then—thanks for the tip! [1] https://github.com/reworkd/bananalyzer https://github.com/reworkd/bananalyzer
- timabdulla 2y agoNeat. Do you have the Bananalyzer eval results for Tarsier published somewhere?
- KhoomeiK 2y agoWe're hoping to release an evals paper about Bananalyzer this summer and compare Tarsier to a variety of other perception systems in it. The hard part with evaluating a perception/context system though is that it's very intertwined with the agent's architecture, and that's not something we're comfortable fully open-sourcing yet. We'll have to think of interesting ways to decouple the perception system and eval them with Bananalyzer.
- dbish 2y agoAwesome, will take a look at this. thank you
- SomaticPirate 2y agoSurprised to hear Azure beats AWS Textract. I found it to be the best OCR offering but that was when I was doing documents.
- navanchauhan 2y agoIn my experience Azure is probably the best OCR offering right now. They are also the only ones to be able to recognize my terrible handwriting.
- davedx 2y agoHow do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.
- KhoomeiK 2y agoTotally agreed—this is a design choice that basically comes from our agent architecture, and the codegen-based architecture that we think will likely proliferate for web agent tasks in the future. We provide Tarsier's text/screenshot to an LLM and have it write code with generically written selectors rather than the naive selectors that Tarsier assigns to each element. It's sort of like when you (as a human) write a web scraper and visually click on individual elements to look at the surrounding HTML structure / their selectors, but then end up writing code with more general selectors—not copypasting the selectors of the elements you clicked.
- davedx 2y agoOoh that's a very neat approach, great idea! Chains of thought across abstraction layers. Definitely worth a blog post I reckon. Good luck!
- KhoomeiK 2y agoThanks! We might put out a paper about it with some Carnegie Mellon collaborators this summer.
- bryanrasmussen 2y agoYou might like to look at https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing.
- ghxst 2y ago
- bckmn 2y agoReminds me of [Language as Intermediate Representation](https://chrisvoncsefalvay.com/posts/lair/ https://chrisvoncsefalvay.com/posts/lair/) - LLMs are optimized for language, so translate an image into language and they'll do better at modeling it.
- KhoomeiK 2y agoCool connection, hadn't seen this before but feels intuitively correct! I also formulate similar (but a bit more out-there) philosophical thoughts on word-meaning as being described by the topological structure of its corresponding images in embedding space, in Section 5.3 of my undergrad thesis [1]. [1] https://arxiv.org/abs/2305.16328 https://arxiv.org/abs/2305.16328
- shekhar101 2y agoTangential - I just want a decent (financial transaction) Table to text conversion that can retain the table structure well enough (e.g. merged cells) and have tried everything under the sun short of fine tuning my own model, including all the multimodal LLMs. None of them work very well without a lot of prompt engineering on case by case basis. Can this help? How can I set it up with a large number of pdfs that are sorted by type and extract tabular information? Any other suggestions?
- KhoomeiK 2y agoThat's an interesting problem—Tarsier probably isn't the best solution here since it's focused on webpage perception rather than any kind of OCR. But one could try adapting the `format_text` function in tarsier/text_format.py to convert any set of OCR annotations to a whitespace-structured string. Curious to see if that works.
- Oras 2y agoHave you tried AWS textract for table extraction then LLM to format the data?
- davedx 2y agoAzure have a decent set of offerings for this too, they work quite well: https://learn.microsoft.com/en-gb/azure/ai-services/document-intelligence/choose-model-feature?view=doc-intel-4.0.0 https://learn.microsoft.com/en-gb/azure/ai-services/document...
- vikp 2y agoThis isn't specifically tuned for tables (more for general pdf to markdown), but it's worked for some people with similar use-cases - https://github.com/VikParuchuri/marker https://github.com/VikParuchuri/marker
- derefr 2y agoOr how about the opposite? Give me a CLI tool to pipe implicitly-tabular space-padded text into — a smart cut(1) — where I can say "give me column 3" and it understands how to analyze the document as a whole (or at least a running sample of a dozen lines or so), to model the correct column boundaries, to extract the contents of that column. (Which would also include trimming off any space-padding from the content. I want the data, not a fixed-width field containing it!) For that matter, give me a CLI tool that takes in an entire such table, and lets me say "give me rows 4-6 of column Foo" — and it reads the table's header (even through fancy box-drawing line-art) to determine which column is Foo, ignores any horizontal dividing lines, etc. I'm not sure whether these tasks actually require full-on ML — probably just a pile of heuristics would work. Anything would be better than the low-level tools we have today.
- shodai80 2y agoHow do you know, for a specific webelement, what label it is associated with for a textbox or select? For instance, I might want to tag as you did where elements are, but I still need an association with a label, quite often, to determine what the actual context of the textbox or select is.
- awtkns 2y agoTarsier provides a mapping of element number (eg: [23]) to xpath. So for any tagged item we're able to map it back to the actual element in the DOM, allowing for easy interaction with the elements on the page.
- shodai80 2y agoI understand that, I assume you are tagging the node and making a basic xpath to the node/attribute with your tag id. Understood. But how relevant is tagging a node when I have no idea what the node is actually for? EX: Given a simple login form, I may not know if the label is above or below the username textbox. A password box would be below it. I have a hard time understanding the relevance to tagging without context. Tagging is basically irrelevant to any automated task if we do not know the context. I am not trying to diminish your great work, don't get me wrong, but if you don't have context I don't see much relevance. Youre doing something that is easily scripted with xpath templates which I've done for over a decade.
- awtkns 2y agoThis is where a LLM comes it. In a typical pipeline would tag a page, transform it into a textual representation and then pass it to an llm which would be able to reason about which field(s) are the one you're looking for much like a human.
- shodai80 2y agoMy point still stands. How do you augment data for an LLM when you know the context of a page? Do you go through every element and setup the data for an associated label? Do you use div scoping via offset parent through a script to generate associated div (good approach, bad in real-life conditions though)? Do you convert the DOM to JSON or some data structure? That means little because you still don't have context, you'd have to do it by hand every time the layout changes...and you would have to be very specific, which is a separate problem for modeling as layouts are modified. What if the UI can be modified to have different layout types, such as label above, label to side, label below...where this can be dynamically set. What I am pointing here is, even data modeling is mostly irrelevant unless you want to go through every page/permutation of a page...all the while hoping the layout isn't modified or back to training all over again...which is downtime, and at some point you'll realize its just better to store user created xpath's, as its quicker to update those than retrain. How do you reason with an LLM without going through any of the above? Automation cannot consistently have downtime for retraining, it's the antithesis for its purpose. Let's not even get into shadow dom issues. I am keying on your third bullet point on Github: "How can you inform a text-only LLM about the page's visual structure?" My questions suggest a gap in your awesome accomplishment.
- abrichr 2y agoCongratulations on shipping! In https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt... we use FastSAM to first segment the UI elements, then have the LLM describe each segment individually. This seems to work quite well; see https://twitter.com/OpenAdaptAI/status/1789430587314336212 https://twitter.com/OpenAdaptAI/status/1789430587314336212 for a demo. More coming soon!
- jackienotchan 2y agoLooking at OpenAdapt, I'm wondering why they didn't integrate Tarsier into AgentGPT, which is their flagship github repo but doesn't seem to be under active development anymore.
- KhoomeiK 2y agoWe have a lot more powerful use-cases for Tarsier in web data extraction at the moment. Stay tuned for a broader launch soon!
- v3ss0n 2y agoSince it is just a wrapper around hosted API if Google , can't be ran as local fully opensource
- KhoomeiK 2y agoMore OCR providers are on the roadmap and we'd love for you to contribute any local OCR models you think could be useful! I wouldn't call it a wrapper though :)
- jackienotchan 2y agoWhy was the Show HN text removed? Too much self promotion? You're a YC company, so I'm surprised the mods would do that. https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=true&query=tarsier&sort=byDate&type=story https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru... > Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be. Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance. Here's the Tarsier pipeline in a nutshell: 1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot 2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo) 3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai https://reworkd.ai). By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems! reworkd.ai/careers
- KhoomeiK 2y agoThanks for pointing this out! Yeah, it's pretty strange. We thought including Show HN text was encouraged to engage with the community?
- jackienotchan 2y agoDid you delete it or the mods?
- KhoomeiK 2y agoMust have been the mods, I spent quite a bit of time on the content lol
- 2y ago
- savy91 2y agoAm I wrong thinking this could very well be the backbone of an alternative to the Rabbit AI? Where you basically end up having possibly infinite tools for your LLM assistant to use to reach a goal without having to build api integrations.
- KhoomeiK 2y agoYup, it could! There are a lot of players in the generalist personal web agent space but I personally think that use-case will be eaten by big players since fundamental foundation model improvements are required. That being said, Tarsier is a great place to start for building an open-source web agent for automating cool little tasks. At Reworkd, we're focused on web agents for data extraction at scale, which isn't as hyped as the generalist agents but we find provides a lot of value and already works pretty well.
- jumploops 2y agoHow does the performance compare to VimGPT[0]? I assume the screenshot-based approach is similar, whereas the text approach should be improved? Very cool either way! [0] https://github.com/ishan0102/vimGPT https://github.com/ishan0102/vimGPT
- KhoomeiK 2y agoVimGPT couples the perception to a specific LLM/agent whereas Tarsier is solely a perception system that you can use for any uni/multi-modal web agent. So it's hard to compare, but you could say that VimGPT's performance probably lies somewhere in the middle of Tarsier's performance distribution (which varies as a function of your specific agent/prompt system).
- reidbarber 2y agoNeat! Been building something similar to the tagging feature in Typescript: https://github.com/reidbarber/webmarker https://github.com/reidbarber/webmarker The Python API on this is really nice though.
- bravura 2y agoA few questions: Does this work in headless mode? Are you getting a screenshot of the whole webpage including scrolling? Or just the visible part. The whole page, like singlepage.js would be great and is much more useful in many circumstances, although I'm not sure sure how to handle infinite scrolling. (If not, clean simple APIs for scrolling that don't require fiddling and experimentation would be great.) Instead of Google OCR (the only OCR), what about Apple's native OCR? That would be amazing.
- KhoomeiK 2y agoYes it does work headless and we do grab a fullpage screenshot including scrolling (by resizing viewport to content height). We haven’t had to deal with infinite scrolling much but that’s an interesting feature we’d appreciate a PR for. We haven’t tried Apple’s OCR but hopefully will integrate Azure OCR soon based on others’ advice.
- bravura 2y agoBy Apple OCR, I mean instead of calling an external cloud API which requires tokens, etc. I simply mean vision which runs on OSX. It can be done in about 30 lines of Swift code.
- esha_manideep 2y agoGreat work guys! How did you benchmark traiser's 10-20% better? Would love to see exactly how each method scored
- KhoomeiK 2y agoGreat question! See this thread: https://news.ycombinator.com/item?id=40369713 https://news.ycombinator.com/item?id=40369713
- jadbox 2y agoAnything like this for nodejs? (This is py)
- KhoomeiK 2y agoMaybe this project another commenter is working on? https://news.ycombinator.com/item?id=40373310 https://news.ycombinator.com/item?id=40373310
- wyclif 2y agoHey! I'm actually in the Philippines now, and I've spent a lot of time on the island of Bohol, which has the world's greatest concentration of tarsiers. In fact, I visited the Tarsier Wildlife Sanctuary on the island of Bohol, Philippines with my wife, which is the world's main tarsier sanctuary. So I was instantly intrigued by the name of the app. https://flickr.com/photos/wyclif/3271137617/in/album-72157613440681039/ https://flickr.com/photos/wyclif/3271137617/in/album-7215761...
- KhoomeiK 2y agoAwesome pics! We love tarsiers too