3 ms·

Very cool. We do something similar by combining OCR along with accessiblity data and other data (speech reco et. al.) for desktop based screensharing understand

by dbish 2y ago

Very cool. We do something similar by combining OCR along with accessiblity data and other data (speech reco et. al.) for desktop based screensharing understanding, but evaluation compared to multi-modal LLMs has not been easy. How are you evaluating to come up with this number "consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%,"? fwiw so far we've seen that Azure has the best OCR for screenshot type data across the proprietary and open source models, though we are far more focused on grabbing data from desktop based applications then web pages so ymmv

KhoomeiK 2y ago
Yup, evals can definitely be tough. We basically have a suite of several hundred web data extraction evals in a tool we built called Bananalyzer [1]. It's made it pretty straightforward for us to benchmark how accurately our agent generates code when it uses Tarsier-text (+ GPT-4) for perception v.s. Tarsier-screenshot (+ GPT-4V/o). Will have to look into supporting Azure OCR in Tarsier then—thanks for the tip! [1] https://github.com/reworkd/bananalyzer https://github.com/reworkd/bananalyzer
- timabdulla 2y ago
  Neat. Do you have the Bananalyzer eval results for Tarsier published somewhere?
  - KhoomeiK 2y ago
    We're hoping to release an evals paper about Bananalyzer this summer and compare Tarsier to a variety of other perception systems in it. The hard part with evaluating a perception/context system though is that it's very intertwined with the agent's architecture, and that's not something we're comfortable fully open-sourcing yet. We'll have to think of interesting ways to decouple the perception system and eval them with Bananalyzer.
- dbish 2y ago
  Awesome, will take a look at this. thank you
SomaticPirate 2y ago
Surprised to hear Azure beats AWS Textract. I found it to be the best OCR offering but that was when I was doing documents.
- navanchauhan 2y ago
  In my experience Azure is probably the best OCR offering right now. They are also the only ones to be able to recognize my terrible handwriting.
- dbish 2y ago
  Yes, Textract does not work as well for desktop screenshots from our testing

by dbish 2y ago

KhoomeiK 2y ago

Yup, evals can definitely be tough. We basically have a suite of several hundred web data extraction evals in a tool we built called Bananalyzer [1]. It's made it pretty straightforward for us to benchmark how accurately our agent generates code when it uses Tarsier-text (+ GPT-4) for perception v.s. Tarsier-screenshot (+ GPT-4V/o). Will have to look into supporting Azure OCR in Tarsier then—thanks for the tip! [1] https://github.com/reworkd/bananalyzer https://github.com/reworkd/bananalyzer

timabdulla 2y ago
Neat. Do you have the Bananalyzer eval results for Tarsier published somewhere?
- KhoomeiK 2y ago
  We're hoping to release an evals paper about Bananalyzer this summer and compare Tarsier to a variety of other perception systems in it. The hard part with evaluating a perception/context system though is that it's very intertwined with the agent's architecture, and that's not something we're comfortable fully open-sourcing yet. We'll have to think of interesting ways to decouple the perception system and eval them with Bananalyzer.
dbish 2y ago
Awesome, will take a look at this. thank you

SomaticPirate 2y ago

Surprised to hear Azure beats AWS Textract. I found it to be the best OCR offering but that was when I was doing documents.

navanchauhan 2y ago
In my experience Azure is probably the best OCR offering right now. They are also the only ones to be able to recognize my terrible handwriting.
dbish 2y ago
Yes, Textract does not work as well for desktop screenshots from our testing