I used molmo to parse screenshots in order to detect locations of UI elements. See the repo below. I think Omni parser from Microsoft would also work well. htt

I used molmo to parse screenshots in order to detect locations of UI elements. See the repo below. I think Omni parser from Microsoft would also work well. https://github.com/logankeenan/george https://github.com/logankeenan/george https://github.com/microsoft/OmniParser https://github.com/microsoft/OmniParser