Right now we are constrained to using the DOM of webpages. This is because we need some reasonable way to detect what parts of a UI are interactable or not (tho

Right now we are constrained to using the DOM of webpages. This is because we need some reasonable way to detect what parts of a UI are interactable or not (though we do some magic to do this); this would be a bigger challenge for arbitrary desktop apps. Also, there would be new security aspects that would have to be meticulously considered if we were to allow an agent to control arbitrary desktop apps. Constraining the agent to the browser has nice alignment in this way.