> wondering how I can hookup a custom backend then for the boundary box detection, which appears is not supported. You can take a screenshot with: pyautog

> wondering how I can hookup a custom backend then for the boundary box detection, which appears is not supported. You can take a screenshot with: pyautogui.screenshot() With your neural network you can have the coordinates of what you want, and act with pyautogui afterwards. In many cases, a neural network can even be overkill, take a look at this https://vimeo.com/352072921 https://vimeo.com/352072921 The script takes a screenshot of the webpage, recognize the current highlighted word with pytesseract and type it in with pyautogui, simple.