3 ms·
How do you think humans experience desktop interfaces? “Basically just OCR'ing screenshots” is exactly what humans do.
by Timwi 3mo ago
How do you think humans experience desktop interfaces? “Basically just OCR'ing screenshots” is exactly what humans do.
- gabriel-uribe 3mo agoI was also under the impression modern AI agents have moved on from just OCR'ing screenshots to leveraging native vision model capabilities.
- ewild 3mo agoThey do. They all use ViTs and have for quite a while.
- reasonableklout 3mo agoIt's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.
- kortilla 3mo agoBut that’s not how models handle images either. They spatially segment and reason about title bars, placement, etc.