4 ms·

I summon Google Assistant and press the what's on my screen button, that one usually allows me to select text even when the apps don't.

• RadiozRadioz 2y ago
  The layers of extreme complexity in this situation are astounding. The app has some text internally, it renders it by rasterizing fonts to a bitmap, the OS takes the bitmap and composites it within the wider UI. Google Assistant grabs a screenshot of the fully composited, post-processed OS UI, sends it over the internet to a server, which uses an OCR model to read all the text, and a different model to work out which is the relevant text, which is then sent back over the internet and displayed on the device. All so the user can copy some text.
  • EddieRingle 2y ago
    On Android it more or less just uses the accessibility APIs to grab the actual text, you can do it without using Google Assistant even by selecting text inside an app's thumbnail window from the Recent Apps screen.
    • Nullabillity 2y ago
      It doesn't. You can try running it when your keyboard is open and see what an absolute trainwreck the OCR makes of that.
  • ludwigschubert 2y ago
    I used to be upset at such absurdities. Now I try to appreciate what a relatively universal interface images are instead! :D It doesn’t matter how text gets drawn if we OCR it. (Though I feel better when we do so on device.)