4 ms·

Screen readers rely on some kind of toolkit a.p.i.? I would assume they would simply read the screen. Are they thus not capable of, for instance, reading a pic

Screen readers rely on some kind of toolkit a.p.i.? I would assume they would simply read the screen. Are they thus not capable of, for instance, reading a picture? They should not be called “screen readers” but “text to speech” if they not actually read the screen on a bitmap level.

riquito 6y ago
Screen readers, despite the name, don't do OCR, they access informations provided by the GUI toolkit (which is one of the vastly improved area in GTK 4 Afaik).
- Blikkentrekker 6y ago
  Then why don't actual screen readers exist when a.i.'s at this point can practically solve captcha's by enhancing the reflexion in the eyes of a highly compressed jpeg and reading the text in there? Certainly there would be significant demand for a sightless man to be able to read the dankest memes from pictures?
  - danShumway 6y ago
    > Then why don't actual screen readers exist when a.i.'s at this point can practically solve captcha's Training a computer to solve a captcha is a lot easier than training a computer to understand interface conventions. There isn't an AI that can look at a jpeg screenshot of an interface and say, "there's two input elements, and it looks like they're grouped together and control the list to the left of them, and one of them is selected, which I can tell because it has some kind of subtle glow effect on it, but not the glow effect you would get if you moused over it." There's nothing that can realistically do that today, and it wouldn't be fast enough or performant enough for low-powered cell phones and laptops even if it did exist. If you're just looking at describing pictures themselves... sure, Facebook does auto-generate alt tags for images if you forget to put one in. And Youtube auto-generates captions. Those are valuable services, but they have a lot of glitches and mistakes. If you're a blind reader, you'd prefer not to have that experience when you're using a piece of software, you'd prefer something that just works reliably. It's the same reason you probably used a keyboard to type this comment instead of speech to text. Speech to text is useful in some cases, but not good enough or accurate enough that you would want to use it as your main input method.
- mwcampbell 6y ago
  Some screen readers can do OCR on a specific part of the screen (e.g. an unlabeled image) on request. But while OCR is useful for getting text out of an image, most implementations can't discern the structure of a UI, e.g. which part is a button, which part is an edit box, etc. Also, OCR is typically done just once on-demand, not continuously as the screen changes. However, VoiceOver for iOS has a new feature called screen recognition, which is exciting because it overcomes these limitations and provides some level of access to applications that are otherwise inaccessible. Hopefully other platforms will catch up. Even then, true screen reading will be much more CPU-intensive than what screen readers currently do. And anyway, it's not here yet, except on iOS. So I will continue to warn developers away from toolkits that are inaccessible, in hopes that some blind person somewhere will be spared the pain of being blocked from doing a task because of an inaccessible application.
deleted 6y ago
[deleted]
simias 6y ago
What you describe might be more robust in some situations but vastly less for software designed to be accessible. Take "alt" attributes in HTML img tags for instance, or various metadata attached to buttons that use an icon instead of text (like a play button, or a X close button and similar things). You can see an example next to this very comment actually: the up and downvote buttons won't be accessible with OCR, but they have "title" attributes describing what they are. And consider that there's more to understanding a given user interface that raw text: radio buttons are tied to certain labels, there's hierarchy, all sorts of layout cues that would be opaque to a screen reader. I suppose that ideally you'd want both: use native accessibility data if available, fallback to OCR when there's no alternative.
danShumway 6y ago
Converting bitmaps to text isn't enough to make an interface usable. You need to be able to quickly convey the structure of the interface and what controls are available, and to do that well you need some kind of semantic insight into the interface. Screenreaders don't just read text, they control the interface itself using standardized keyboard shortcuts and input components within whatever graphical framework you're using, and they communicate what that interface is using a set of standardized terminology.
- Blikkentrekker 6y ago
  I indeed am imparted that the term “screen reader” is a misnomer and not what I expected it to be. This seems to be more so an “accessibility suite” of which text-to-speech is a component rather than what I would call a “screen reader".
  - danShumway 6y ago
    Sorry, I guess? It's been an industry standard phrase for a pretty long time. Screenreaders probably have a historical reason why they're named the way they are, but the short version is that's just what everyone started using. A lot of software terminology is like that, it's weird to people who are unfamiliar with it because there's no central committee somewhere that decides what everything should be named. If you're trying to do a search online for the kind of tool you're looking for, probably the phrase you would want to search for is "OCR software", short for Optical Character Recognition, or if you're trying to tag images just straight-up "image recognition."

riquito 6y ago

Screen readers, despite the name, don't do OCR, they access informations provided by the GUI toolkit (which is one of the vastly improved area in GTK 4 Afaik).

Blikkentrekker 6y ago
Then why don't actual screen readers exist when a.i.'s at this point can practically solve captcha's by enhancing the reflexion in the eyes of a highly compressed jpeg and reading the text in there? Certainly there would be significant demand for a sightless man to be able to read the dankest memes from pictures?
- danShumway 6y ago
  > Then why don't actual screen readers exist when a.i.'s at this point can practically solve captcha's Training a computer to solve a captcha is a lot easier than training a computer to understand interface conventions. There isn't an AI that can look at a jpeg screenshot of an interface and say, "there's two input elements, and it looks like they're grouped together and control the list to the left of them, and one of them is selected, which I can tell because it has some kind of subtle glow effect on it, but not the glow effect you would get if you moused over it." There's nothing that can realistically do that today, and it wouldn't be fast enough or performant enough for low-powered cell phones and laptops even if it did exist. If you're just looking at describing pictures themselves... sure, Facebook does auto-generate alt tags for images if you forget to put one in. And Youtube auto-generates captions. Those are valuable services, but they have a lot of glitches and mistakes. If you're a blind reader, you'd prefer not to have that experience when you're using a piece of software, you'd prefer something that just works reliably. It's the same reason you probably used a keyboard to type this comment instead of speech to text. Speech to text is useful in some cases, but not good enough or accurate enough that you would want to use it as your main input method.
mwcampbell 6y ago
Some screen readers can do OCR on a specific part of the screen (e.g. an unlabeled image) on request. But while OCR is useful for getting text out of an image, most implementations can't discern the structure of a UI, e.g. which part is a button, which part is an edit box, etc. Also, OCR is typically done just once on-demand, not continuously as the screen changes. However, VoiceOver for iOS has a new feature called screen recognition, which is exciting because it overcomes these limitations and provides some level of access to applications that are otherwise inaccessible. Hopefully other platforms will catch up. Even then, true screen reading will be much more CPU-intensive than what screen readers currently do. And anyway, it's not here yet, except on iOS. So I will continue to warn developers away from toolkits that are inaccessible, in hopes that some blind person somewhere will be spared the pain of being blocked from doing a task because of an inaccessible application.

deleted 6y ago

[deleted]

simias 6y ago

What you describe might be more robust in some situations but vastly less for software designed to be accessible. Take "alt" attributes in HTML img tags for instance, or various metadata attached to buttons that use an icon instead of text (like a play button, or a X close button and similar things). You can see an example next to this very comment actually: the up and downvote buttons won't be accessible with OCR, but they have "title" attributes describing what they are. And consider that there's more to understanding a given user interface that raw text: radio buttons are tied to certain labels, there's hierarchy, all sorts of layout cues that would be opaque to a screen reader. I suppose that ideally you'd want both: use native accessibility data if available, fallback to OCR when there's no alternative.

danShumway 6y ago

Converting bitmaps to text isn't enough to make an interface usable. You need to be able to quickly convey the structure of the interface and what controls are available, and to do that well you need some kind of semantic insight into the interface. Screenreaders don't just read text, they control the interface itself using standardized keyboard shortcuts and input components within whatever graphical framework you're using, and they communicate what that interface is using a set of standardized terminology.

Blikkentrekker 6y ago
I indeed am imparted that the term “screen reader” is a misnomer and not what I expected it to be. This seems to be more so an “accessibility suite” of which text-to-speech is a component rather than what I would call a “screen reader".
- danShumway 6y ago
  Sorry, I guess? It's been an industry standard phrase for a pretty long time. Screenreaders probably have a historical reason why they're named the way they are, but the short version is that's just what everyone started using. A lot of software terminology is like that, it's weird to people who are unfamiliar with it because there's no central committee somewhere that decides what everything should be named. If you're trying to do a search online for the kind of tool you're looking for, probably the phrase you would want to search for is "OCR software", short for Optical Character Recognition, or if you're trying to tag images just straight-up "image recognition."