5 ms·

How Chrome Accessibility Works

59 points by lelandfe 2y ago

mltony 2y ago
Could you guys fix this accessibility bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1260726 https://bugs.chromium.org/p/chromium/issues/detail?id=126072... - it's been open for years.
vhiremath4 2y ago
It seems like small multimodal LLMs have a killer use case to be bundled with browsers for accessibility. Eventually: * if an image doesn’t have alt text * you need to be read the page * you need to be described what’s happening in a video A model built into the OS or browser seems like a no-brainer.
- qingcharles 2y ago
  Right now I use LLMs to generate alt text for images, and they are better than any I would have written by hand. Only in about 1% of cases do I need to correct anything.
- yellow_postit 2y ago
  Edge started doing this a few years ago: https://blogs.windows.com/msedgedev/2022/03/17/appears-to-say-microsoft-edge-auto-generated-image-labels/ https://blogs.windows.com/msedgedev/2022/03/17/appears-to-sa...
- deleted 2y ago
  [deleted]
- dmazzoni 2y ago
  Chrome already has optional built-in support for generating alt text for images. It's been there for years, using a server-based API. It does seem possible that this could be replaced with a local model in the near future. It's not clear the average user has the hardware specs for this to be an option today, but it will increasingly be plausible. Keep in mind, though, that alt text is just one small part of making a web site accessible.
  - RandallBrown 2y ago
    > It does seem possible that this could be replaced with a local model in the near future. It's not clear the average user has the hardware specs for this to be an option today, but it will increasingly be plausible. Siri does something like this when reading messages into your AirPods. It will give brief descriptions of photos sent in the message. I'm pretty sure it's all run locally.
    - freeone3000 2y ago
      Siri has the advantage of running on either an iPhone or a Macbook. Chrome has to run on budget android phones and chromebooks.
- bbb651 2y ago
  https://hacks.mozilla.org/2024/05/experimenting-with-local-alt-text-generation-in-firefox-nightly/ https://hacks.mozilla.org/2024/05/experimenting-with-local-a...
- mananaysiempre 2y ago
  > A model built into the OS or browser seems like a no-brainer. How about localizing it into all languages supported by a major OS or browser?
- bbertucc 2y ago
  LLM-generated descriptions miss lots of context. For instance, depending on the site and content, we might mention people's races or fashion. Other times we don't.
partiallypro 2y ago
Slightly unrelated, but there's a whole industry on ADA compliance and randomly trying to sue people over it in the digital space (for website specifically.) Which seems like an issue that could -largely- be solved by the browsers themselves (animation pauses, contrast, font sizes, readable font switching, etc.) There's no telling how much companies spend on services like Accessible or Userway, etc. There's also a new industry on cookie & privacy compliance because of this too. I get privacy and ADA compliance, but some of this is just clearly written by the people making the tools. It's regulatory capture to the fullest extent. I found this out recently with a client wanting to pay through the nose to essentially have a whole separate company paid monthly to make sure the privacy policy and cookie policies meet various state laws. It's a bit ridiculous.
- joemi 2y ago
  I'm not so sure that "solving" accessibility at the browser level would eliminate those kinds of lawsuits. It might be able to be argued that their "client" uses a different browser that doesn't include those accessibility features, and the website in question (that relies on the browsers fixing accessibility) isn't ADA compliant in that case.
  - zamadatix 2y ago
    I don't know how realistic the approach in general is but "This site only works in ${SpecificBrowser}" or "Your browser does not support ${'Standard'}" banner refusals have yet to result in a business getting sued for not displaying properly in an arbitrary browser that isn't supported.
    - joemi 2y ago
      I think the reason behind that is simple: In my experience, there are orders of magnitude more not-entirely-ADA-compliant websites than there are sites with banners such as the ones you mentioned. But yeah, I don't know if the site-isn't-compliant-in-certain-browsers-equals-discrimination argument would succeed or not. But I bet the people whose income dries as a result of browser-based-compliance up would definitely try that tactic.

59 points by lelandfe 2y ago

Could you guys fix this accessibility bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1260726 https://bugs.chromium.org/p/chromium/issues/detail?id=126072... - it's been open for years.

vhiremath4 2y ago

It seems like small multimodal LLMs have a killer use case to be bundled with browsers for accessibility. Eventually: * if an image doesn’t have alt text * you need to be read the page * you need to be described what’s happening in a video A model built into the OS or browser seems like a no-brainer.

qingcharles 2y ago
Right now I use LLMs to generate alt text for images, and they are better than any I would have written by hand. Only in about 1% of cases do I need to correct anything.
yellow_postit 2y ago
Edge started doing this a few years ago: https://blogs.windows.com/msedgedev/2022/03/17/appears-to-say-microsoft-edge-auto-generated-image-labels/ https://blogs.windows.com/msedgedev/2022/03/17/appears-to-sa...
deleted 2y ago
[deleted]
dmazzoni 2y ago
Chrome already has optional built-in support for generating alt text for images. It's been there for years, using a server-based API. It does seem possible that this could be replaced with a local model in the near future. It's not clear the average user has the hardware specs for this to be an option today, but it will increasingly be plausible. Keep in mind, though, that alt text is just one small part of making a web site accessible.
- RandallBrown 2y ago
  > It does seem possible that this could be replaced with a local model in the near future. It's not clear the average user has the hardware specs for this to be an option today, but it will increasingly be plausible. Siri does something like this when reading messages into your AirPods. It will give brief descriptions of photos sent in the message. I'm pretty sure it's all run locally.
  - freeone3000 2y ago
    Siri has the advantage of running on either an iPhone or a Macbook. Chrome has to run on budget android phones and chromebooks.
bbb651 2y ago
https://hacks.mozilla.org/2024/05/experimenting-with-local-alt-text-generation-in-firefox-nightly/ https://hacks.mozilla.org/2024/05/experimenting-with-local-a...
mananaysiempre 2y ago
> A model built into the OS or browser seems like a no-brainer. How about localizing it into all languages supported by a major OS or browser?
bbertucc 2y ago
LLM-generated descriptions miss lots of context. For instance, depending on the site and content, we might mention people's races or fashion. Other times we don't.

partiallypro 2y ago

Slightly unrelated, but there's a whole industry on ADA compliance and randomly trying to sue people over it in the digital space (for website specifically.) Which seems like an issue that could -largely- be solved by the browsers themselves (animation pauses, contrast, font sizes, readable font switching, etc.) There's no telling how much companies spend on services like Accessible or Userway, etc. There's also a new industry on cookie & privacy compliance because of this too. I get privacy and ADA compliance, but some of this is just clearly written by the people making the tools. It's regulatory capture to the fullest extent. I found this out recently with a client wanting to pay through the nose to essentially have a whole separate company paid monthly to make sure the privacy policy and cookie policies meet various state laws. It's a bit ridiculous.

joemi 2y ago
I'm not so sure that "solving" accessibility at the browser level would eliminate those kinds of lawsuits. It might be able to be argued that their "client" uses a different browser that doesn't include those accessibility features, and the website in question (that relies on the browsers fixing accessibility) isn't ADA compliant in that case.
- zamadatix 2y ago
  I don't know how realistic the approach in general is but "This site only works in ${SpecificBrowser}" or "Your browser does not support ${'Standard'}" banner refusals have yet to result in a business getting sued for not displaying properly in an arbitrary browser that isn't supported.
  - joemi 2y ago
    I think the reason behind that is simple: In my experience, there are orders of magnitude more not-entirely-ADA-compliant websites than there are sites with banners such as the ones you mentioned. But yeah, I don't know if the site-isn't-compliant-in-certain-browsers-equals-discrimination argument would succeed or not. But I bet the people whose income dries as a result of browser-based-compliance up would definitely try that tactic.