As we were discussing recently how blind people use the computer, navigate the web, and write programs in code editors - ScreenAI and other ways of giving LLMs

As we were discussing recently how blind people use the computer, navigate the web, and write programs in code editors - ScreenAI and other ways of giving LLMs a visual mode are promising, giving people the ability to understand and interact with visual interfaces using natural language.