You don't really need metadata to automate in a terminal. Literally everything is there in the clear and ANSI is used for control. It's all characters; no need

You don't really need metadata to automate in a terminal. Literally everything is there in the clear and ANSI is used for control. It's all characters; no need to think about pixels (I made something that automates TUIs[0] BTW. For GUIs you have to handle the particular metadata of a given framework, if it exposes metadata at all for the controls. Otherwise you're dealing with raw pixels which is an instant nightmare, though not so much now there are visual LLMs, but you're still paying for the tokens and/or extra processing. [0] https://gitlab.com/skeledrew/jaiver https://gitlab.com/skeledrew/jaiver