3 ms·
SAM is used to segment GUI elements, which are then sent to GPT4-V to be described. We then prompt GPT-4 with the user's actions and the descriptions of the GUI
by abrichr 3y ago
SAM is used to segment GUI elements, which are then sent to GPT4-V to be described. We then prompt GPT-4 with the user's actions and the descriptions of the GUI elements to generate new actions to accomplish different behaviors.