3 ms·
At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/Ope
by abrichr 3y ago
At OpenAdapt we have had excellent results combining Segment Anything Model (SAM) with GPT-4 for screen understanding.
Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 https://github.com/OpenAdaptAI/OpenAdapt/pull/610
- spxneo 3y agoMIT license too (getting rare)! mad respects, thank you.
- abrichr 3y agoThank you!
- williamdelo32 3y agoInteresting. Is this relying on SAM to segment text to send to GPT? How does it perform compared to GPT-V?
- abrichr 3y agoSAM is used to segment GUI elements, which are then sent to GPT4-V to be described. We then prompt GPT-4 with the user's actions and the descriptions of the GUI elements to generate new actions to accomplish different behaviors.