4 ms·

We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https

We haven't been able to use Claude 3 Opus vision yet because we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress). Unfortunately we can't compare it to ScreenAI directly since as far as I can tell it is not generally available. However ScreenAI does not appear to use a separate segmentation step, which we needed to implement in order to get good results.

spxneo 3y ago
can you elaborate on "extremely well"? where is it currently falling short?
- abrichr 3y ago
  You can see how it performs in describing GUI elements in the linked PR if you scroll down -- here's a direct link: https://private-user-images.githubusercontent.com/774615/320280312-508d763f-fab9-4c21-aafb-eb47d16a723a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI3MTYwNzQsIm5iZiI6MTcxMjcxNTc3NCwicGF0aCI6Ii83NzQ2MTUvMzIwMjgwMzEyLTUwOGQ3NjNmLWZhYjktNGMyMS1hYWZiLWViNDdkMTZhNzIzYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDEwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQxMFQwMjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZDNiYTEwMWQ2MjJkYzNjYjU3ZTYzZThmZDBlN2FmZTI4NmZhOTg4NGQzNzFiYTFlOTkwYWI4MDczOWIyOGZmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9o1iugVRDddReqz9Gs5yjcSBTR2LBbbiJj5I_V5CrXA https://private-user-images.githubusercontent.com/774615/320... Regarding failure modes, we have yet to do extensive testing, but I've seen it confuse the divide and subtract buttons on the calculator before only once.
abrichr 3y ago
Update: it looks like Anthropic now accepts Canadian credit cards. The results are not as good as GPT4-V or Gemini. I've posted the output for each of `gpt-4-vision-preview`, `gpt-4-turbo-2024-04-09`, `gemini-1.5-pro-latest`, and `claude-3-opus-20240229`. Claude is the only one who makes mistakes, at least on that test.

spxneo 3y ago

can you elaborate on "extremely well"? where is it currently falling short?

abrichr 3y ago
You can see how it performs in describing GUI elements in the linked PR if you scroll down -- here's a direct link: https://private-user-images.githubusercontent.com/774615/320280312-508d763f-fab9-4c21-aafb-eb47d16a723a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI3MTYwNzQsIm5iZiI6MTcxMjcxNTc3NCwicGF0aCI6Ii83NzQ2MTUvMzIwMjgwMzEyLTUwOGQ3NjNmLWZhYjktNGMyMS1hYWZiLWViNDdkMTZhNzIzYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDEwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQxMFQwMjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZDNiYTEwMWQ2MjJkYzNjYjU3ZTYzZThmZDBlN2FmZTI4NmZhOTg4NGQzNzFiYTFlOTkwYWI4MDczOWIyOGZmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9o1iugVRDddReqz9Gs5yjcSBTR2LBbbiJj5I_V5CrXA https://private-user-images.githubusercontent.com/774615/320... Regarding failure modes, we have yet to do extensive testing, but I've seen it confuse the divide and subtract buttons on the calculator before only once.

abrichr 3y ago

Update: it looks like Anthropic now accepts Canadian credit cards. The results are not as good as GPT4-V or Gemini. I've posted the output for each of `gpt-4-vision-preview`, `gpt-4-turbo-2024-04-09`, `gemini-1.5-pro-latest`, and `claude-3-opus-20240229`. Claude is the only one who makes mistakes, at least on that test.