4 ms·

Overall, Operator seems the same as Claude's Computer Use demo from a few months ago, including architecture requiring user to launch a VM, and a tendency to be

Overall, Operator seems the same as Claude's Computer Use demo from a few months ago, including architecture requiring user to launch a VM, and a tendency to be incorrect: https://news.ycombinator.com/item?id=41914989 https://news.ycombinator.com/item?id=41914989 Notably, Claude's Computer Use implementation made few waves in the AI Agent industry since that announcement despite the hype.

usaar333 2y ago
38% on osworld vs 22% for Claude. That seems like a jump
- achierius 2y ago
  But of course, after all the benchmark issues we've had thus far -- memorization, conflicts of interest, and just plainly low-quality questions -- I think it's fair to be suspicious of the extent to which these numbers will actually map to usability in the real world.
ninininino 2y ago
It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page). So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them. Like a general instructing an army.
famouswaffles 2y ago
Big jumps in benchmarks from Claude's Computer Use though. 87% vs 56% on Webvoyager 58.1% vs 36.2% on WebArena 38.1% vs 22% on OsWorld These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)
- timabdulla 2y ago
  OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. It is a big improvement over Claude Computer Use, but it is more of the same in the specific domain of browser tasks when comparing against browser-use agents (which can use the DOM, browser-specific APIs, and so on.) The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces. You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
  - famouswaffles 2y ago
    Well that's fair. I wasn't saying that this was necessarily at a level of competence to be useful, simply that it seemed to be a lot better than Claude.
  - cubefox 2y ago
    > OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. No. It's not matching them, it's clearly exceeding them. The previous post provided the numbers.
    - deleted 2y ago
      [deleted]
    - timabdulla 2y ago
      Those numbers are not the full story. Note that GP specifically says: "Big jumps in benchmarks from _Claude's Computer Use_ though." Claude Computer Use was not SOTA for browser tasks at the time of its release (and is still not.) In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same. See here for details: https://openai.com/index/computer-using-agent/ https://openai.com/index/computer-using-agent/
bko 2y ago
I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges. This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it) Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like: 1. Use web to search for [companies] with conditions 2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience 3. Collect the names for review Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.
- gregpr07 2y ago
  If you are worried about costs you can use Browser Use with deepseek which becomes super cheap! https://github.com/browser-use/browser-use https://github.com/browser-use/browser-use
  - bko 2y ago
    Exactly what I was look for. Thank you. I wish they had Gemini since the free tier is generous but I guess it's in the works. I'll take a look and see how bad it would be implement
    - jerpint 2y ago
      Browser use supports gemini via langchain
- kridsdale3 2y ago
  > Is there any open source or cheap ways to automate things on your computer? MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it. Personally, I hate it.
fsndz 2y ago
This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-operator-its-first?r=56ql7&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/transitions/p/openai-launches-...
- karmasimida 2y ago
  R1 is a fundamental blow to their value proposition right now, the uniqueness is gone, and forever open sourced. Unless o3 is the game changer of game changer, I am not seeing they are getting the narrative back soon.
  - MagMueller 2y ago
    You can use browser-use as open-source alternative for Operator
    - fsndz 2y ago
      possible to use it with R1 for the reasoning part ?
      - porridgeraisin 2y ago
        Yeah I saw someone do this on Twitter recently... Can't seem to find it, as you do with twitter
minimaxir 2y ago
Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.
mceoin 2y ago
Claude's Computer Use API has been good for us and I'm surprised it isn't more popular. It can be slow, and definitely gets things wrong, but so far we've had thousands of people make and edit spreadsheets on "autopilot" so the value is already there today on simple tasks, even in an alpha state. I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL. Will report back how it compares vs Operator CUA once we get access!

usaar333 2y ago

38% on osworld vs 22% for Claude. That seems like a jump

achierius 2y ago
But of course, after all the benchmark issues we've had thus far -- memorization, conflicts of interest, and just plainly low-quality questions -- I think it's fair to be suspicious of the extent to which these numbers will actually map to usability in the real world.

ninininino 2y ago

It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page). So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them. Like a general instructing an army.

famouswaffles 2y ago

Big jumps in benchmarks from Claude's Computer Use though. 87% vs 56% on Webvoyager 58.1% vs 36.2% on WebArena 38.1% vs 22% on OsWorld These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)

timabdulla 2y ago
OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. It is a big improvement over Claude Computer Use, but it is more of the same in the specific domain of browser tasks when comparing against browser-use agents (which can use the DOM, browser-specific APIs, and so on.) The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces. You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
- famouswaffles 2y ago
  Well that's fair. I wasn't saying that this was necessarily at a level of competence to be useful, simply that it seemed to be a lot better than Claude.
- cubefox 2y ago
  > OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. No. It's not matching them, it's clearly exceeding them. The previous post provided the numbers.
  - deleted 2y ago
    [deleted]
  - timabdulla 2y ago
    Those numbers are not the full story. Note that GP specifically says: "Big jumps in benchmarks from _Claude's Computer Use_ though." Claude Computer Use was not SOTA for browser tasks at the time of its release (and is still not.) In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same. See here for details: https://openai.com/index/computer-using-agent/ https://openai.com/index/computer-using-agent/

bko 2y ago

I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges. This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it) Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like: 1. Use web to search for [companies] with conditions 2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience 3. Collect the names for review Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.

gregpr07 2y ago
If you are worried about costs you can use Browser Use with deepseek which becomes super cheap! https://github.com/browser-use/browser-use https://github.com/browser-use/browser-use
- bko 2y ago
  Exactly what I was look for. Thank you. I wish they had Gemini since the free tier is generous but I guess it's in the works. I'll take a look and see how bad it would be implement
  - jerpint 2y ago
    Browser use supports gemini via langchain
kridsdale3 2y ago
> Is there any open source or cheap ways to automate things on your computer? MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it. Personally, I hate it.

fsndz 2y ago

This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-operator-its-first?r=56ql7&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/transitions/p/openai-launches-...

karmasimida 2y ago
R1 is a fundamental blow to their value proposition right now, the uniqueness is gone, and forever open sourced. Unless o3 is the game changer of game changer, I am not seeing they are getting the narrative back soon.
- MagMueller 2y ago
  You can use browser-use as open-source alternative for Operator
  - fsndz 2y ago
    possible to use it with R1 for the reasoning part ?
    - porridgeraisin 2y ago
      Yeah I saw someone do this on Twitter recently... Can't seem to find it, as you do with twitter

minimaxir 2y ago

Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.

mceoin 2y ago

Claude's Computer Use API has been good for us and I'm surprised it isn't more popular. It can be slow, and definitely gets things wrong, but so far we've had thousands of people make and edit spreadsheets on "autopilot" so the value is already there today on simple tasks, even in an alpha state. I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL. Will report back how it compares vs Operator CUA once we get access!