6 ms·

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computers

77 points by kristianpaul 2y ago

stavros 2y ago
I built a small Python script so I could let GPT-4 debug my system issues: https://github.com/skorokithakis/sysaidmin https://github.com/skorokithakis/sysaidmin It works surprisingly well!
- ec109685 2y ago
  What’s the most complicated thing it was able to debug?
  - stavros 2y ago
    I had a library missing and it started using ldd to see where things were linked, and it changed the library path to the proper location, it was impressive.
    - bamboozled 2y ago
      Sounds like the state of your computer is the real problem here.
      - stavros 2y ago
        As opposed to?
        bamboozled 2y ago
        https://nixos.org/ https://nixos.org/ But just in general, I've been a Linux pro for about 15 years, it's extremely rare I'd have to manually link anything?
        stavros 2y ago
        Well, when you need to run software that needs a library and your system doesn't have it because the library is too old, what do you do?
TheRoque 2y ago
Gotta love people working on replacing themselves. Jokes aside, seeing an AI interacting with a computer is kind of scary. It's not just outputting text anymore, it's doing the full work of a human working on a computer, meaning... a ton of people
- mindcrime 2y ago
  Gotta love people working on replacing themselves. So here's the thing: I want the computer to replace me... with regards to things that I don't want to do, or that aren't a good use of my time (boring, repetitive, trite, tedious, etc.). BUT the caveat is, I want it to be in a context where I'm in control of the computer and own the output, etc. Now companies and other actors also want the computer to replace me, but they want to be in control and to own the output. But here's the rub: they're not going to stop trying to come up with ways to replace me and minimize my value just because I don't like it. No man can stop the tide from coming in, or something like that (insert pithy saying here?). But if you know the tide is coming, you can retreat further inland OR build a boat, or climb a pole, etc. So where I'm at with this is, the more I work on AI, the more I know about AI, the more I can use it to enhance my own value outside of the context of corporate controlled IT systems, the more I can exercise a (possibly insignificant, but maybe not) degree of control over my fate. I can try to build my own boat, or at least outrun the tide for a little while. I don't know how things are going to turn out, and there's a reason that I am not in fact joking when I crack the occasional remark about "after this conversation I'm going to run to Bass Pro Shops and pick up a few more boxes of ammo" or whatever. Maybe a cyberpunk dystopia is what awaits us all. Maybe not. I'm just going to try to make my best guess at what the possible / likely future scenarios are and take whatever actions I can now to situate myself as favorably as I can.
  - squigz 2y ago
    > "after this conversation I'm going to run to Bass Pro Shops and pick up a few more boxes of ammo" or whatever. Maybe a cyberpunk dystopia is what awaits us all. I don't mean this to sound accusatory, but do you think there's at all a connection between the probability of a dystopic future and the prevalence of people betting that that's the future?
ec109685 2y ago
Buried in their presentation is the current effectiveness of agents to complete desktop computing tasks. Humans are able to complete the tasks given at 70%+ effectiveness while the best model is at 12% (GPT4-v). Most of the other models were <5% effective.
- nicklecompte 2y ago
  I think this is a clear case where “human-level” is a totally meaningless expression, since you have to ask about which humans[1]. On one hand, these were CS undergrads, so naively I would expect their score to be at least a bit higher than the white-collar worker average. On the other hand, these undergrads weren’t actually familiar with most of the software being tested - surely less familiar than GPT-4! And on the other-other hand I would expect an actual IT professional to do better. There is a difficult conflation with subject-matter knowledge and actual intelligence which makes “human-level” especially problematic. “CS undergrad-level” doesn’t roll off the tongue, but “human-level” seems downright misleading. Nevertheless, this is a striking result, indicating categorical weakness in out-of-distribution reasoning. And I think these tasks being so familiar - real human problem-solving where your ability to think makes up for gaps in knowledge - helps illustrate the problem with transformers: they’re just not smart enough. [1] I get very annoyed with discussions about “AGI” for systems that are totally incapable of solving problems that hunter-gatherers need to solve. But that’s another rant.
- apantel 2y ago
  Give it time. The technology is a year old.
- mhuffman 2y ago
  I am not an AGI stan, but it is clear to me that this gap will close substantially, and quickly ... like 6 months or so. The break-neck speed of development and (to me) the counterintuitive zeal with which open source developers are just donating tools and code that are going to hamstring them somewhat pretty soon is breathtaking. Think about how "most" companies use tech. Most companies are not big, they have a minimum set of tools to get what they need done. They currently, and routinely, have human mistakes. These are the types of tasks that agents trained by human demonstration can do soon. The only one that I suspect will be slow for adoption is bookkeeping, because you could theoretically end up on the wrong side of the IRS with mistakes.
abrichr 2y ago
Thank you for making this available! Check out https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt for a cross platform (Mac and Windows) open source library that learns to perform tasks in desktop apps by observing human demonstrations. We believe a major shortcoming with conventional approaches to AI agents is expecting them to be able to figure tasks of arbitrary complexity from first principles. While understandable from an academic perspective, this is unnecessary for practical utility, since humans perform these tasks constantly. With OpenAdapt you can demonstrate to a model how to perform a task, then have it take over the task, with additional user-supplied natural language instructions. I have created an issue to evaluate OpenAdapt on OSWorld: https://github.com/OpenAdaptAI/OpenAdapt/issues/642 https://github.com/OpenAdaptAI/OpenAdapt/issues/642. Contributions welcome! Edit: from https://github.com/xlang-ai/OSWorld/tree/main/evaluation_examples https://github.com/xlang-ai/OSWorld/tree/main/evaluation_exa...: > The ./trajectories file contains the annotated trajectories for each data item in ./examples for finishing the task. Unfortunately this file does not appear to be included in the repo. I have submitted an issue here: https://github.com/xlang-ai/OSWorld/issues/30 https://github.com/xlang-ai/OSWorld/issues/30
bitwize 2y ago
Coming soon: Human-trained AI that can actuate a robotic hand to fill in paper forms with a Selectric typewriter. The doom of us all!
rosslazer 2y ago
Dumb question - What actually needs to be done to close the gap?

77 points by kristianpaul 2y ago

I built a small Python script so I could let GPT-4 debug my system issues: https://github.com/skorokithakis/sysaidmin https://github.com/skorokithakis/sysaidmin It works surprisingly well!

ec109685 2y ago
What’s the most complicated thing it was able to debug?
- stavros 2y ago
  I had a library missing and it started using ldd to see where things were linked, and it changed the library path to the proper location, it was impressive.
  - bamboozled 2y ago
    Sounds like the state of your computer is the real problem here.
    - stavros 2y ago
      As opposed to?
      - bamboozled 2y ago
        https://nixos.org/ https://nixos.org/ But just in general, I've been a Linux pro for about 15 years, it's extremely rare I'd have to manually link anything?
        stavros 2y ago
        Well, when you need to run software that needs a library and your system doesn't have it because the library is too old, what do you do?

TheRoque 2y ago

Gotta love people working on replacing themselves. Jokes aside, seeing an AI interacting with a computer is kind of scary. It's not just outputting text anymore, it's doing the full work of a human working on a computer, meaning... a ton of people

mindcrime 2y ago
Gotta love people working on replacing themselves. So here's the thing: I want the computer to replace me... with regards to things that I don't want to do, or that aren't a good use of my time (boring, repetitive, trite, tedious, etc.). BUT the caveat is, I want it to be in a context where I'm in control of the computer and own the output, etc. Now companies and other actors also want the computer to replace me, but they want to be in control and to own the output. But here's the rub: they're not going to stop trying to come up with ways to replace me and minimize my value just because I don't like it. No man can stop the tide from coming in, or something like that (insert pithy saying here?). But if you know the tide is coming, you can retreat further inland OR build a boat, or climb a pole, etc. So where I'm at with this is, the more I work on AI, the more I know about AI, the more I can use it to enhance my own value outside of the context of corporate controlled IT systems, the more I can exercise a (possibly insignificant, but maybe not) degree of control over my fate. I can try to build my own boat, or at least outrun the tide for a little while. I don't know how things are going to turn out, and there's a reason that I am not in fact joking when I crack the occasional remark about "after this conversation I'm going to run to Bass Pro Shops and pick up a few more boxes of ammo" or whatever. Maybe a cyberpunk dystopia is what awaits us all. Maybe not. I'm just going to try to make my best guess at what the possible / likely future scenarios are and take whatever actions I can now to situate myself as favorably as I can.
- squigz 2y ago
  > "after this conversation I'm going to run to Bass Pro Shops and pick up a few more boxes of ammo" or whatever. Maybe a cyberpunk dystopia is what awaits us all. I don't mean this to sound accusatory, but do you think there's at all a connection between the probability of a dystopic future and the prevalence of people betting that that's the future?

ec109685 2y ago

Buried in their presentation is the current effectiveness of agents to complete desktop computing tasks. Humans are able to complete the tasks given at 70%+ effectiveness while the best model is at 12% (GPT4-v). Most of the other models were <5% effective.

nicklecompte 2y ago
I think this is a clear case where “human-level” is a totally meaningless expression, since you have to ask about which humans[1]. On one hand, these were CS undergrads, so naively I would expect their score to be at least a bit higher than the white-collar worker average. On the other hand, these undergrads weren’t actually familiar with most of the software being tested - surely less familiar than GPT-4! And on the other-other hand I would expect an actual IT professional to do better. There is a difficult conflation with subject-matter knowledge and actual intelligence which makes “human-level” especially problematic. “CS undergrad-level” doesn’t roll off the tongue, but “human-level” seems downright misleading. Nevertheless, this is a striking result, indicating categorical weakness in out-of-distribution reasoning. And I think these tasks being so familiar - real human problem-solving where your ability to think makes up for gaps in knowledge - helps illustrate the problem with transformers: they’re just not smart enough. [1] I get very annoyed with discussions about “AGI” for systems that are totally incapable of solving problems that hunter-gatherers need to solve. But that’s another rant.
apantel 2y ago
Give it time. The technology is a year old.
mhuffman 2y ago
I am not an AGI stan, but it is clear to me that this gap will close substantially, and quickly ... like 6 months or so. The break-neck speed of development and (to me) the counterintuitive zeal with which open source developers are just donating tools and code that are going to hamstring them somewhat pretty soon is breathtaking. Think about how "most" companies use tech. Most companies are not big, they have a minimum set of tools to get what they need done. They currently, and routinely, have human mistakes. These are the types of tasks that agents trained by human demonstration can do soon. The only one that I suspect will be slow for adoption is bookkeeping, because you could theoretically end up on the wrong side of the IRS with mistakes.

abrichr 2y ago

Thank you for making this available! Check out https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt for a cross platform (Mac and Windows) open source library that learns to perform tasks in desktop apps by observing human demonstrations. We believe a major shortcoming with conventional approaches to AI agents is expecting them to be able to figure tasks of arbitrary complexity from first principles. While understandable from an academic perspective, this is unnecessary for practical utility, since humans perform these tasks constantly. With OpenAdapt you can demonstrate to a model how to perform a task, then have it take over the task, with additional user-supplied natural language instructions. I have created an issue to evaluate OpenAdapt on OSWorld: https://github.com/OpenAdaptAI/OpenAdapt/issues/642 https://github.com/OpenAdaptAI/OpenAdapt/issues/642. Contributions welcome! Edit: from https://github.com/xlang-ai/OSWorld/tree/main/evaluation_examples https://github.com/xlang-ai/OSWorld/tree/main/evaluation_exa...: > The ./trajectories file contains the annotated trajectories for each data item in ./examples for finishing the task. Unfortunately this file does not appear to be included in the repo. I have submitted an issue here: https://github.com/xlang-ai/OSWorld/issues/30 https://github.com/xlang-ai/OSWorld/issues/30

bitwize 2y ago

Coming soon: Human-trained AI that can actuate a robotic hand to fill in paper forms with a Selectric typewriter. The doom of us all!

rosslazer 2y ago

Dumb question - What actually needs to be done to close the gap?