6 ms·
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
- lokimedes 3y agoSeemingly beating GPT-4 at GAIA.
- hackerlight 3y agoThey use GPT-4.
- nomel 3y agoYes, their use is beating raw GPT-4.
- WiSaGaN 3y agoThis is not an LLM, so it doesn't make much sense to compare it with one. A more appropriate comparison would be something along the lines of GPT-4 OS-Copilot vs. GPT-4 CoT or vs. GPT-4 zero-shot.
- abeppu 3y agoAt multiple points in the paper, they describe the agent as "embodied", but it seems like they don't really justify their use of that term. Embodiment is a loaded word with a lot of associations in neighboring fields. I think it's a little wild that they're claiming this aspect in their abstract and conclusion and not really touching on it substantively in the body of the paper.
- xpe 3y agoWhat fields do you think use the term relatively clearly?
- cdogl 3y agoCognitive science. “The Embodied Mind”[1] defines embodiment quite clearly and argues for its centrality to consciousness. https://mitpress.mit.edu/9780262220422/ https://mitpress.mit.edu/9780262220422/
- abeppu 3y agoBut not just "consciousness" which has a bunch of problems of poor definition and epistemic inaccessibility, but "cognition" more broadly. In this "OS" situation, aside from not even simulating a virtual body, it's not clear that the inputs available to the agent are especially tied to the agent's ongoing actions. By comparison, what you see is a function of how you move your body, head, eyes and eyelids, and "seeing" involves saccades that let you take in the multiple important parts of a scene. So I think this doesn't even have an instantiation which acts _analogously_ to being embodied.
- naasking 3y agoOff the cuff, I think any system that has a perceptual connection to an "environment" separate from itself and can induce changes in that environment can be generally considered "embodied". This ability to cognitively distinguish self from environment and to understand one's effect on that environment are the crucial elements. I think an explicitly spatial notion of embodiment typical of humans, is insufficiently general.
- dimatura 3y agoAs someone in CV/ML with a background in cognitive science, and partial to the work of Lakoff and others in regards to the role of embodiment, it does irk me how loose papers in vision can be with the term. In this case specifically, it's rather hard to justify. I'm guessing it may be influenced by other papers in vision that use agents in a simulated 3D environment. In those papers saying that there is some kind of embodiment is iffy, but depending on the quality of the simulation, maybe defensible.
- haswell 3y agoIt's going to be really interesting to see how work like this impacts the Robotic Process Automation or "RPA" product space. Many of the enterprise products in this category have used computer vision to help reduce brittleness, but for all of the improvements, these tools have remained highly error prone, not to mention extremely expensive. The explosion of interest in the category across a broader community of researchers and developers seems like both a boon for the RPA space, and a major threat of disruption.
- grugagag 3y agoAdd another layer of AI on top of RPA and the mess will be harder to or impossible to ever untangle. If it works with high accuracy it will still be a great thing but a complete blackbox and expensive one to maintain too.
- ukuina 3y agoWon't this replace the entire RPA layer rather than sit atop? I would never have considered RPA, but I would build my own layer via LLM for my automation needs.
- grugagag 3y agoWhen does a layer get removed, ever??
- haswell 3y agoI suspect they’ll co-exist, but I don’t think full replacement is likely in many cases. Existing RPA products will continue to exist if for no other reason that they’re already deeply embedded in many environments. Over time, those same products will just get easier to use. One of the core features of RPA products is centralized visibility and management of the lifecycle of automations. This will remain even if the entire interaction layer changes. Something like the OS-Copilot layer lowers the barrier to entry though and I could easily see there being multiple choices for orgs looking to do this kind of automation in the future, and this is where I see the opportunity for disruption. Most likely the RPA products will add all of the necessary buzzwords to keep people in buying roles interested and more likely to keep renewing.
- marojejian 3y agoInteresting. But gut reaction is that the main use for this is impersonating a human, evading captchas etc.
- dannyw 3y agoYou can hire human captcha farms for sub pennies already. Systems like reCATCHA are behaviour / profile-based*; the actual captcha bit is almost a stagescreen. *At certain thresholds, even solving the captcha correctly will falsely claim it’s wrong. Particularly if you’re not logged into a Google account.
- tudorw 3y agoIn my wildest speculation behind the gossip of recent goings on I imagine a board comprised of individuals saturated in the technology, presented with the latest technology, a leading edge AI, tasked with improving it's own code, as the board and co. sat back and watched, the machine made improvements rapidly, rapidly enough that the engineers had to admit they could no longer follow the code changes, watched as the AI redeveloped the underlying language then kernel and started planning fabrication technology and how to fund it, mere trillions, the benchmarks were smashed and they hit stop, what do you do next?
- bamboozled 3y agoKeep it going because your company might be the next unicorn if you do ?
- tudorw 3y agoI recall reading a roadmap for camera sensors, this might be folklore it was a long time ago, the takeaway was that they had the technology for the 100mp sensor and the roadmap showed how they would maximise profit using a 20 year curve working up to the release of the final sensor, by which time a new method would meet that curve to continue their dominance, if I had AGI, I would let it out in very small drips.
- bamboozled 3y agoToo much fame, money and notoriety at stake, imagine the CEO of OpenAI having 20 years of patience ?
- nirav72 3y agoHe’s a relatively young guy that obviously lives well enough right now to be able wait 20 years.
- bamboozled 3y ago
- troymc 3y agoWhen I first heard the term "Windows Copilot", I assumed it was something like this, but so far it seems way less useful.
- Aerbil313 3y agoThis is the endgame that’ll put many out of their jobs. Not that it’s bad. I personally think humanity will unfortunately need greater and greater efficiency in the coming decades.
- causality0 3y agoI hope it's better at interacting with the OS than ChatGPT is. If I ask ChatGPT to write me a powershell/batch script to do something it works about 25% of the time.
- tatersolid 3y agoChatGPT and Copilot both frequently hallucinate entire plausible-sounding APIs or properties/methods that simply don’t exist. Often this takes a lot of time to debug/resolve. They also often omit proper input validation, error handling, and logging even when prompted to include it. I want to do a structured efficiency study of programming tasks of human vs human-plus-AI from problem statement through production-ready code. But my org doesn’t have enough devs to make it statistically significant, nor the spare human capacity to invest on duplicative tasks. I assume there must be some studies out there anyone have a reference?
- factormeta 3y agoWonder if it can tain Automator in Mac? https://support.apple.com/guide/automator/welcome/mac https://support.apple.com/guide/automator/welcome/mac
- rationalfaith 3y ago[dead]
- deleted 3y ago[deleted]
- anonzzzies 3y agoI asked this when UFO was trending but does this work? I cannot try more of them. I want this, but so far everything is so terrible. So who tried it for non trivial/cherry picked stuff?
- dartharva 3y agoMicrosoft's UFO yesterday, and this today. Just beautiful. I am eagerly looking forward to getting the chance to use these once matured and offload all my gruntwork.