6 ms·

R1 Computer Use

104 points by mountainriver 2y ago

Hey HN, We are working to apply the ideas of R1 to computer use. The primary struggle is creating reliable neural reward models since hard-verification rewards are not available at scale in GUI interactions. Our team is currently deep in the weeds of collecting reasoning annotation data for GUI interfaces to train a reliable reward model. We would love all thoughts, feedback, and collaborations!

mountainriver 2y ago
Hey HN, We are working to apply the ideas of R1 to computer use. The primary struggle is creating reliable neural reward models since hard-verification rewards are not available at scale in GUI interactions. Our team is currently deep in the weeds of collecting reasoning annotation data for GUI interfaces to train a reliable reward model. We would love all thoughts, feedback, and collaborations!
llama-mini 2y ago
It seems a placeholder for now? No content? Right?
- mountainriver 2y ago
  We are in the process of collecting the data right now which is fairly involved, we are going to be opening up that platform for others as well shortly.
  - throwaway314155 2y ago
    It's not my intention to be dismissive because the project idea seems really cool. I'm just curious, why not wait until the source code is ready to post it on HN?
    - mountainriver 2y ago
      Because gauging interest early and finding other people interested in building is a good idea and frankly very inline with YC thinking. We have already open sourced an enormous amount of code and datasets for computer use https://github.com/agentsea https://github.com/agentsea
      - throwaway314155 2y ago
        Fair enough. Sorry if it felt like I was piling on.
        mountainriver 2y ago
        No worries! Its fair criticism
- dimitry12 2y ago
  No content, no code. "Roadmap" and "Training pipeline" in README are summaries of Section 2.3 of "DeepSeek-R1"-paper. Sad.
  - Redoubts 2y ago
    Many such cases
  - mountainriver 2y ago
    Sharing ideas early is not a bad thing and very much encouraged by YC, we are gauging interest in collaboration on the topic. Our company has open sourced almost our entire compute use stack already https://github.com/agentsea https://github.com/agentsea
falcor84 2y ago
> @software{r1_computer_use, title = {R1-Computer-Use: Reasoning-First Computer Interaction}, author = {Barker, Patrick}, year = {2025}, url = {https://github.com/agentsea/r1-computer-use https://github.com/agentsea/r1-computer-use}, } Sorry to be a party-pooper, but does it really make sense to add a citation when you don't have fully working code yet, let alone a paper about it?
- mountainriver 2y ago
  Fair, in the process of opening more of it up, so this seems a bit odd right now
- soheil 2y ago
  And who is "we"? Is he using the royal we? "We aim to replace hard-coded.."
  - sva_ 2y ago
    Probably their company "AgentSea"?
fkyoureadthedoc 2y ago
This is the type of post some VP at my company sees and starts telling people that R1 can use computer and then I have to be like "well actually" to 25 people. Computer use is pretty exciting stuff in general though, good luck
- mountainriver 2y ago
  Ha! Sorry about that!
refulgentis 2y ago
Free advice (though, worth less than free, because A) it's unsolicited B) it's saying "don't do it") TL;DR: - Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons) - You're not going to be able out-do model companies on building models, they have too much funds. - Try writing GUI-based integration tests. Then imagine an LLM, miraculously, always chooses the right route. Does the UX look good? - Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. - Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools. CLI, though....
- mountainriver 2y ago
  > Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons) I would never buy plane tickets that way, we built it because there are tons of things we couldn't automate and this was the only way to do it > - You're not going to be able out-do model companies on building models, they have too much funds. There are plenty of people edging out the model companies all over the place. We aren't powerless to them > Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools. The reason why they are bad at tool calling is they aren't trained on it. The current reasoning models require hard-verification reward models, we don't have those for tool calling, whereas that data is easy to get for math/code. Reasoning will improve tool calling, OpenAI just talked about it recently as being the answer to autonomous agents
crazygringo 2y ago
I can't wait for something like this to be built. People have tons of workflows that involve a lot of clicks and typing in response to data that are too difficult or one-off to automate with fragile macros. But if my computer can quickly realize that I'm deleting every odd-numbered page of a PDF, or renaming every file to add a prefix, or following each link on a website and saving an image... and then just instantly automate the next 100 times... that's going to be huge!
- deleted 2y ago
  [deleted]
- soheil 2y ago
  Check out AutoGPT you don't have to wait, it's already built. https://github.com/Significant-Gravitas/AutoGPT https://github.com/Significant-Gravitas/AutoGPT
  - ai-christianson 2y ago
    Does this do "Computer Use" in that it looks at the screen, controls the mouse, keyboard (e.g. how Anthropic computer use does?)
  - Philpax 2y ago
    AutoGPT is an unserious project.
  - crazygringo 2y ago
    I don't think that's what I'm describing. That's about manually setting up agents, that run on a server, that seem to interact largely with the web (from the examples). I'm talking about not manually setting up anything -- I'm talking about an AI that simply observes the repetitive actions you're taking on your computer, infers patterns from them, and then offers to take over and finish the job.
    - mcny 2y ago
      As a start, I want to see if the agent can figure out how I play a clicker style game such as adventure capitalist on the computer. I am thinking I have a certain style of playing. I still don't understand how an agent can somehow figure out valid gameplay (earth, moon, mars, events) AND figure out a valid gameplay much less play the game in my own style. I think we should start with something simple, repeatable, and does little to no harm if/when things go wrong. Edit: repetitive -> repeatable
iiJDSii 2y ago
What does your perception look like, are you using raw screenshots? GUI snapshots? Vision is very difficult for these, and snapshots are incomplete, is what I've found in some earlier experiments.
- mountainriver 2y ago
  Perception is just 1-2 screenshots. A number of recent VLM models have a lot more pretraining data on GUI interactions, which helps.
  - iiJDSii 2y ago
    Such as? Are they able to recognize arbitrary GUI elements from various desktop programs, web browsers, etc?
    - mountainriver 2y ago
      Qwen2.5-vl seems to be the best right now by our tests. UI-TARS by bytedance also has a good amount of pretraining. Molmo is also very good at coordinates.
mkagenius 2y ago
Training a base model just for computer use seems like an overkill as normal reasoning model like o3 for planning + a vision model like gemini-flash is good enough[1] without being trained specifically for computer use. But if you still want to try out this path, Google has made the screenQA dataset(rico) available[2] along with bounding boxes. 1. A framework to use local/hosted models for android use/control - https://github.com/BandarLabs/clickclickclick https://github.com/BandarLabs/clickclickclick 2. https://github.com/google-research-datasets/screen_qa https://github.com/google-research-datasets/screen_qa
- mountainriver 2y ago
  They aren't good enough to be reliable, and these agents aren't much use without being reliable. We initially started that way, and tried just about everything before focusing on training a base model.
emregucerr 2y ago
i wonder how good is R1 at counting pixels from a screenshot. what enabled claude and OAI's CUA to develop computer use was being able to precisely give x-y coordinates of a click location. also, how big of a gain to have reasoning for computer use? i feel like reasoning unlocks a lot when there is a single complex question but not so much better at taking actions in a long term plan.
- mountainriver 2y ago
  Yep, coordinate grounding is key, we use Ai2's pixmo for a lot of that https://huggingface.co/datasets/allenai/pixmo-points https://huggingface.co/datasets/allenai/pixmo-points We had previously created https://huggingface.co/datasets/agentsea/wave-ui https://huggingface.co/datasets/agentsea/wave-ui but that was superseded by pixmo as it contains over a million data points.
3s 2y ago
Are people concerned about the privacy implications of computer use at all? This is why I haven’t been using Claude computer use personally. Somehow the idea of sending everything I do on my computer to a random third party seems creepy. There are a lot of applications of AI (rewind comes to mind) that I simply cannot accept the idea of sharing my screen with
- burky 2y ago
  I was thinking it would just use a local copy through LM Studio or Ollama. Wouldn’t that work?
- mountainriver 2y ago
  Agree! Its a bridge too far for me as well, we hope to offer the ecosystem private alternatives.
- stainablesteel 2y ago
  yeah i have no interest in having an AI control anything on my computer, i would just make a script for that
- sva_ 2y ago
  It does not appear that this aims to use an API, but rather to train/finetune their own model using insights from R1 ([mostly] pure RL approach towards bootstrapping reasoning in an LLM)
- n144q 2y ago
  * you can always choose to self a small model, although it probably doesn't work as well * it's not a "random third party". You know to whom the data is being sent, and at least according to service agreements, most services don't use your data for training. If you don't trust Claude, you could trust AWS hosted version, or GPT/Deepseek hosted on Azure. Well, if you think Amazon/Microsoft is not trustworthy and they may misuse your data in these cloud services (not some random consumer facing service where you are the product), you might as well give up your digital life.
  - 2y ago

R1 Computer Use

104 points by mountainriver 2y ago

mountainriver 2y ago

llama-mini 2y ago

It seems a placeholder for now? No content? Right?

mountainriver 2y ago
We are in the process of collecting the data right now which is fairly involved, we are going to be opening up that platform for others as well shortly.
- throwaway314155 2y ago
  It's not my intention to be dismissive because the project idea seems really cool. I'm just curious, why not wait until the source code is ready to post it on HN?
  - mountainriver 2y ago
    Because gauging interest early and finding other people interested in building is a good idea and frankly very inline with YC thinking. We have already open sourced an enormous amount of code and datasets for computer use https://github.com/agentsea https://github.com/agentsea
    - throwaway314155 2y ago
      Fair enough. Sorry if it felt like I was piling on.
      - mountainriver 2y ago
        No worries! Its fair criticism
dimitry12 2y ago
No content, no code. "Roadmap" and "Training pipeline" in README are summaries of Section 2.3 of "DeepSeek-R1"-paper. Sad.
- Redoubts 2y ago
  Many such cases
- mountainriver 2y ago
  Sharing ideas early is not a bad thing and very much encouraged by YC, we are gauging interest in collaboration on the topic. Our company has open sourced almost our entire compute use stack already https://github.com/agentsea https://github.com/agentsea

falcor84 2y ago

> @software{r1_computer_use, title = {R1-Computer-Use: Reasoning-First Computer Interaction}, author = {Barker, Patrick}, year = {2025}, url = {https://github.com/agentsea/r1-computer-use https://github.com/agentsea/r1-computer-use}, } Sorry to be a party-pooper, but does it really make sense to add a citation when you don't have fully working code yet, let alone a paper about it?

mountainriver 2y ago
Fair, in the process of opening more of it up, so this seems a bit odd right now
soheil 2y ago
And who is "we"? Is he using the royal we? "We aim to replace hard-coded.."
- sva_ 2y ago
  Probably their company "AgentSea"?

fkyoureadthedoc 2y ago

This is the type of post some VP at my company sees and starts telling people that R1 can use computer and then I have to be like "well actually" to 25 people. Computer use is pretty exciting stuff in general though, good luck

mountainriver 2y ago
Ha! Sorry about that!

refulgentis 2y ago

Free advice (though, worth less than free, because A) it's unsolicited B) it's saying "don't do it") TL;DR: - Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons) - You're not going to be able out-do model companies on building models, they have too much funds. - Try writing GUI-based integration tests. Then imagine an LLM, miraculously, always chooses the right route. Does the UX look good? - Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. - Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools. CLI, though....

mountainriver 2y ago
> Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons) I would never buy plane tickets that way, we built it because there are tons of things we couldn't automate and this was the only way to do it > - You're not going to be able out-do model companies on building models, they have too much funds. There are plenty of people edging out the model companies all over the place. We aren't powerless to them > Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools. The reason why they are bad at tool calling is they aren't trained on it. The current reasoning models require hard-verification reward models, we don't have those for tool calling, whereas that data is easy to get for math/code. Reasoning will improve tool calling, OpenAI just talked about it recently as being the answer to autonomous agents

crazygringo 2y ago

I can't wait for something like this to be built. People have tons of workflows that involve a lot of clicks and typing in response to data that are too difficult or one-off to automate with fragile macros. But if my computer can quickly realize that I'm deleting every odd-numbered page of a PDF, or renaming every file to add a prefix, or following each link on a website and saving an image... and then just instantly automate the next 100 times... that's going to be huge!

deleted 2y ago
[deleted]
soheil 2y ago
Check out AutoGPT you don't have to wait, it's already built. https://github.com/Significant-Gravitas/AutoGPT https://github.com/Significant-Gravitas/AutoGPT
- ai-christianson 2y ago
  Does this do "Computer Use" in that it looks at the screen, controls the mouse, keyboard (e.g. how Anthropic computer use does?)
- Philpax 2y ago
  AutoGPT is an unserious project.
- crazygringo 2y ago
  I don't think that's what I'm describing. That's about manually setting up agents, that run on a server, that seem to interact largely with the web (from the examples). I'm talking about not manually setting up anything -- I'm talking about an AI that simply observes the repetitive actions you're taking on your computer, infers patterns from them, and then offers to take over and finish the job.
  - mcny 2y ago
    As a start, I want to see if the agent can figure out how I play a clicker style game such as adventure capitalist on the computer. I am thinking I have a certain style of playing. I still don't understand how an agent can somehow figure out valid gameplay (earth, moon, mars, events) AND figure out a valid gameplay much less play the game in my own style. I think we should start with something simple, repeatable, and does little to no harm if/when things go wrong. Edit: repetitive -> repeatable

iiJDSii 2y ago

What does your perception look like, are you using raw screenshots? GUI snapshots? Vision is very difficult for these, and snapshots are incomplete, is what I've found in some earlier experiments.

mountainriver 2y ago
Perception is just 1-2 screenshots. A number of recent VLM models have a lot more pretraining data on GUI interactions, which helps.
- iiJDSii 2y ago
  Such as? Are they able to recognize arbitrary GUI elements from various desktop programs, web browsers, etc?
  - mountainriver 2y ago
    Qwen2.5-vl seems to be the best right now by our tests. UI-TARS by bytedance also has a good amount of pretraining. Molmo is also very good at coordinates.

mkagenius 2y ago

Training a base model just for computer use seems like an overkill as normal reasoning model like o3 for planning + a vision model like gemini-flash is good enough[1] without being trained specifically for computer use. But if you still want to try out this path, Google has made the screenQA dataset(rico) available[2] along with bounding boxes. 1. A framework to use local/hosted models for android use/control - https://github.com/BandarLabs/clickclickclick https://github.com/BandarLabs/clickclickclick 2. https://github.com/google-research-datasets/screen_qa https://github.com/google-research-datasets/screen_qa

mountainriver 2y ago
They aren't good enough to be reliable, and these agents aren't much use without being reliable. We initially started that way, and tried just about everything before focusing on training a base model.

emregucerr 2y ago

i wonder how good is R1 at counting pixels from a screenshot. what enabled claude and OAI's CUA to develop computer use was being able to precisely give x-y coordinates of a click location. also, how big of a gain to have reasoning for computer use? i feel like reasoning unlocks a lot when there is a single complex question but not so much better at taking actions in a long term plan.

mountainriver 2y ago
Yep, coordinate grounding is key, we use Ai2's pixmo for a lot of that https://huggingface.co/datasets/allenai/pixmo-points https://huggingface.co/datasets/allenai/pixmo-points We had previously created https://huggingface.co/datasets/agentsea/wave-ui https://huggingface.co/datasets/agentsea/wave-ui but that was superseded by pixmo as it contains over a million data points.

3s 2y ago

Are people concerned about the privacy implications of computer use at all? This is why I haven’t been using Claude computer use personally. Somehow the idea of sending everything I do on my computer to a random third party seems creepy. There are a lot of applications of AI (rewind comes to mind) that I simply cannot accept the idea of sharing my screen with

burky 2y ago
I was thinking it would just use a local copy through LM Studio or Ollama. Wouldn’t that work?
mountainriver 2y ago
Agree! Its a bridge too far for me as well, we hope to offer the ecosystem private alternatives.
stainablesteel 2y ago
yeah i have no interest in having an AI control anything on my computer, i would just make a script for that
sva_ 2y ago
It does not appear that this aims to use an API, but rather to train/finetune their own model using insights from R1 ([mostly] pure RL approach towards bootstrapping reasoning in an LLM)
n144q 2y ago
* you can always choose to self a small model, although it probably doesn't work as well * it's not a "random third party". You know to whom the data is being sent, and at least according to service agreements, most services don't use your data for training. If you don't trust Claude, you could trust AWS hosted version, or GPT/Deepseek hosted on Azure. Well, if you think Amazon/Microsoft is not trustworthy and they may misuse your data in these cloud services (not some random consumer facing service where you are the product), you might as well give up your digital life.
- 2y ago