6 ms·
OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something ap
by vessenes 1y ago
OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.
- almosthere 1y agoIts even cooler if humans find something to be excited about in this world, since AI is replacing everything we do.
- Fuzzwah 1y agoYou do realise the things that AI are replacing are very new things humans have only been doing for the last ~50 years?
- Workaccount2 1y agoI desperately want an AI agent that can use my phone for me. Just something that takes instructs for each screen and execute it. "Open Chrome" "Go to xyz.com" "open hamburger menu" "Click login" etc. etc.
- tantalor 1y agoIsn't that what the voice a11y tools have been doing for years. Why do you need AI for that. https://support.google.com/accessibility/android/answer/6151848 https://support.google.com/accessibility/android/answer/6151... https://support.apple.com/en-us/111778 https://support.apple.com/en-us/111778
- bckr 1y agoMy friend that is AI. However, it can get a lot better: be more aware of screen content, follow multiple instructions at once, keep context in mind throughout the conversation and from past interactions
- luciferin 1y agoAI commonly means LLM. Where are you determining this is using a LLM for proccessing?
- mastazi 1y agoAI has existed for several decades before the first LLM was ever created.[1][2][3] And that's not even considering machine learning and deep learning which also have existed for many years before LLMs. Even if you consider the current usage of the word AI in popular culture, it includes things that are not an LLM like Stable Diffusion and Suno [1] https://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system [2] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) [3] https://en.wikipedia.org/wiki/Lisp_machine#Historical_context https://en.wikipedia.org/wiki/Lisp_machine#Historical_contex...
- exe34 1y agoYou must be new around here, kid. AI has been there since the birth of the programmable computer.
- dyauspitr 1y agoThat’s all programmatic, you can’t throw curve balls at it.
- onion2k 1y agoDroidrun did a Show HN recently. It's exactly that.
- qingcharles 1y agoI have this in my bookmarks: https://dafdef.com/aikey https://dafdef.com/aikey
- Stevvo 1y agoIs it though? A machine could play a game frame-by-frame.
- mxkopy 1y agoPython dxcam + windows hook API listening for HID messages
- amelius 1y agoControlling the mouse?
- ksh09 1y agoWhen, manus.ai came out I wondered the same, the "use computer" mode seemed really interesting to me, although I've seen JAYU [1] which implemented computer use with gemini. Moreover I saw somewhere I really don't remember navigating web browser through layouts akin vimium like experience. Initially (my impression of) computer use was only opening chrome and doing things inside chrome, chrome-as-os experience, I think maybe cloudflare could do something better here, with their workers? Per user instance seems really costly I really do wonder how did they architect it. [1] https://ai.google.dev/competition/projects/jayu https://ai.google.dev/competition/projects/jayu