I have been working on a similar space for the last 8 months. Creating a way for the AI use your OS. There are many projects out there trying to achieve this: O

I have been working on a similar space for the last 8 months. Creating a way for the AI use your OS. There are many projects out there trying to achieve this: Open Interpreter, OpenAdapt, etc. The thing that is hard to achieve is not to get your LLM to operate your computer but to make it do the right thing. And by right thing I mean, the thing you expect it to do in the way you want. Example I always show to people when I give a demo of what I built: Say you want to buy a new Macbook. If you ask the computer to buy you one, it might: A) Go to Google and search for a Macbook B) Go to Apple and search for a Macbook C) Go to Amazon and search for a Macbook Now, depending on your implementation it might then ask you to give you more details about the Macbook you need (pro? air? how big?) The tough part is to wrap all this together in a way that doesn't frustrate the user. In a way it's like when you do a Google search: the first result is the best guess of what you want. But how likely is it that it's not? Pretty much 50% of the time you look at the top 2-3 results right? Well it's the same with having the AI control your computer, at this point in time, you want to have "multiple sessions" - sort of like a parallel universe where the AI has done 2-3 things and lets you pick from A, B ... or C and then go from there. Another approach would be to collate a library of most used "prompts" - i.e. if you are trying to buy a laptop and someone else managed to get the AI to go through a very thorough process where you're asked all the important details that lead to purchasing the Macbook, then you should be able to re-use that one workflow from the library, rather than have the AI start from scratch. Anyway, it's very very tough - more so from a UX experience rather than an AI perspective.