3 ms·

Giving AI access to OS APIs is trivial for Apple and Google. The problem has been that AI assistant were always too dumb to understand user requests. LLMs cha

Giving AI access to OS APIs is trivial for Apple and Google. The problem has been that AI assistant were always too dumb to understand user requests. LLMs changed everything.

troupo 3y ago
First, it's not as trivial to give access to OS APIs as you think. Second, giving access to APIs... gives you access to APIs, and that's it. Even the most advanced LLMs are dumb as shit when it comes to anything non-trivial in programming. What makes you think that a modern LLM will manage to translate "move tab X to a different window" into a dozen or more API calls with proper structures filled in, and in the correct order?
- aurareturn 3y ago
  It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.
  - themoonisachees 3y ago
    You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.
  - troupo 3y ago
    > It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
    - everforward 3y ago
      > BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
      - troupo 3y ago
        In theory, yes. Someone still needs to compile them and match them to functions etc.
    - aurareturn 3y ago
      Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
      - troupo 3y ago
        > Mapping JSON responses to OS APIs is the easiest part. Please show me how you will do that for the tens of thousands of OS APIs and data structures. Edit: because it's not just "a function call" is it? It's often: struct1 = set up/get hold of a complex struct 1 struct 2 = set up/get hold of a complex struct 2 struct 3 = set up/get hold of a complex struct 3 using one or both of the previous ones call some specific function 1 call some specific function 2 using some or all of the structs above free the structs above, often in a specific order So the question becomes how can this: { "function_name": "X", parameters: [...] } be easily converted to all that? Don't forget about failure modes. Where you have to check for the validity of some but not all structs passed around. Repeat that for any combination of any of the 10k+ APIs and structures
        aurareturn 3y ago
        If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Instead, you should focus on what Apple and Google would do internally to make things as easy for an LLM as possible. Up until now, the hardest part was always understanding what exactly users want. GPT4 is a thousand times better than what Siri and Google Assistant are currently doing. Again, mapping OS APIs is the easiest part. By far.
datadrivenangel 3y ago
The problem is that Google walked back their integrations for anti-trust reasons, and Apple is now being sued by the US government for having too many apple only integrations. AI assistants only barely had enough data to understand your requests, and now have even less.

Giving AI access to OS APIs is trivial for Apple and Google. The problem has been that AI assistant were always too dumb to understand user requests. LLMs changed everything.

troupo 3y ago

First, it's not as trivial to give access to OS APIs as you think. Second, giving access to APIs... gives you access to APIs, and that's it. Even the most advanced LLMs are dumb as shit when it comes to anything non-trivial in programming. What makes you think that a modern LLM will manage to translate "move tab X to a different window" into a dozen or more API calls with proper structures filled in, and in the correct order?

aurareturn 3y ago
It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.
- themoonisachees 3y ago
  You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.
- troupo 3y ago
  > It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
  - everforward 3y ago
    > BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
    - troupo 3y ago
      In theory, yes. Someone still needs to compile them and match them to functions etc.
  - aurareturn 3y ago
    Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
    - troupo 3y ago
      > Mapping JSON responses to OS APIs is the easiest part. Please show me how you will do that for the tens of thousands of OS APIs and data structures. Edit: because it's not just "a function call" is it? It's often: struct1 = set up/get hold of a complex struct 1 struct 2 = set up/get hold of a complex struct 2 struct 3 = set up/get hold of a complex struct 3 using one or both of the previous ones call some specific function 1 call some specific function 2 using some or all of the structs above free the structs above, often in a specific order So the question becomes how can this: { "function_name": "X", parameters: [...] } be easily converted to all that? Don't forget about failure modes. Where you have to check for the validity of some but not all structs passed around. Repeat that for any combination of any of the 10k+ APIs and structures
      - aurareturn 3y ago
        If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Instead, you should focus on what Apple and Google would do internally to make things as easy for an LLM as possible. Up until now, the hardest part was always understanding what exactly users want. GPT4 is a thousand times better than what Siri and Google Assistant are currently doing. Again, mapping OS APIs is the easiest part. By far.

datadrivenangel 3y ago

The problem is that Google walked back their integrations for anti-trust reasons, and Apple is now being sued by the US government for having too many apple only integrations. AI assistants only barely had enough data to understand your requests, and now have even less.