3 ms·

It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT

It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.

themoonisachees 3y ago
You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.
troupo 3y ago
> It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
- everforward 3y ago
  > BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
  - troupo 3y ago
    In theory, yes. Someone still needs to compile them and match them to functions etc.
- aurareturn 3y ago
  Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
  - troupo 3y ago
    > Mapping JSON responses to OS APIs is the easiest part. Please show me how you will do that for the tens of thousands of OS APIs and data structures. Edit: because it's not just "a function call" is it? It's often: struct1 = set up/get hold of a complex struct 1 struct 2 = set up/get hold of a complex struct 2 struct 3 = set up/get hold of a complex struct 3 using one or both of the previous ones call some specific function 1 call some specific function 2 using some or all of the structs above free the structs above, often in a specific order So the question becomes how can this: { "function_name": "X", parameters: [...] } be easily converted to all that? Don't forget about failure modes. Where you have to check for the validity of some but not all structs passed around. Repeat that for any combination of any of the 10k+ APIs and structures
    - aurareturn 3y ago
      If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Instead, you should focus on what Apple and Google would do internally to make things as easy for an LLM as possible. Up until now, the hardest part was always understanding what exactly users want. GPT4 is a thousand times better than what Siri and Google Assistant are currently doing. Again, mapping OS APIs is the easiest part. By far.
      - troupo 3y ago
        > If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. You keep skipping the question of how. > It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Somehow you have this mystical magical idea of "oh, it's just this small insignificant little thing". Literally this: LLMs understand human requests * magic * Things happen in the OS You, yes you already have basically the same access as developers of the OS have. You already have access to tens of thousands of OS APIs and to the LLMs. And yet we haven't seen a single implementation that does what you want. > Again, mapping OS APIs is the easiest part. By far. If it is, it would make it trivially easy how to do this trivial and easy task for a small subset of those APIs, wouldn't it? Can you show me how you would do it?

themoonisachees 3y ago

You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.

troupo 3y ago

> It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.

everforward 3y ago
> BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
- troupo 3y ago
  In theory, yes. Someone still needs to compile them and match them to functions etc.
aurareturn 3y ago
Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
- troupo 3y ago
  > Mapping JSON responses to OS APIs is the easiest part. Please show me how you will do that for the tens of thousands of OS APIs and data structures. Edit: because it's not just "a function call" is it? It's often: struct1 = set up/get hold of a complex struct 1 struct 2 = set up/get hold of a complex struct 2 struct 3 = set up/get hold of a complex struct 3 using one or both of the previous ones call some specific function 1 call some specific function 2 using some or all of the structs above free the structs above, often in a specific order So the question becomes how can this: { "function_name": "X", parameters: [...] } be easily converted to all that? Don't forget about failure modes. Where you have to check for the validity of some but not all structs passed around. Repeat that for any combination of any of the 10k+ APIs and structures
  - aurareturn 3y ago
    If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Instead, you should focus on what Apple and Google would do internally to make things as easy for an LLM as possible. Up until now, the hardest part was always understanding what exactly users want. GPT4 is a thousand times better than what Siri and Google Assistant are currently doing. Again, mapping OS APIs is the easiest part. By far.
    - troupo 3y ago
      > If Apple and Google wants to give access to the OS to an LLM such as Siri, they'd make it easy. You keep skipping the question of how. > It's the trivial part of all this. You're getting too much into the details of what is available to developers today. Somehow you have this mystical magical idea of "oh, it's just this small insignificant little thing". Literally this: LLMs understand human requests * magic * Things happen in the OS You, yes you already have basically the same access as developers of the OS have. You already have access to tens of thousands of OS APIs and to the LLMs. And yet we haven't seen a single implementation that does what you want. > Again, mapping OS APIs is the easiest part. By far. If it is, it would make it trivially easy how to do this trivial and easy task for a small subset of those APIs, wouldn't it? Can you show me how you would do it?