5 ms·

Seeing the fast development of AI in the last couple of years, it's incredible how stuck AI assistants are. Basic things like "Hey Google, turn up the volume" d

Seeing the fast development of AI in the last couple of years, it's incredible how stuck AI assistants are. Basic things like "Hey Google, turn up the volume" don't work. Just imagine what a good AI assistant could do with todays LLM tech. Actual things I would have used in the last two days: - Hey Google, what's the song that goes "da da da da da daaa da"? There are skeletons and elephants in the video? - Hey Siri, can you move all the windows of project XY to a fresh desktop? - Move all the shopping tabs to a new window. - Find the last contract draft for customer ABC.

troupo 2y ago
Many of those have nothing to do with AI, and everything to do with access points and integrations in applications and OS. But for everything else they do suck.
- aurareturn 2y ago
  Giving AI access to OS APIs is trivial for Apple and Google. The problem has been that AI assistant were always too dumb to understand user requests. LLMs changed everything.
  - troupo 2y ago
    First, it's not as trivial to give access to OS APIs as you think. Second, giving access to APIs... gives you access to APIs, and that's it. Even the most advanced LLMs are dumb as shit when it comes to anything non-trivial in programming. What makes you think that a modern LLM will manage to translate "move tab X to a different window" into a dozen or more API calls with proper structures filled in, and in the correct order?
    - aurareturn 2y ago
      It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.
      - themoonisachees 2y ago
        You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.
      - troupo 2y ago
        > It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
        everforward 2y ago
        > BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
        troupo 2y ago
        In theory, yes. Someone still needs to compile them and match them to functions etc.
        aurareturn 2y ago
        Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
  - datadrivenangel 2y ago
    The problem is that Google walked back their integrations for anti-trust reasons, and Apple is now being sued by the US government for having too many apple only integrations. AI assistants only barely had enough data to understand your requests, and now have even less.
donatj 2y ago
I suspect Amazon has connected Alexa to some sort of LLM in recent weeks because her responses to the insane stream of consciousness questions I ask basically endlessly have become both more certain and more unhinged. For instance, I recently asked her “How many cigarettes does the average jazz band smoke per night.” and instead of the prior usual “I’m sorry, I can’t answer that” she replied a very confident “15 cigarettes”. I was honestly really surprised and had to check the log in the app to be sure she had heard what I had said correctly. She had indeed. Another one I have asked her many times is “How many goats are in a goat boat?” (I really just ask endlessly insane crap every time I am in my kitchen, it’s some sort of addiction) and again where she used to reply the “I’m sorry” response, she now replies with the confidence of an LLM “2 goats”
burnte 2y ago
> Basic things like "Hey Google, turn up the volume" don't work. Just an FYI this type of instruction has worked for about a decade, actually.

troupo 2y ago

Many of those have nothing to do with AI, and everything to do with access points and integrations in applications and OS. But for everything else they do suck.

aurareturn 2y ago
Giving AI access to OS APIs is trivial for Apple and Google. The problem has been that AI assistant were always too dumb to understand user requests. LLMs changed everything.
- troupo 2y ago
  First, it's not as trivial to give access to OS APIs as you think. Second, giving access to APIs... gives you access to APIs, and that's it. Even the most advanced LLMs are dumb as shit when it comes to anything non-trivial in programming. What makes you think that a modern LLM will manage to translate "move tab X to a different window" into a dozen or more API calls with proper structures filled in, and in the correct order?
  - aurareturn 2y ago
    It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. I don't sense that GPT4 is "dumb as shit". I sense that it's extremely capable and very close to changing everything if, for example, macOS completely integrates GPT4.
    - themoonisachees 2y ago
      You should try it. Go Ask chtagpt to generate a program that finds all windows in XYZ project and move them to a 'ew workspace and I guarantee it'll fail. Maybe it'll get it right if you feed it back error messages, but you can't do that in production.
    - troupo 2y ago
      > It's trivial. For example, most macOS apps can already get these permissions. Apple can grant special permissions for Siri quite easily. Of course it's not trivial if you think for more than a second about it. LLMs produce output in exactly three ways: - text - images - video What you think is trivial is to convert that output into an arbitrary function call for any arbitrary OS-level API. And that is _before_ we start thinking about things like "what do we do with incomplete LLM output" and "what to do when LLM hallucinates". You can literally try and implement this yourself, today, to see how trivial it is. You already have access to tens of thousands OS APIs, so you can try and implement a very small subset of what you're thinking about. BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls.
      - everforward 2y ago
        > BTW, if your answer is "but function calls", they are not function calls. They are structured JSON responses that you have to manually convert to actual function calls. You could in theory send ASTs as JSON.
        troupo 2y ago
        In theory, yes. Someone still needs to compile them and match them to functions etc.
      - aurareturn 2y ago
        Mapping JSON responses to OS APIs is the easiest part. It's trivial compared to understanding what users actually want.
- datadrivenangel 2y ago
  The problem is that Google walked back their integrations for anti-trust reasons, and Apple is now being sued by the US government for having too many apple only integrations. AI assistants only barely had enough data to understand your requests, and now have even less.

donatj 2y ago

I suspect Amazon has connected Alexa to some sort of LLM in recent weeks because her responses to the insane stream of consciousness questions I ask basically endlessly have become both more certain and more unhinged. For instance, I recently asked her “How many cigarettes does the average jazz band smoke per night.” and instead of the prior usual “I’m sorry, I can’t answer that” she replied a very confident “15 cigarettes”. I was honestly really surprised and had to check the log in the app to be sure she had heard what I had said correctly. She had indeed. Another one I have asked her many times is “How many goats are in a goat boat?” (I really just ask endlessly insane crap every time I am in my kitchen, it’s some sort of addiction) and again where she used to reply the “I’m sorry” response, she now replies with the confidence of an LLM “2 goats”

burnte 2y ago

> Basic things like "Hey Google, turn up the volume" don't work. Just an FYI this type of instruction has worked for about a decade, actually.