3 ms·

I think it's largely because speech recognition operates on only one sentence at a time. If you need to clarify a term said earlier or provide an explanatory p

I think it's largely because speech recognition operates on only one sentence at a time. If you need to clarify a term said earlier or provide an explanatory phrase describing an earlier one, it can't connect the second utterance to the first. No context is retained. That's a surprisingly big limitation in terms of human conversation. Until NLP can create a short-term memory model to retain info on prior utterances, Alexa's comprehension will be limited to self-contained single phrases -- spells and incantations.

jimbokun 4y ago
Which is strange because that's exactly what very large language models like GPT-3 are good at. Keeping the context and generating plausible next sentences, paragraphs, etc. Wonder what's blocking that from being applied to systems like Alexa.
- groby_b 4y ago
  Unpredictability. You (Amazon) don't want your precious little bot say random things that land you in the press. And oh, GPT-3 will. Point two is that models that can be goaded to a specific place. So you'll need to account for that too, or... oh look, press. Point 3 is that they're trained on the Internet at large, and you don't want that in your house :)
  - randcraw 4y ago
    And if it chooses the wrong prior topic (among several possibilities) the smartbot could behave maddeningly obtusely. I suspect even the best of these today will have little idea what part of the conversation is more relevant to the human's focus than any other part. It's just as likely to turn its attention to an irrelevant 'token' that came before as a relevant one and say something obvious or fatuous. It's a lot easier to converse gracefully than to add useful info to a conversation. This is essentially what has made past intelligent assistants so dysfunctional (like Microsoft's Clippy). They couldn't model cognitive concepts like 'relevant', 'interesting', or 'necessary' relative to the active topic.
visarga 4y ago
> Until NLP can create a short-term memory model Transformers can fit about 4000 tokens in the context window. That's a lot of dialogue rounds. The problem is these models are too expensive to offer at scale. We almost never get to see the latest AI in big corporation products. It is always AI from a couple generations ago that is cheaper to run. When we get the latest and greatest, like GPT-3 and DallE, then it's super expensive to use, not free.

jimbokun 4y ago

Which is strange because that's exactly what very large language models like GPT-3 are good at. Keeping the context and generating plausible next sentences, paragraphs, etc. Wonder what's blocking that from being applied to systems like Alexa.

groby_b 4y ago
Unpredictability. You (Amazon) don't want your precious little bot say random things that land you in the press. And oh, GPT-3 will. Point two is that models that can be goaded to a specific place. So you'll need to account for that too, or... oh look, press. Point 3 is that they're trained on the Internet at large, and you don't want that in your house :)
- randcraw 4y ago
  And if it chooses the wrong prior topic (among several possibilities) the smartbot could behave maddeningly obtusely. I suspect even the best of these today will have little idea what part of the conversation is more relevant to the human's focus than any other part. It's just as likely to turn its attention to an irrelevant 'token' that came before as a relevant one and say something obvious or fatuous. It's a lot easier to converse gracefully than to add useful info to a conversation. This is essentially what has made past intelligent assistants so dysfunctional (like Microsoft's Clippy). They couldn't model cognitive concepts like 'relevant', 'interesting', or 'necessary' relative to the active topic.

visarga 4y ago

> Until NLP can create a short-term memory model Transformers can fit about 4000 tokens in the context window. That's a lot of dialogue rounds. The problem is these models are too expensive to offer at scale. We almost never get to see the latest AI in big corporation products. It is always AI from a couple generations ago that is cheaper to run. When we get the latest and greatest, like GPT-3 and DallE, then it's super expensive to use, not free.