6 ms·

We don’t know how to build conversational software yet

59 points by guifortaine 11y ago

lfowles 11y ago
* level 1 — command & response Functions? * level 2 — hard-coded conversation flows Scripts? * level 3 — fuzzy/continuous/fluid state. Threading? I can't shake the feeling that deep learning is being shoehorned into a bot with a specific purpose (ShoeBot, etc). Talking to a ShoeBot, I would expect it to have predictable responses or state changes. I don't want it to suddenly infer that I'm looking for shoes for my wife because of an opaque model it's formed from an insufficient set of training data. Many applications of deep learning seem to be just a lazy way to have the machines do the work automagically.
- Mtinie 11y ago
  > I don't want it to suddenly infer that I'm looking for shoes for my wife... If the end goal is a human-like level of conversation, isn't this a desirable outcome? Inference and allusion are two elements that are used all the time during conversations between people -- with positive and negative consequences. I guess it depends on the ultimate outcome you are seeking from the conversation.
  - lfowles 11y ago
    Ultimate outcome is that I want to buy a good pair of shoes for myself :) I'm not particularly interested in the conversation unless the bot happened to be a domain expert (I find it unlikely that there is enough training data available to make it a credible domain expert..). The point was that I'm especially wary of its internal state incorrectly transitioning but now no one knows why due to the deep learning implementation.
    - Mtinie 11y ago
      Fair point, especially if the bot becomes biased in some way after that point and fails to adapt to your cues ("no, these shoes are for me, not my wife..").
- kastnerkyle 11y ago
  As you say, if your only goal is ShoeBot this (scripts, functions, etc.) is a great way to go. One of the key points of deep learning approaches is that you have this abstract, powerful computational device that is trained to extract the necessary features and also perform the task jointly - and it just happens to be a ShoeBot due to the training data. This usually results in improved performance along with "ease of use" in transitioning to new but related applications. A model just happens to be a ShoeBot when trained on specific data, but ostensibly a person or company could make ShoeBot, CarBot, ApartmentBot, etc with the exact same approach, given enough data. This is very different than a workflow of "craft tons of features for domain X, write custom scripts/conversation logic for domain X, etc.". These choices between feature based approaches and "deep" techniques are tradeoffs reminiscent of "you aren't gonna need it" versus "room to scale", but in the ML algorithms you choose rather than the software stack/implementation. It depends on things like how much data you have, how much compute you are willing to pay for, how many users you expect, and so on - but neither approach is necessarily wrong. In general if deep learning approaches don't roundly beat feature engineered or hand crafted approaches, you don't have enough data or are trying to shoehorn (pardon the pun) a solution that doesn't fit. Right tool for the job and all that.
  - lfowles 11y ago
    Like Google Photos, this might just be something I have to see to believe. However... Google Photos routinely categorizes pictures of my dog as Bear or Cat. My greatest concern is about being able to acquire the conversations necessary to train these niche applications beyond a laughable state.
awinter-py 11y ago
surprising that 'theory of mind' doesn't appear in the article.
edwhitesell 11y ago
I'd venture a bet that most of the advancement of levels 2 & 3 happen in private and/or are in use in areas you aren't already thinking of. Have a look at the conversational examples Voicebox lists on their website: http://www.voicebox.com/technology/ http://www.voicebox.com/technology/ (disclosure: I worked there for a while.) The wide-open "talk about anything" software is still a few years out, I think. But, having a "human-like" conversation/interaction with a bot/AI already happens today.
overgard 11y ago
I think the bigger question is: why would I want to converse with software? Is that actually a good interface? People imagine the computer from star trek, but that was a plot device. The character was talking to the viewer with the computer as a proxy.
- faitswulff 11y ago
  It's easy to imagine a large proportion of the population wanting to use natural language to achieve complex technical tasks. It is, after all, the same API that people use with each other. Someone could then describe what they want out of a spreadsheet instead of manually editing it. What interface would you prefer?
  - mmm123456789 11y ago
    I just built a very rudimentary version of this "speak to spreadsheet" idea: https://github.com/diesendruck/ggspeak https://github.com/diesendruck/ggspeak Or for the demo video: https://www.youtube.com/watch?v=xmtrGxVvVyg https://www.youtube.com/watch?v=xmtrGxVvVyg If you're interested in working on it together, get in touch.
  - cinquemb 11y ago
    >It's easy to imagine a large proportion of the population wanting to use natural language to achieve complex technical tasks. It is, after all, the same API that people use with each other. True, and this api will be deprecated for those who want more efficient means of communication in the future and willing to work towards it. >What interface would you prefer? EM fluctuations in real time of the neurons in my body to machine code via trillions of nano sensors/transmitters/stimulators. Vocal cords and limbs need not apply… will be way more intuitive than "natural language".
  - pjc50 11y ago
    the same API that people use with each other Have you ever tried to do a "complex technical task" by directing an unskilled person over the phone? It's a tremendously frustrating experience. Natural language alone isn't useful without your interlocutor having enough understanding of the problem domain to work out what you mean and help you. So a natural language interface to the spreadsheet would have to be on the level of "perform linear regression on these sales figures" in order to be more useful than mere speech-to-text.
vpkaihla 11y ago
Should we come up with a language that's easy to teach for both computers and humans? A sort of generic human/machine DSL?
jonbaer 11y ago
It was all done years ago ... https://en.wikipedia.org/wiki/AIML https://en.wikipedia.org/wiki/AIML
- kzhahou 11y ago
  If by done, you mean "a team designed a rigid flow-chart and rules-based engine where you can painfully map out every specific interaction in your system"... then yes!
voltagex_ 11y ago
This mentions being able to build very simple faux-conversational interfaces within a hackathon. I've got one such hackathon coming up in a few months, could I go from zero knowledge to working proof-of-concept?
- rvense 11y ago
  Eliza (for English) is close to a one-liner in any modern language. (If you don't count the templates and canned responses.)
- mmitchell 11y ago
  I believe he was talking about using an existing tool to add a 'level 1 — command & response' style interface to whatever your project is. So yeah, using something like wit.ai you could likely integrate that style of interface with zero prior knowledge fairly quickly.
ommunist 11y ago
Perhaps we forgot. Eliza bot written in PROLOG by Joseph Weizenbaum circa 1964 at MIT was quite up for the task. I am sure it is still about 90% more conversational than 80% of the current online English-speaking audience.

59 points by guifortaine 11y ago

* level 1 — command & response Functions? * level 2 — hard-coded conversation flows Scripts? * level 3 — fuzzy/continuous/fluid state. Threading? I can't shake the feeling that deep learning is being shoehorned into a bot with a specific purpose (ShoeBot, etc). Talking to a ShoeBot, I would expect it to have predictable responses or state changes. I don't want it to suddenly infer that I'm looking for shoes for my wife because of an opaque model it's formed from an insufficient set of training data. Many applications of deep learning seem to be just a lazy way to have the machines do the work automagically.

Mtinie 11y ago
> I don't want it to suddenly infer that I'm looking for shoes for my wife... If the end goal is a human-like level of conversation, isn't this a desirable outcome? Inference and allusion are two elements that are used all the time during conversations between people -- with positive and negative consequences. I guess it depends on the ultimate outcome you are seeking from the conversation.
- lfowles 11y ago
  Ultimate outcome is that I want to buy a good pair of shoes for myself :) I'm not particularly interested in the conversation unless the bot happened to be a domain expert (I find it unlikely that there is enough training data available to make it a credible domain expert..). The point was that I'm especially wary of its internal state incorrectly transitioning but now no one knows why due to the deep learning implementation.
  - Mtinie 11y ago
    Fair point, especially if the bot becomes biased in some way after that point and fails to adapt to your cues ("no, these shoes are for me, not my wife..").
kastnerkyle 11y ago
As you say, if your only goal is ShoeBot this (scripts, functions, etc.) is a great way to go. One of the key points of deep learning approaches is that you have this abstract, powerful computational device that is trained to extract the necessary features and also perform the task jointly - and it just happens to be a ShoeBot due to the training data. This usually results in improved performance along with "ease of use" in transitioning to new but related applications. A model just happens to be a ShoeBot when trained on specific data, but ostensibly a person or company could make ShoeBot, CarBot, ApartmentBot, etc with the exact same approach, given enough data. This is very different than a workflow of "craft tons of features for domain X, write custom scripts/conversation logic for domain X, etc.". These choices between feature based approaches and "deep" techniques are tradeoffs reminiscent of "you aren't gonna need it" versus "room to scale", but in the ML algorithms you choose rather than the software stack/implementation. It depends on things like how much data you have, how much compute you are willing to pay for, how many users you expect, and so on - but neither approach is necessarily wrong. In general if deep learning approaches don't roundly beat feature engineered or hand crafted approaches, you don't have enough data or are trying to shoehorn (pardon the pun) a solution that doesn't fit. Right tool for the job and all that.
- lfowles 11y ago
  Like Google Photos, this might just be something I have to see to believe. However... Google Photos routinely categorizes pictures of my dog as Bear or Cat. My greatest concern is about being able to acquire the conversations necessary to train these niche applications beyond a laughable state.

awinter-py 11y ago

surprising that 'theory of mind' doesn't appear in the article.

edwhitesell 11y ago

I'd venture a bet that most of the advancement of levels 2 & 3 happen in private and/or are in use in areas you aren't already thinking of. Have a look at the conversational examples Voicebox lists on their website: http://www.voicebox.com/technology/ http://www.voicebox.com/technology/ (disclosure: I worked there for a while.) The wide-open "talk about anything" software is still a few years out, I think. But, having a "human-like" conversation/interaction with a bot/AI already happens today.

overgard 11y ago

I think the bigger question is: why would I want to converse with software? Is that actually a good interface? People imagine the computer from star trek, but that was a plot device. The character was talking to the viewer with the computer as a proxy.

faitswulff 11y ago
It's easy to imagine a large proportion of the population wanting to use natural language to achieve complex technical tasks. It is, after all, the same API that people use with each other. Someone could then describe what they want out of a spreadsheet instead of manually editing it. What interface would you prefer?
- mmm123456789 11y ago
  I just built a very rudimentary version of this "speak to spreadsheet" idea: https://github.com/diesendruck/ggspeak https://github.com/diesendruck/ggspeak Or for the demo video: https://www.youtube.com/watch?v=xmtrGxVvVyg https://www.youtube.com/watch?v=xmtrGxVvVyg If you're interested in working on it together, get in touch.
- cinquemb 11y ago
  >It's easy to imagine a large proportion of the population wanting to use natural language to achieve complex technical tasks. It is, after all, the same API that people use with each other. True, and this api will be deprecated for those who want more efficient means of communication in the future and willing to work towards it. >What interface would you prefer? EM fluctuations in real time of the neurons in my body to machine code via trillions of nano sensors/transmitters/stimulators. Vocal cords and limbs need not apply… will be way more intuitive than "natural language".
- pjc50 11y ago
  the same API that people use with each other Have you ever tried to do a "complex technical task" by directing an unskilled person over the phone? It's a tremendously frustrating experience. Natural language alone isn't useful without your interlocutor having enough understanding of the problem domain to work out what you mean and help you. So a natural language interface to the spreadsheet would have to be on the level of "perform linear regression on these sales figures" in order to be more useful than mere speech-to-text.

vpkaihla 11y ago

Should we come up with a language that's easy to teach for both computers and humans? A sort of generic human/machine DSL?

jonbaer 11y ago

It was all done years ago ... https://en.wikipedia.org/wiki/AIML https://en.wikipedia.org/wiki/AIML

kzhahou 11y ago
If by done, you mean "a team designed a rigid flow-chart and rules-based engine where you can painfully map out every specific interaction in your system"... then yes!

voltagex_ 11y ago

This mentions being able to build very simple faux-conversational interfaces within a hackathon. I've got one such hackathon coming up in a few months, could I go from zero knowledge to working proof-of-concept?

rvense 11y ago
Eliza (for English) is close to a one-liner in any modern language. (If you don't count the templates and canned responses.)
mmitchell 11y ago
I believe he was talking about using an existing tool to add a 'level 1 — command & response' style interface to whatever your project is. So yeah, using something like wit.ai you could likely integrate that style of interface with zero prior knowledge fairly quickly.

ommunist 11y ago

Perhaps we forgot. Eliza bot written in PROLOG by Joseph Weizenbaum circa 1964 at MIT was quite up for the task. I am sure it is still about 90% more conversational than 80% of the current online English-speaking audience.