5 ms·

How would you use an accurate Natural Language Understanding (NLU) API?

51 points by apetresc 14y ago

rhizome 14y ago
How accurate? Accuracy is a value, not a state.
- apetresc 14y ago
  Our technology was initially designed to be used with external ASR. For sentences within our 18 domains, where the ASR is correct, we see accuracy (defined by mapping into the current domain and action) around 90%. The reality of course though is ASR is sometimes wrong, which does decrease our accuracy in practice. The best direct demonstration of practical accuracy is just try out the app ;)
  - deleted 14y ago
    [deleted]
hooande 14y ago
I would use a good natural language understanding api to do basic research, ideally monitoring many different feeds for topics of interest. It could also be a good way to monitor trends or competitors. I can't think of many uses cases for natural language as an interface to a traditional website. If there's something that's more convenient to access with text/speech, the user can just use set commands. On the topic of building a personal assistant app, from what I've seen the real value lies in the act of interactive scheduling. You can't say "schedule a meeting with george" without checking with george's schedule to make sure he's available, and potentially re-scheduling the event at a time of mutual availability. Of course this goes far beyond the scope of understanding language, but I think this is a big part of what will differentiate a good personal assistant app from Siri. EDIT: Someone on the thread brought up customer service, which is also a great opportunity. Interacting with a company's service representatives is a huge time sink for people everywhere, especially on the phone. Anything that lets people spend less time on hold is a huge win for mankind.
- drewch 14y ago
  That's a pretty sweet idea. I wonder how difficult it is to set up sharing of calendars, and the permissions that are involved in it.
  - icebraining 14y ago
    Google Calendar lets you share only the Free/Busy times of your schedule, without revealing the details of each event. Ideally, of course, your scheduler should be able to "propose" some times to George's, and let it decide and reply back with the chosen one.
- juharris 14y ago
  Actually, I've worked for a company that specializes in finding topics based on your interests. http://primal.com/ http://primal.com/
vhf 14y ago
I cannot answer your question -- I have no idea. (edited : well, why not for "customer service", along with the traditional FAQ ? Or use it instead of the useless "solve this problem" useless multiple choice walkthrough in Windows or such ? ) But I upvoted because I really liked to discover this company, what it does, and more importantly how it does it. Thanks for the clear examples and output provided !
- jpantony 14y ago
  Thank you for your up-vote :)
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
languagehacker 14y ago
From what I read, I don't really see a robust NLU engine here. I see a probabilistic transformation engine (maybe bootstrapped with some linguistic data structures) that identifies 7 buckets and then rephrases the query or request into a format better suited to achieve the desired result in that engine. This isn't too far from saying that you have a search engine that "knows" how to answer questions, because you have hard-coded it to omit question words and transform question phrases into something declarative in hopes of finding the best match (e.g. "What is the largest mountain in Asia?" --> "The largest mountain in Asia", and then list the top search result from Google). There's this tremendous amount of pragmatism that goes into some of the example questions, too. If I ask someone if they can recommend a good sushi place, personally, I'm probably more interested in the best sushi hit they've ever had than whether then place I can walk to from work is decent. You're going to have to make a lot of assumptions that you cannot adequately contextualize to take a broad question like that and say "Well, here are two within two blocks of you." I might actually be ranking proximity lower than, say, quality, popularity, or memorability. By hard-coding such assumptions into your reasoning, you're not in the trade of understanding natural language; you're in the trade of doing exactly what Bing does now, at a smaller scale, with a few more buckets. If you're really into building NLU, I have a few somewhat snarky suggestions for you: 1) Swing by a cryogenics facility and check back in with us in 20-30 years. Repeat until cognitive scientists have figured out a way to transform human thought into something that can be quantified and transformed into words. 2) Do some reading in cognitive science to understand the immensely difficult problem it is to quantify meaning, both within the context of a discourse as well as discourse-agnostic (archetypal meaning versus contextualized meaning). Look at some charts, and realize that word meaning is not only relative to other words in the same discourse, but other words in shared classes. 3) Read up on Grice, and understand that, despite his philosophical approach to linguistics, his points on implicature and the value of what goes unsaid in a conversation lend as much to context as what goes said. Then try to sit back and figure out how to program and analyze what's going unsaid in a search query. What I'm trying to say is that I would do nothing with this API because I can't trust it to deliver on its very lofty promises. Natural language understanding is essentially a pipe dream in the current state of computer science and bioinformatics. Word meaning is so idiosyncratic that it is a miracle that humans can generally agree on what a word represents; most of these are extremely fuzzy, and cannot be divorced from the contexts in which they are uttered. It's far too difficult to contextualize even a sentence outside of a pure information retrieval standpoint. I appreciate what you're aiming at here, but I doubt you'll ever be able to succeed at it. That being said, if you rephrased it into something like a decision engine, I probably wouldn't be so irritated by it, and might even give it a shot.
fatbat 14y ago
I am interested in the calendar + reminder domain of it. More so if it works offline (ie- Soundhound/Shazam app). Can you elaborate on the calendar domain?
- pdat 14y ago
  hi, Well currently we support 4 actions within calendar, "CREATE_EVENT", "SEARCH_CALENDAR", "MODIFY_EVENT" and "DELETE_EVENT". There are a number of different entities including "dateRange, lengthOfTime, timeRange, date, time, meetingTopic, contactName, friendOrRelative, location, originalTime, originalDate, originalLocation, replacementTime, replacementDate" and a few others I'm sure I've missed. For reminder we support "CREATE_REMINDER" and "SEARCH_REMINDER", we also support the entities "message", "time" and "arrivalLocation" and "leaveLocation". Does this answer your question or did you want a more specific answer? If you want more information feel free to email us directly :) .
  - fatbat 14y ago
    Hi! Thanks for the reply! One more quick one. :) Are the "time" entities relative or absolute time or either?
    - jpantony 14y ago
      Currently we support relative to the current time ("three hours from now") or absolute ("six pm"). We'd be happy to extend it however if we saw demand for extra functionality :) .
      - fatbat 14y ago
        A strtotime() with voice? Awesome and thank you!
    - apetresc 14y ago
      We normalize relative times so that they're absolute. So, "remind me to marinate the chicken in 2 hours" will get normalized to 6:00 PM (assuming it's being said at 4:00 PM)
deleted 14y ago
[deleted]
ckrailo 14y ago
I wish there was something I could hook up to my electronic life (calendars, search engines, knowledge stores, bug tracking, etc) and send IMs to from all devices. Like Siri, but more useful and more connected. A general, all around AI instead of a glorified Google search with speech. Example commands I wish I could do (some would require me hosting a server to communicate with my home, but I'm okay with this): "Lock my doors." "I'm going home." (Notify people, set AC, bring up a map, etc.) "What's the weather tonight?" "Write 'happy birthday!' on my friends' Facebook walls each time it's their birthday." "Ask my girlfriend what's for dinner when she gets on Skype." ...and so on... That said, I'm super excited about this and would love to see Maluuba's work accessible on all devices so I can just IM the service instead of using a phone when I'm at home.
- jpantony 14y ago
  I'm excited to see someone talking about a single unified nexus for everything. That is part of the future vision of our company, and smart homes are an example of where we want to go in the next few years :)
- StavrosK 14y ago
  I had this idea not too long ago. An assistant as a service, that would be accessible on IM and would do increasingly more things, starting with handling the scheduling of appointments between two people completely automatically (it would email them and ask them if they're available, of course). I made a small prototype, but it was way too limited, as NLP is a huge problem. Even a simple TODO list/alerts/meeting scheduling app with calendar integration was very, very complicated. I'm pretty sure lots of people would pay for it, though.
  - caublestone 14y ago
    https://lizibot.com/ https://lizibot.com/ It's pretty cool. Try it out.
flipstewart 14y ago
Siri satisfies all my query related NLU needs, BUT... There are other applications for NLU that I'd use daily were they available. A prime example is taking text and producing a tl;dr version for me.
skrebbel 14y ago
I wouldn't even consider it when "all" it understands is English. The world is a big place!
chezmo 14y ago
As written in the article, the technology is meant to understand short queries, in other words short commands. I guess everybody agrees, nowadays entering commands to any kind of machine is still much more accurate and quicker if you use your hands for doing so. This said, i think the technologies is in particular useful in situations in which you are not able to use your hands. This could be for example because you are busy working or you are handicapped. So one idea which comes to my mind would be an assistant for handicapped people, e.g. a device which can be used to control an electric wheelchair.

51 points by apetresc 14y ago

rhizome 14y ago

How accurate? Accuracy is a value, not a state.

apetresc 14y ago
Our technology was initially designed to be used with external ASR. For sentences within our 18 domains, where the ASR is correct, we see accuracy (defined by mapping into the current domain and action) around 90%. The reality of course though is ASR is sometimes wrong, which does decrease our accuracy in practice. The best direct demonstration of practical accuracy is just try out the app ;)
- deleted 14y ago
  [deleted]

hooande 14y ago

I would use a good natural language understanding api to do basic research, ideally monitoring many different feeds for topics of interest. It could also be a good way to monitor trends or competitors. I can't think of many uses cases for natural language as an interface to a traditional website. If there's something that's more convenient to access with text/speech, the user can just use set commands. On the topic of building a personal assistant app, from what I've seen the real value lies in the act of interactive scheduling. You can't say "schedule a meeting with george" without checking with george's schedule to make sure he's available, and potentially re-scheduling the event at a time of mutual availability. Of course this goes far beyond the scope of understanding language, but I think this is a big part of what will differentiate a good personal assistant app from Siri. EDIT: Someone on the thread brought up customer service, which is also a great opportunity. Interacting with a company's service representatives is a huge time sink for people everywhere, especially on the phone. Anything that lets people spend less time on hold is a huge win for mankind.

drewch 14y ago
That's a pretty sweet idea. I wonder how difficult it is to set up sharing of calendars, and the permissions that are involved in it.
- icebraining 14y ago
  Google Calendar lets you share only the Free/Busy times of your schedule, without revealing the details of each event. Ideally, of course, your scheduler should be able to "propose" some times to George's, and let it decide and reply back with the chosen one.
juharris 14y ago
Actually, I've worked for a company that specializes in finding topics based on your interests. http://primal.com/ http://primal.com/

vhf 14y ago

I cannot answer your question -- I have no idea. (edited : well, why not for "customer service", along with the traditional FAQ ? Or use it instead of the useless "solve this problem" useless multiple choice walkthrough in Windows or such ? ) But I upvoted because I really liked to discover this company, what it does, and more importantly how it does it. Thanks for the clear examples and output provided !

jpantony 14y ago
Thank you for your up-vote :)

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

languagehacker 14y ago

From what I read, I don't really see a robust NLU engine here. I see a probabilistic transformation engine (maybe bootstrapped with some linguistic data structures) that identifies 7 buckets and then rephrases the query or request into a format better suited to achieve the desired result in that engine. This isn't too far from saying that you have a search engine that "knows" how to answer questions, because you have hard-coded it to omit question words and transform question phrases into something declarative in hopes of finding the best match (e.g. "What is the largest mountain in Asia?" --> "The largest mountain in Asia", and then list the top search result from Google). There's this tremendous amount of pragmatism that goes into some of the example questions, too. If I ask someone if they can recommend a good sushi place, personally, I'm probably more interested in the best sushi hit they've ever had than whether then place I can walk to from work is decent. You're going to have to make a lot of assumptions that you cannot adequately contextualize to take a broad question like that and say "Well, here are two within two blocks of you." I might actually be ranking proximity lower than, say, quality, popularity, or memorability. By hard-coding such assumptions into your reasoning, you're not in the trade of understanding natural language; you're in the trade of doing exactly what Bing does now, at a smaller scale, with a few more buckets. If you're really into building NLU, I have a few somewhat snarky suggestions for you: 1) Swing by a cryogenics facility and check back in with us in 20-30 years. Repeat until cognitive scientists have figured out a way to transform human thought into something that can be quantified and transformed into words. 2) Do some reading in cognitive science to understand the immensely difficult problem it is to quantify meaning, both within the context of a discourse as well as discourse-agnostic (archetypal meaning versus contextualized meaning). Look at some charts, and realize that word meaning is not only relative to other words in the same discourse, but other words in shared classes. 3) Read up on Grice, and understand that, despite his philosophical approach to linguistics, his points on implicature and the value of what goes unsaid in a conversation lend as much to context as what goes said. Then try to sit back and figure out how to program and analyze what's going unsaid in a search query. What I'm trying to say is that I would do nothing with this API because I can't trust it to deliver on its very lofty promises. Natural language understanding is essentially a pipe dream in the current state of computer science and bioinformatics. Word meaning is so idiosyncratic that it is a miracle that humans can generally agree on what a word represents; most of these are extremely fuzzy, and cannot be divorced from the contexts in which they are uttered. It's far too difficult to contextualize even a sentence outside of a pure information retrieval standpoint. I appreciate what you're aiming at here, but I doubt you'll ever be able to succeed at it. That being said, if you rephrased it into something like a decision engine, I probably wouldn't be so irritated by it, and might even give it a shot.

fatbat 14y ago

I am interested in the calendar + reminder domain of it. More so if it works offline (ie- Soundhound/Shazam app). Can you elaborate on the calendar domain?

pdat 14y ago
hi, Well currently we support 4 actions within calendar, "CREATE_EVENT", "SEARCH_CALENDAR", "MODIFY_EVENT" and "DELETE_EVENT". There are a number of different entities including "dateRange, lengthOfTime, timeRange, date, time, meetingTopic, contactName, friendOrRelative, location, originalTime, originalDate, originalLocation, replacementTime, replacementDate" and a few others I'm sure I've missed. For reminder we support "CREATE_REMINDER" and "SEARCH_REMINDER", we also support the entities "message", "time" and "arrivalLocation" and "leaveLocation". Does this answer your question or did you want a more specific answer? If you want more information feel free to email us directly :) .
- fatbat 14y ago
  Hi! Thanks for the reply! One more quick one. :) Are the "time" entities relative or absolute time or either?
  - jpantony 14y ago
    Currently we support relative to the current time ("three hours from now") or absolute ("six pm"). We'd be happy to extend it however if we saw demand for extra functionality :) .
    - fatbat 14y ago
      A strtotime() with voice? Awesome and thank you!
  - apetresc 14y ago
    We normalize relative times so that they're absolute. So, "remind me to marinate the chicken in 2 hours" will get normalized to 6:00 PM (assuming it's being said at 4:00 PM)

deleted 14y ago

[deleted]

ckrailo 14y ago

I wish there was something I could hook up to my electronic life (calendars, search engines, knowledge stores, bug tracking, etc) and send IMs to from all devices. Like Siri, but more useful and more connected. A general, all around AI instead of a glorified Google search with speech. Example commands I wish I could do (some would require me hosting a server to communicate with my home, but I'm okay with this): "Lock my doors." "I'm going home." (Notify people, set AC, bring up a map, etc.) "What's the weather tonight?" "Write 'happy birthday!' on my friends' Facebook walls each time it's their birthday." "Ask my girlfriend what's for dinner when she gets on Skype." ...and so on... That said, I'm super excited about this and would love to see Maluuba's work accessible on all devices so I can just IM the service instead of using a phone when I'm at home.

jpantony 14y ago
I'm excited to see someone talking about a single unified nexus for everything. That is part of the future vision of our company, and smart homes are an example of where we want to go in the next few years :)
StavrosK 14y ago
I had this idea not too long ago. An assistant as a service, that would be accessible on IM and would do increasingly more things, starting with handling the scheduling of appointments between two people completely automatically (it would email them and ask them if they're available, of course). I made a small prototype, but it was way too limited, as NLP is a huge problem. Even a simple TODO list/alerts/meeting scheduling app with calendar integration was very, very complicated. I'm pretty sure lots of people would pay for it, though.
- caublestone 14y ago
  https://lizibot.com/ https://lizibot.com/ It's pretty cool. Try it out.

flipstewart 14y ago

Siri satisfies all my query related NLU needs, BUT... There are other applications for NLU that I'd use daily were they available. A prime example is taking text and producing a tl;dr version for me.

skrebbel 14y ago

I wouldn't even consider it when "all" it understands is English. The world is a big place!

chezmo 14y ago

As written in the article, the technology is meant to understand short queries, in other words short commands. I guess everybody agrees, nowadays entering commands to any kind of machine is still much more accurate and quicker if you use your hands for doing so. This said, i think the technologies is in particular useful in situations in which you are not able to use your hands. This could be for example because you are busy working or you are handicapped. So one idea which comes to my mind would be an assistant for handicapped people, e.g. a device which can be used to control an electric wheelchair.