10 ms·

The point of this isn't asking how to apply boolean search operators, it's showing that the largest AI-focused companies in the world absolutely suck at NLP.

by meritt 6y ago

Justin_K 6y ago
I'd go as far as saying "AI" is only as good as someone has taken the time to program the use case.
- alexpapworth 6y ago
  So, it's really just a series of if statements?
  - seph-reed 6y ago
    Either that or vague network of trial and error generalizations (deep learning/gut learning).
  - joshmn 6y ago
    Don't get ahead of yourself: it's SELECT with GROUP BY.
    - Justin_K 6y ago
      Don't forget AND keyword LIKE '%searchStr%'
  - Justin_K 6y ago
    Not that but maybe someone can explain if the technology is "AI" why it still takes tens of thousands of hours to implement a project? Shouldn't we call it AI when the frameworks can be implemented with relative ease? My point is that the level of customization required to achieve AI, in my opinion, doesn't make it very intelligent.
  - fluffything 6y ago
    Yes, each of which wraps a "Not Hotdog" app. Turns out that "Not Hotdog" is quite hard already.
  - xemdetia 6y ago
    One of the primary methods to join multiple learned models is a decision tree (or combination of decision trees making a decision forest), which can be simplified as a series of if statements/conditionals. So if you join a 'is it a shirt' model with a 'is it striped' model you get two sets of things, and with how big data approaches this it is something you can do quickly. As other people have pointed out the issue here is that the NLP of the actual search is not creating a negation of two sets, it is returning the intersect of sets 'is it a shirt', 'is it striped' and shrugging its shoulders and either intersecting it with 'things with the text without', or throwing up its hands entirely based on the context because it wasn't programmed to do something smarter.
- scabarott 6y ago
  Huh? It's supposed to be the opposite, hence the "I"
  - andrewzah 6y ago
    The term has been bastardized. AI is now any app that uses if statements, so long as it's obscured to the user. So now "AGI" exists, because apparently the word "general" now means "intelligent", unlike "intelligence". At some point in the future marketers will learn about AGI, and we'll have to make yet another term, maybe artificial general practical intelligence?
    - marcosdumay 6y ago
      AGI is some AI that isn't specialized in a single problem. There is nothing on the word "intelligence" to imply it's not specialized.
      - andrewzah 6y ago
        Right, because when I think of "intelligence" I assume it's specialized and otherwise is dumb as rocks. Naturally. Which is why marketers are really just advertising specialized tools, not agi. Obviously. I know what agi is; I just find the terms backward.
        marcosdumay 6y ago
        When you think of intelligence, you think of the human kind. As we create different things that have some of the properties we usually identify as "intelligence", but lack others that were completely correlated until that time, we need to create better words that can express the new situation. It's perfectly ok to prefer the qualifier to go the other way around, keep intelligence general and change the name of the specialized form. But that's just not how our language evolved.
- gumby 6y ago
  Yes, basically 1990s Expert Systems but with even more manual labor.
scabarott 6y ago
Or maybe there's just not a lot of AI in their main search product (for whatever reason). They seem to be pretty good in other areas (Translate, Cloud speech-to-text, Alexa etc.)
- still_grokking 6y ago
  For the matter of translation they actually aren't pretty good. DeepL outsmarted all of them (at least for the supported languages). Given the resources thrown at this problem by the different companies I would even say the results of Google, MS and such are actually quite disappointing. (And I don't think one could say "but this companies did the upfront work" as the basic ideas are something 60 years old).
waynecochran 6y ago
Every systems sucks at NLP. It is an extremely hard problem and we are nowhere near matching the human cerebral cortex.
- binarymax 6y ago
  Generally, yes. But you can go very far and cover a lot of cases when you stick to a narrow domain and common patterns that customers of your product will typically use.
  - waynecochran 6y ago
    My reaction to Siri is either amazement at how well it worked or disgust in its failure on seemingly simple queries.
    - binarymax 6y ago
      Siri has lots of known problems. My biggest gripes are the lack of context and lack of a "discussion" state. But Siri is a general domain problem, which is really really hard. Siri set the expectation you can ask it anything, and it works terribly and for most questions just gives up and runs a web search. If you are an e-commerce company though, that's a narrow enough domain, because you know that for most people, they're looking for products to buy or compare. It's not an unbounded Q&A service.
  - huffmsa 6y ago
    Yeah the examples would probably do better if they had a front layer model determining the general topic "clothing" then shipping the query off to the clothing and fashion specific model for details. Lots of focus on a general purpose mono-model, but I think a collection of specialized subsystems is a better representation and would produce better results, faster.
packetlost 6y ago
Why would you really apply NLP to a search engine though? Generally speaking a weighted keyword search is good enough 95% of the time and requires significantly less resources to perform.
- rjurney 6y ago
  Because there's something called conceptual search that people are dying for to find what they actually want.
- binarymax 6y ago
  I work specifically in this field with clients, and deliver training on applying NLP to search. You’d be surprised how effective NLP is for use when identifying query intent, and pulling out modifiers that should apply as metadata filters. Weighted keyword search works a lot, but it fails hard for many long tail queries (especially in e-commerce and other attribute heavy domains). IMO there really isn’t a good excuse for these firms to fail at queries like this. The query itself isn’t particularly difficult when using a decent NLP stack and following well known practices.
  - skewart 6y ago
    If it's technically possible then presumably it's a deliberate product choice to not have better search results for "shirt without stripes". And that seems entirely plausible. Google is already by far the most widely used search engine, so they don't really need to innovate or improve the search product very much in order to attract and retain users. Presumably capturing more advertising spending from the companies paying for ads is a bigger priority. Microsoft under Satya Nadella has been all about enterprise and cloud, and I doubt Bing is a strategic priority any more, so it's not surprising that they wouldn't put a lot of resources into making it better. Amazon is a little surprising. You'd think they'd have a lot to gain from making it easier for people to find what they're looking for. But maybe less than perfect search results are deliberate? Maybe it's like how supermarkets put basic items in the back of the store and high-margin impulse buys in the front - so you have to walk past chocolates and chips if you want to buy a carton of milk. If Amazon is deliberately nerfing search results then maybe Google would stand to benefit from having better shopping-related results - people would get frustrated trying to find a shirt without stripes on Amazon and just use Google instead, letting Google profit from advertising in the process. But maybe people selling shirts aren't willing to pay much for ads, so there isn't much money for Google to make by getting better at finding specific types of shirts. I dunno if any of these conjectures are anywhere near accurate, but it's interesting to think about.
dbliss 6y ago
It's not as simple as NLP. The shirts them selves have to labelled as having stripes or not. If striped is not a attribute of the shirt, it doesn't matter of the NLP can parse the meaning of striped.

The point of this isn't asking how to apply boolean search operators, it's showing that the largest AI-focused companies in the world absolutely suck at NLP.

by meritt 6y ago

The point of this isn't asking how to apply boolean search operators, it's showing that the largest AI-focused companies in the world absolutely suck at NLP.

Justin_K 6y ago

I'd go as far as saying "AI" is only as good as someone has taken the time to program the use case.

alexpapworth 6y ago
So, it's really just a series of if statements?
- seph-reed 6y ago
  Either that or vague network of trial and error generalizations (deep learning/gut learning).
- joshmn 6y ago
  Don't get ahead of yourself: it's SELECT with GROUP BY.
  - Justin_K 6y ago
    Don't forget AND keyword LIKE '%searchStr%'
- Justin_K 6y ago
  Not that but maybe someone can explain if the technology is "AI" why it still takes tens of thousands of hours to implement a project? Shouldn't we call it AI when the frameworks can be implemented with relative ease? My point is that the level of customization required to achieve AI, in my opinion, doesn't make it very intelligent.
- fluffything 6y ago
  Yes, each of which wraps a "Not Hotdog" app. Turns out that "Not Hotdog" is quite hard already.
- xemdetia 6y ago
  One of the primary methods to join multiple learned models is a decision tree (or combination of decision trees making a decision forest), which can be simplified as a series of if statements/conditionals. So if you join a 'is it a shirt' model with a 'is it striped' model you get two sets of things, and with how big data approaches this it is something you can do quickly. As other people have pointed out the issue here is that the NLP of the actual search is not creating a negation of two sets, it is returning the intersect of sets 'is it a shirt', 'is it striped' and shrugging its shoulders and either intersecting it with 'things with the text without', or throwing up its hands entirely based on the context because it wasn't programmed to do something smarter.
scabarott 6y ago
Huh? It's supposed to be the opposite, hence the "I"
- andrewzah 6y ago
  The term has been bastardized. AI is now any app that uses if statements, so long as it's obscured to the user. So now "AGI" exists, because apparently the word "general" now means "intelligent", unlike "intelligence". At some point in the future marketers will learn about AGI, and we'll have to make yet another term, maybe artificial general practical intelligence?
  - marcosdumay 6y ago
    AGI is some AI that isn't specialized in a single problem. There is nothing on the word "intelligence" to imply it's not specialized.
    - andrewzah 6y ago
      Right, because when I think of "intelligence" I assume it's specialized and otherwise is dumb as rocks. Naturally. Which is why marketers are really just advertising specialized tools, not agi. Obviously. I know what agi is; I just find the terms backward.
      - marcosdumay 6y ago
        When you think of intelligence, you think of the human kind. As we create different things that have some of the properties we usually identify as "intelligence", but lack others that were completely correlated until that time, we need to create better words that can express the new situation. It's perfectly ok to prefer the qualifier to go the other way around, keep intelligence general and change the name of the specialized form. But that's just not how our language evolved.
gumby 6y ago
Yes, basically 1990s Expert Systems but with even more manual labor.

scabarott 6y ago

Or maybe there's just not a lot of AI in their main search product (for whatever reason). They seem to be pretty good in other areas (Translate, Cloud speech-to-text, Alexa etc.)

still_grokking 6y ago
For the matter of translation they actually aren't pretty good. DeepL outsmarted all of them (at least for the supported languages). Given the resources thrown at this problem by the different companies I would even say the results of Google, MS and such are actually quite disappointing. (And I don't think one could say "but this companies did the upfront work" as the basic ideas are something 60 years old).

waynecochran 6y ago

Every systems sucks at NLP. It is an extremely hard problem and we are nowhere near matching the human cerebral cortex.

binarymax 6y ago
Generally, yes. But you can go very far and cover a lot of cases when you stick to a narrow domain and common patterns that customers of your product will typically use.
- waynecochran 6y ago
  My reaction to Siri is either amazement at how well it worked or disgust in its failure on seemingly simple queries.
  - binarymax 6y ago
    Siri has lots of known problems. My biggest gripes are the lack of context and lack of a "discussion" state. But Siri is a general domain problem, which is really really hard. Siri set the expectation you can ask it anything, and it works terribly and for most questions just gives up and runs a web search. If you are an e-commerce company though, that's a narrow enough domain, because you know that for most people, they're looking for products to buy or compare. It's not an unbounded Q&A service.
- huffmsa 6y ago
  Yeah the examples would probably do better if they had a front layer model determining the general topic "clothing" then shipping the query off to the clothing and fashion specific model for details. Lots of focus on a general purpose mono-model, but I think a collection of specialized subsystems is a better representation and would produce better results, faster.

packetlost 6y ago

Why would you really apply NLP to a search engine though? Generally speaking a weighted keyword search is good enough 95% of the time and requires significantly less resources to perform.

rjurney 6y ago
Because there's something called conceptual search that people are dying for to find what they actually want.
binarymax 6y ago
I work specifically in this field with clients, and deliver training on applying NLP to search. You’d be surprised how effective NLP is for use when identifying query intent, and pulling out modifiers that should apply as metadata filters. Weighted keyword search works a lot, but it fails hard for many long tail queries (especially in e-commerce and other attribute heavy domains). IMO there really isn’t a good excuse for these firms to fail at queries like this. The query itself isn’t particularly difficult when using a decent NLP stack and following well known practices.
- skewart 6y ago
  If it's technically possible then presumably it's a deliberate product choice to not have better search results for "shirt without stripes". And that seems entirely plausible. Google is already by far the most widely used search engine, so they don't really need to innovate or improve the search product very much in order to attract and retain users. Presumably capturing more advertising spending from the companies paying for ads is a bigger priority. Microsoft under Satya Nadella has been all about enterprise and cloud, and I doubt Bing is a strategic priority any more, so it's not surprising that they wouldn't put a lot of resources into making it better. Amazon is a little surprising. You'd think they'd have a lot to gain from making it easier for people to find what they're looking for. But maybe less than perfect search results are deliberate? Maybe it's like how supermarkets put basic items in the back of the store and high-margin impulse buys in the front - so you have to walk past chocolates and chips if you want to buy a carton of milk. If Amazon is deliberately nerfing search results then maybe Google would stand to benefit from having better shopping-related results - people would get frustrated trying to find a shirt without stripes on Amazon and just use Google instead, letting Google profit from advertising in the process. But maybe people selling shirts aren't willing to pay much for ads, so there isn't much money for Google to make by getting better at finding specific types of shirts. I dunno if any of these conjectures are anywhere near accurate, but it's interesting to think about.

dbliss 6y ago

It's not as simple as NLP. The shirts them selves have to labelled as having stripes or not. If striped is not a attribute of the shirt, it doesn't matter of the NLP can parse the meaning of striped.