7 ms·

Why would you really apply NLP to a search engine though? Generally speaking a weighted keyword search is good enough 95% of the time and requires significantly

Why would you really apply NLP to a search engine though? Generally speaking a weighted keyword search is good enough 95% of the time and requires significantly less resources to perform.

rjurney 7y ago
Because there's something called conceptual search that people are dying for to find what they actually want.
binarymax 7y ago
I work specifically in this field with clients, and deliver training on applying NLP to search. You’d be surprised how effective NLP is for use when identifying query intent, and pulling out modifiers that should apply as metadata filters. Weighted keyword search works a lot, but it fails hard for many long tail queries (especially in e-commerce and other attribute heavy domains). IMO there really isn’t a good excuse for these firms to fail at queries like this. The query itself isn’t particularly difficult when using a decent NLP stack and following well known practices.
- skewart 7y ago
  If it's technically possible then presumably it's a deliberate product choice to not have better search results for "shirt without stripes". And that seems entirely plausible. Google is already by far the most widely used search engine, so they don't really need to innovate or improve the search product very much in order to attract and retain users. Presumably capturing more advertising spending from the companies paying for ads is a bigger priority. Microsoft under Satya Nadella has been all about enterprise and cloud, and I doubt Bing is a strategic priority any more, so it's not surprising that they wouldn't put a lot of resources into making it better. Amazon is a little surprising. You'd think they'd have a lot to gain from making it easier for people to find what they're looking for. But maybe less than perfect search results are deliberate? Maybe it's like how supermarkets put basic items in the back of the store and high-margin impulse buys in the front - so you have to walk past chocolates and chips if you want to buy a carton of milk. If Amazon is deliberately nerfing search results then maybe Google would stand to benefit from having better shopping-related results - people would get frustrated trying to find a shirt without stripes on Amazon and just use Google instead, letting Google profit from advertising in the process. But maybe people selling shirts aren't willing to pay much for ads, so there isn't much money for Google to make by getting better at finding specific types of shirts. I dunno if any of these conjectures are anywhere near accurate, but it's interesting to think about.
- uoaei 7y ago
  I feel like that's stretching the definition of NLP though. Technically you are processing natural language but it seems like you've found that doing essentially a keyword match but using certain keywords as more advanced filters rather than just search terms.
zymhan 7y ago
Because not everyone thinks in search operator keywords.
onion2k 7y ago
Why would you really apply NLP to a search engine though? If you go to Google's homepage and click the microphone at the end of the search input box you can search by speaking. All it does is convert to speech to text, but it implies you might be able to search in a more "natural language" way.
aglionby 7y ago
Depends on your audience, but I imagine many find the answer boxes on Google search pretty useful. Getting the population of a city without having to click any links is probably good for your perceived value. For this you need some NLP tech to extract intent from the query and match it to the right entity in their knowledge graph (in addition to something to help you build the graph in the first place). Google have a blog post from October last year with some more complex examples of where more sophisticated NLP helps https://www.blog.google/products/search/search-language-understanding-bert/ https://www.blog.google/products/search/search-language-unde...
michelpp 7y ago
Because you want to sell shirts without stripes to customers who want them.
- shawnz 7y ago
  I expect most users genuinely looking for such a product would just search "plain shirt"
  - cheschire 7y ago
    That is an inclusive search though. What if the user actually wanted to simply exclude stripes but wanted to see polka dots and paisley?
    - shawnz 7y ago
      I think that is probably not a common enough use case to optimize for. Additionally it would be easy for a user in that position to just search "shirt" and ignore the occasional striped one, or to search "polka dot" and "paisley" seperately
      - PascLeRasc 7y ago
        That sounds like a lot of repetitive searching. I wonder if we could get a computer to do that for us.
        shawnz 7y ago
        I understand the point, but there are plenty of bigger fish that I would want Amazon and Google to fry before spending their engineers' time on a triviality like this. I just don't think that having to make three queries instead of one in this occasional situation is such a big deal.
      - lolc 7y ago
        You're arguing the search is "good enough". Because we can adapt to the machines. But the company who will not force us to do so will get our business. The company who creates the best digital butler will win. They know this. They try hard. And they still fail at simple stuff when judged by humans.
headcanon 7y ago
From a product perspective, I would say there is a reasonable expectation that a customer will provide that query and expect the results to come back with plain shirts. Anything different is a degraded customer experience. Sure, a technical user will understand which queries to provide better, but 90% of customers won't have that skillset. Its our job as engineers to serve those people, and the queries they provide. So NLP is totally a thing you want to have in search. Arguably, its the whole point of search as it exists now.
gravitas 7y ago
Certain query styles which the NLP operators help clarify. Recent personal examples: "is butyl rubber an organic or inorganic compound?" and "which gloves are best for both acetone and nitric acid?"
balnaphone 7y ago
Perhaps it's considered "good enough" because users balk at performing NLP-style queries when they find they almost never work properly.
DenisM 7y ago
5% revenue sales and revenue increase? Why indeed!
uoaei 7y ago
I agree, I think it's a better idea to improve the knowledge management (in terms of a relational hypergraph) before trying to create the bridge between the natural/digital knowledge representations. I have a hunch a good knowledge management system would be more amenable to being queried using natural language since relations are probably constructed similarly to the patterns we see in natural language syntax.
PascLeRasc 7y ago
Agreed. I love how I can go to Men's Wearhouse and tell the salesman: "Shirts" "Polka dot shirts" "Floral shirts" "Wikipedia list of clothing patterns" "Houndstooth shirts"
tsukurimashou 7y ago
Because that's how people use it and kind of marketed as such
notatoad 7y ago
all three of these search engines offer "voice assistant" platforms that encourage you to speak to them in natural language, and send your query to the search engine verbatim under a broad set of circumstances.

Why would you really apply NLP to a search engine though? Generally speaking a weighted keyword search is good enough 95% of the time and requires significantly less resources to perform.

rjurney 7y ago

Because there's something called conceptual search that people are dying for to find what they actually want.

binarymax 7y ago

I work specifically in this field with clients, and deliver training on applying NLP to search. You’d be surprised how effective NLP is for use when identifying query intent, and pulling out modifiers that should apply as metadata filters. Weighted keyword search works a lot, but it fails hard for many long tail queries (especially in e-commerce and other attribute heavy domains). IMO there really isn’t a good excuse for these firms to fail at queries like this. The query itself isn’t particularly difficult when using a decent NLP stack and following well known practices.

skewart 7y ago
If it's technically possible then presumably it's a deliberate product choice to not have better search results for "shirt without stripes". And that seems entirely plausible. Google is already by far the most widely used search engine, so they don't really need to innovate or improve the search product very much in order to attract and retain users. Presumably capturing more advertising spending from the companies paying for ads is a bigger priority. Microsoft under Satya Nadella has been all about enterprise and cloud, and I doubt Bing is a strategic priority any more, so it's not surprising that they wouldn't put a lot of resources into making it better. Amazon is a little surprising. You'd think they'd have a lot to gain from making it easier for people to find what they're looking for. But maybe less than perfect search results are deliberate? Maybe it's like how supermarkets put basic items in the back of the store and high-margin impulse buys in the front - so you have to walk past chocolates and chips if you want to buy a carton of milk. If Amazon is deliberately nerfing search results then maybe Google would stand to benefit from having better shopping-related results - people would get frustrated trying to find a shirt without stripes on Amazon and just use Google instead, letting Google profit from advertising in the process. But maybe people selling shirts aren't willing to pay much for ads, so there isn't much money for Google to make by getting better at finding specific types of shirts. I dunno if any of these conjectures are anywhere near accurate, but it's interesting to think about.
uoaei 7y ago
I feel like that's stretching the definition of NLP though. Technically you are processing natural language but it seems like you've found that doing essentially a keyword match but using certain keywords as more advanced filters rather than just search terms.

zymhan 7y ago

Because not everyone thinks in search operator keywords.

onion2k 7y ago

Why would you really apply NLP to a search engine though? If you go to Google's homepage and click the microphone at the end of the search input box you can search by speaking. All it does is convert to speech to text, but it implies you might be able to search in a more "natural language" way.

aglionby 7y ago

Depends on your audience, but I imagine many find the answer boxes on Google search pretty useful. Getting the population of a city without having to click any links is probably good for your perceived value. For this you need some NLP tech to extract intent from the query and match it to the right entity in their knowledge graph (in addition to something to help you build the graph in the first place). Google have a blog post from October last year with some more complex examples of where more sophisticated NLP helps https://www.blog.google/products/search/search-language-understanding-bert/ https://www.blog.google/products/search/search-language-unde...

michelpp 7y ago

Because you want to sell shirts without stripes to customers who want them.

shawnz 7y ago
I expect most users genuinely looking for such a product would just search "plain shirt"
- cheschire 7y ago
  That is an inclusive search though. What if the user actually wanted to simply exclude stripes but wanted to see polka dots and paisley?
  - shawnz 7y ago
    I think that is probably not a common enough use case to optimize for. Additionally it would be easy for a user in that position to just search "shirt" and ignore the occasional striped one, or to search "polka dot" and "paisley" seperately
    - PascLeRasc 7y ago
      That sounds like a lot of repetitive searching. I wonder if we could get a computer to do that for us.
      - shawnz 7y ago
        I understand the point, but there are plenty of bigger fish that I would want Amazon and Google to fry before spending their engineers' time on a triviality like this. I just don't think that having to make three queries instead of one in this occasional situation is such a big deal.
    - lolc 7y ago
      You're arguing the search is "good enough". Because we can adapt to the machines. But the company who will not force us to do so will get our business. The company who creates the best digital butler will win. They know this. They try hard. And they still fail at simple stuff when judged by humans.

headcanon 7y ago

From a product perspective, I would say there is a reasonable expectation that a customer will provide that query and expect the results to come back with plain shirts. Anything different is a degraded customer experience. Sure, a technical user will understand which queries to provide better, but 90% of customers won't have that skillset. Its our job as engineers to serve those people, and the queries they provide. So NLP is totally a thing you want to have in search. Arguably, its the whole point of search as it exists now.

gravitas 7y ago

Certain query styles which the NLP operators help clarify. Recent personal examples: "is butyl rubber an organic or inorganic compound?" and "which gloves are best for both acetone and nitric acid?"

balnaphone 7y ago

Perhaps it's considered "good enough" because users balk at performing NLP-style queries when they find they almost never work properly.

DenisM 7y ago

5% revenue sales and revenue increase? Why indeed!

uoaei 7y ago

I agree, I think it's a better idea to improve the knowledge management (in terms of a relational hypergraph) before trying to create the bridge between the natural/digital knowledge representations. I have a hunch a good knowledge management system would be more amenable to being queried using natural language since relations are probably constructed similarly to the patterns we see in natural language syntax.

PascLeRasc 7y ago

Agreed. I love how I can go to Men's Wearhouse and tell the salesman: "Shirts" "Polka dot shirts" "Floral shirts" "Wikipedia list of clothing patterns" "Houndstooth shirts"

tsukurimashou 7y ago

Because that's how people use it and kind of marketed as such

notatoad 7y ago

all three of these search engines offer "voice assistant" platforms that encourage you to speak to them in natural language, and send your query to the search engine verbatim under a broad set of circumstances.