3 ms·

Semantics Beats Syntax

4 points by reckel 5y ago

PaulHoule 5y ago
So far "semantic search" has been a scam, and I say that having developed a commercial "semantic search" product that was on the market more than 6 years before what's been announced by Bing. Most search engines assign documents to a vector space based on a simple algorithm on words (e.g. BM25) This method was patented in the 1980s https://en.wikipedia.org/wiki/Latent_semantic_analysis https://en.wikipedia.org/wiki/Latent_semantic_analysis which kept people from trying it and realizing that their algorithm wasn't very accurate and didn't scale well to more than 3000 documents or so. This method https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation is more scalable, works better and was never patented. It learns that articles about "chemistry" tend to use words like "proton" and "cyanide" and lumps the into one topic. The search engine I worked on worked a lot like LDA but used a neural network instead of a linear transformation. When we stood up our new patent search engine demo we got a call the next day from the USPTO who wanted to buy a license for it. It was that good, but it was that good because patent searchers often enter a whole paragraph of text describing an invention, and similarity matching makes sense there. Real semantics involve a number of things that are crisp, not approximate, and don't fit the vector model well. In particular, most words can have their meaning reversed by other constructions in the sentence (e.g. "not", "never", "it is not the case that...".) There are enough semantic attributes in a sentence/query that can at best be determined with 80% accuracy individually that when you put them together it's not immediately plausible that you do much better than bag of words in most cases. Newer neural vectors are better than the old but there are fundamental problems in their structure (e.g. the document gets bigger but the vector doesn't!) that prove that current approaches won't solve the semantic problems just as Minsky proved single layer perceptrons can't do much.

4 points by reckel 5y ago

So far "semantic search" has been a scam, and I say that having developed a commercial "semantic search" product that was on the market more than 6 years before what's been announced by Bing. Most search engines assign documents to a vector space based on a simple algorithm on words (e.g. BM25) This method was patented in the 1980s https://en.wikipedia.org/wiki/Latent_semantic_analysis https://en.wikipedia.org/wiki/Latent_semantic_analysis which kept people from trying it and realizing that their algorithm wasn't very accurate and didn't scale well to more than 3000 documents or so. This method https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation is more scalable, works better and was never patented. It learns that articles about "chemistry" tend to use words like "proton" and "cyanide" and lumps the into one topic. The search engine I worked on worked a lot like LDA but used a neural network instead of a linear transformation. When we stood up our new patent search engine demo we got a call the next day from the USPTO who wanted to buy a license for it. It was that good, but it was that good because patent searchers often enter a whole paragraph of text describing an invention, and similarity matching makes sense there. Real semantics involve a number of things that are crisp, not approximate, and don't fit the vector model well. In particular, most words can have their meaning reversed by other constructions in the sentence (e.g. "not", "never", "it is not the case that...".) There are enough semantic attributes in a sentence/query that can at best be determined with 80% accuracy individually that when you put them together it's not immediately plausible that you do much better than bag of words in most cases. Newer neural vectors are better than the old but there are fundamental problems in their structure (e.g. the document gets bigger but the vector doesn't!) that prove that current approaches won't solve the semantic problems just as Minsky proved single layer perceptrons can't do much.