6 ms·

Neural Search for medium sized corpora

53 points by raphaelty 5y ago

deleted 5y ago
[deleted]
akie 5y ago
If you're confused (like me) and don't know what "neural search" is, here's an explainer https://medium.com/jina-ai/what-is-jina-and-neural-search-7a9e166608ab https://medium.com/jina-ai/what-is-jina-and-neural-search-7a... (scroll down a bit)
- gk1 5y ago
  “Vector search” is more commonly used, so you might’ve heard it called that. In case not: https://www.pinecone.io/learn/what-is-similarity-search/ https://www.pinecone.io/learn/what-is-similarity-search/
- throwaway81523 5y ago
  That article is very basic and fluffy, but the more technical article that it is based on and links to is excellent, so thanks for (indirectly) pointing to it: https://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End-Product-Search-using-Tensorflow/ https://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End... The other stuff on that guy's (Han Xiao's) blog also looks interesting. I think this neural search stuff explains why we're all getting such low-precision results on Google/DDG by the standards we were used to. But, the claimed advantages are impressive. This is a big change in how search engines work, and I was unfamiliar with it before. I still have reservations about where enough training data is supposed to come from, without already having a big system and user base. But, maybe some more info about that is out there. So this is another thing to look into. Wow.
gk1 5y ago
Great to see another neural search (better known as vector search) option, especially with an end-to-end pipeline. There were 3 good discussions on vector search about a month ago: https://news.ycombinator.com/item?id=29551947 https://news.ycombinator.com/item?id=29551947 - About semantic search (ie, vector search applied to text) https://news.ycombinator.com/item?id=29554986 https://news.ycombinator.com/item?id=29554986 - About vector search at Google https://news.ycombinator.com/item?id=29555780 https://news.ycombinator.com/item?id=29555780 - Open-source vector search index from Facebook
- raphaelty 5y ago
  Thank you for these great resources.
agencies 5y ago
How big is a medium sized corpus? 10m documents?
- raphaelty 5y ago
  I think 10 million documents is a large corpus. A retriever like Sklearn TfIdf will have a hard time handling it in a reasonable time. The main goal of Cherche is to prototype a neural search engine quickly and with a large choice of retrievers and rankers for corpus sizes < 1 million documents which is a common use case in the industry. Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.

53 points by raphaelty 5y ago

deleted 5y ago

[deleted]

akie 5y ago

If you're confused (like me) and don't know what "neural search" is, here's an explainer https://medium.com/jina-ai/what-is-jina-and-neural-search-7a9e166608ab https://medium.com/jina-ai/what-is-jina-and-neural-search-7a... (scroll down a bit)

gk1 5y ago
“Vector search” is more commonly used, so you might’ve heard it called that. In case not: https://www.pinecone.io/learn/what-is-similarity-search/ https://www.pinecone.io/learn/what-is-similarity-search/
throwaway81523 5y ago
That article is very basic and fluffy, but the more technical article that it is based on and links to is excellent, so thanks for (indirectly) pointing to it: https://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End-Product-Search-using-Tensorflow/ https://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End... The other stuff on that guy's (Han Xiao's) blog also looks interesting. I think this neural search stuff explains why we're all getting such low-precision results on Google/DDG by the standards we were used to. But, the claimed advantages are impressive. This is a big change in how search engines work, and I was unfamiliar with it before. I still have reservations about where enough training data is supposed to come from, without already having a big system and user base. But, maybe some more info about that is out there. So this is another thing to look into. Wow.

gk1 5y ago

Great to see another neural search (better known as vector search) option, especially with an end-to-end pipeline. There were 3 good discussions on vector search about a month ago: https://news.ycombinator.com/item?id=29551947 https://news.ycombinator.com/item?id=29551947 - About semantic search (ie, vector search applied to text) https://news.ycombinator.com/item?id=29554986 https://news.ycombinator.com/item?id=29554986 - About vector search at Google https://news.ycombinator.com/item?id=29555780 https://news.ycombinator.com/item?id=29555780 - Open-source vector search index from Facebook

raphaelty 5y ago
Thank you for these great resources.

agencies 5y ago

How big is a medium sized corpus? 10m documents?

raphaelty 5y ago
I think 10 million documents is a large corpus. A retriever like Sklearn TfIdf will have a hard time handling it in a reasonable time. The main goal of Cherche is to prototype a neural search engine quickly and with a large choice of retrievers and rankers for corpus sizes < 1 million documents which is a common use case in the industry. Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.