4 ms·
Thanks so much for the detailed feedback. I agree with your high level conclusions: semantic search is an open research area, and there are definitely cases key
by svcrunch 6y ago
Thanks so much for the detailed feedback. I agree with your high level conclusions: semantic search is an open research area, and there are definitely cases keyword search still works better. But even in it's current form, I think it has some commercial applications.
We are working on improving the retrieval algorithms, so hopefully in the future the platform will not fail on some of the examples you provided.
> It seems it is using sentence level embeddings from the text, and does not use title?
The platform actually does use the document title when matching results, as well as selecting a relevant snippet (often sentence-length, but doesn't have to be), if it finds one. However, the snippet might not be good enough to display. For example, if you look at the second search result for the query: "most difficult proofs"
https://quantasearch.club/#most%20difficult%20proofs https://quantasearch.club/#most%20difficult%20proofs
"Pentagon Tiling Proof Solves Century-Old Math Problem"
The document matched, but the snippet wasn't good enough to show, so the introductory sentence of the article is shown instead.
For this particular demo, though, I wanted more fine-grained control of how the title was weighed against the body of the document, so I issued two separate queries to the platform and merged the results. I hope to release that code as part of an upcoming tutorial.
> Can you provide more details about how does the demo work?
Yes, there were a few steps to building it:
1. I gathered the Quanta articles.
2. I converted them into JSON formatted documents. Alternatively, PDF, markdown, and protocol buffers are supported.
3. Within my account on the ZIR platform, I created a corpus to hold the articles.
4. Uploaded the documents (through either the API or drag-and-drop web interface).
The actual demo server is coded in Java.
5. It receives the query from the user, and passes it directly to the platform for searching (Here we actually issue 2 queries, but usually, a single query is sufficient. Can discuss details if you'd like).
6. Platform returns matching document ids and snippets, which are joined to the document body and converted to JSON, which is sent to the browser for rendering.
Hopefully this clarifies the architecture a bit.