7 ms·

Show HN: I made a website to semantically search ArXiv papers

324 points by Quizzical4230 2y ago

As a grad student (and an ADHDer), I had trouble doing literature review systematically. To combat this, I made a website that finds similar papers using the meaning of the thing I am looking for. I used MixedBread's [^1] embedding model to generate vectors from the abstracts. I store and search similar vectors using Milvus [^2] and finally use Gradio [^3] to serve the frontend. I update the vector database weekly by pulling the metadata dataset from Kaggle [^4]. To speed up the search process on my free oracle instance, I binarise the embeddings and use Hamming distance as a metric. I would love your feedback on the site :) Happy Holidays! [1]: https://www.mixedbread.ai/docs/embeddings/mxbai-embed-large-v1 https://www.mixedbread.ai/docs/embeddings/mxbai-embed-large-... [2]: https://milvus.io/ https://milvus.io/ [3]: https://www.gradio.app/ https://www.gradio.app/ [4]: https://www.kaggle.com/datasets/Cornell-University/arxiv https://www.kaggle.com/datasets/Cornell-University/arxiv

shigeru94 2y ago
Is this similar to https://www.semanticscholar.org https://www.semanticscholar.org (from Allen Institute for AI) ?
- triilman 2y ago
  I think more like this website https://arxivxplorer.com/ https://arxivxplorer.com/
- Quizzical4230 2y ago
  It is more like what triilman commented, but with all components open-source. I plan to add filters soon enough with keywords support! (actually waiting for milvus)
lgas 2y ago
This might've saved you some time: https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv
- cluckindan 2y ago
  The dataset there is almost a year old.
  - dmezzetti 2y ago
    It was just updated last week. The dataset page on HF only has the scripts, the raw data resides over on Kaggle.
- Quizzical4230 2y ago
  Actually, yeah XD
shishy 2y ago
I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews includes besides search (used to work at scite.ai)
- Quizzical4230 2y ago
  Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. This sounds like a real issue in expanding the coverage. | Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? I did, but maybe not thoroughly enough. I will check these and add complementing features. | You might consider what else a dedicated product workflow for lit reviews includes besides search Do you mean a reference management system like Mendeley/Zotero? [1]: https://papermatchbio.mitanshu.tech/ https://papermatchbio.mitanshu.tech/ [2]: https://papermatchmed.mitanshu.tech/ https://papermatchmed.mitanshu.tech/
  - eric-burel 2y ago
    Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular). A "dedicated product workflow" could be about taking business use cases like that into account. This is a real business problem, the Google Scholar lock up is annoying and I would pay for something better than what exists.
mrjay42 2y ago
I think you have an encoding problem <3 If you search for "UPC high performance computing evaluation", you'll see paper with buggy characters in the authors name (second results with that search).
- Quizzical4230 2y ago
  Most definitely. Thank you for pointing this out!
antman 2y ago
Nice work. Any other technical comments, why did you use those embeddings, did you binarzue them, did you use any dpecial prompts?
- Quizzical4230 2y ago
  At the beginning of the project, MixedBread's embedding model was small and leading the MTEB leaderboard [^1], hence I went with it. Yes, I did binarize them for a faster search experience. However, I think the search quality degrades significantly after the first 10 results, which are same as fp32 search but with a shuffled order. I am planning to add a reranking strategy to boost better results upwards. At the moment, this is plain search with no special prompts. [1]: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
bubaumba 2y ago
This is cool, but how about local semantic search through tens of thousands articles and books. Sure I'm not the first, there should be some tools already.
- Quizzical4230 2y ago
  I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?
  - bubaumba 2y ago
    Absolutely worth doing. Here is interesting related video, local RAG: https://www.youtube.com/watch?v=bq1Plo2RhYI https://www.youtube.com/watch?v=bq1Plo2RhYI I'm not an expert, but I'll do it for learning. Then open source if it works. As far as I understand this approach requires a vector database and LLM which doesn't have to be big. Technically it can be implemented as local web server. Should be easy to use, just type and get a sorted by relevance list.
    - Quizzical4230 2y ago
      Perfect! Although, atm I am only using retrieval without any LLM involved. Might try integrating if it significantly improves UX without compromising speeds.
- ttpphd 2y ago
  Try Semantra https://github.com/freedmand/semantra https://github.com/freedmand/semantra
dmezzetti 2y ago
Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai (https://github.com/neuml/annotateai https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above. paperai (https://github.com/neuml/paperai https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) paperetl (https://github.com/neuml/paperetl https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.
- shishy 2y ago
  paperetl is cool, saving that for later, nice! did something similar in-house with grobid in the past (great project by patrice).
  - dmezzetti 2y ago
    Grobid is great. paperetl is the workhorse of the projects mentioned above. Good ole programming and multiprocessing to churn through data.
- Quizzical4230 2y ago
  Thank you for your kind words. These look like great projects, I will surely check them out :D
namanyayg 2y ago
What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp. Some of the current ideas I had: 1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc. I don't know if either are good enough business problems worth solving tho.
- bubaumba 2y ago
  3. Quick lookup into internal documents. Almost any company needs it. Navigating file-system like hierarchy is slow and limited. That was old way. 4. Quick lookup into the code to find relevant parts even when the wording in comments is different.
  - imadethis 2y ago
    For 4, it would be neat to first pass each block of code (function or class or whatever) through an llm to extract meaning, and then embed some combination of llm parsed meaning, docstring and comments, and function name. Then do semantic search against that. That way you’d cover what the human thinks the block is for vs what an LLM “thinks” it’s for. Should cover some amount of drift in names and comments that any codebase sees.
- jondwillis 2y ago
  Please stop making ad tech better. Someone else might, but you don’t have to.
realvinifabbro 2y ago
[dead]
ukuina 2y ago
Related: emergentmind.com
- Quizzical4230 2y ago
  Thank you for the link. Would you know any reliable small model to add on top of vanilla search for a similar experience?
tokai 2y ago
Nice but I have to point out that a systematic review cannot be done with semantic search and should never be done in a preprint collection.
- Quizzical4230 2y ago
  Agreed.
- dmezzetti 2y ago
  Why?
  - Quizzical4230 2y ago
    Not sure about the semantic search, but preprints are peer reviewed and hence not vetted. However, at the current pace of papers on arXiv (5k+/week) peer review alone might halt the progress.
    - dmezzetti 2y ago
      Why not semantic search was the bigger question.
    - OutOfHere 2y ago
      You mean to say that preprints are not peer reviewed.
- WolfOliver 2y ago
  but it can provide recommendations
omarhaneef 2y ago
For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?
- Quizzical4230 2y ago
  All benefits depend on the ability of the embedding model. Semantic embeddings understand nuances, so they can match abstracts that align conceptually even if no exact keywords overlap. For example, "neural networks" vs. "deep learning." can and should fetch similar papers. Subjectively, yes. I sent this around my peers and they said it helped them find new authors/papers in the field while preparing their manuscripts. | Is this more useful in certain domains? I don't think I have the capacity to comment on this.
- feznyng 2y ago
  One of the factors is how users phrase their queries. On some level people are used to full text search but semantic shines when they ask literal questions with terminology that may not match the answer.
  - woodson 2y ago
    Query keyword expansion works quite well for that without semantic search (although it can reduce precision).
  - Quizzical4230 2y ago
    Exactly. Full text paradigm has it's own pros and I believe we need those tools in the new vector search to take full advantage. I am planning to add keywords feature where if a user enters something in "quotes", the would need to be in the shown results. Just like you can do with a google search.
    - feznyng 2y ago
      You might be interested in hybrid search which issues both a full text and semantic search and then merges the results via reciprocal rank fusion.
gaborme 2y ago
Nice. Why not use a full-text search like self-hosted Typesense?
- Quizzical4230 2y ago
  Full text search would be redundant as arXiv.org already supports it. For semantic search, Typesense has limited collection of embedding models. [^1] [1]: https://huggingface.co/typesense/models/tree/main https://huggingface.co/typesense/models/tree/main
mskar 2y ago
This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa (https://github.com/Future-House/paper-qa https://github.com/Future-House/paper-qa). Then paper-qa users would be able to use your semantic search as part of its workflow.
- Quizzical4230 2y ago
  paper-qa looks pretty cool. I will do so!
- OutOfHere 2y ago
  I advise against it since binarized hamming distance isn't exactly that good unless your vector length is say a million.
  - Quizzical4230 2y ago
    I have the fp32 embeddings saved. It is for the website that I use binarised ones to combat latency.
andai 2y ago
Did you notice a difference in performance after binarization? Do you have a way to measure performance?
- Quizzical4230 2y ago
  Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different indexes used for the search. [^2] [1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/heads/main/search_time_vs_input_type_auto_legend.png https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he... [2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvus-visual-guide https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...
  - binarymax 2y ago
    That looks great for speed, but what about recall?
    - Quizzical4230 2y ago
      That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
      - intalentive 2y ago
        Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…) MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
        Quizzical4230 2y ago
madbutcode 2y ago
This looks great! I have used the biorXiv version of papermatch and it gives pretty good results!
- Quizzical4230 2y ago
  Thank you for your kind words!
Maro 2y ago
Very cool! Add a "similar papers" link to each paper, that will make this the obvious way to discover topics by clicking along the similar papers.
- Quizzical4230 2y ago
  Amazing! I will do so :D
swyx 2y ago
1. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 https://arxiv.org/abs/2412.09871 <- which is more recent, more people might want also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance.
- Quizzical4230 2y ago
  1. The model size was small enough to process the corpus fast-ish using the limited resources I have. They also support MRL and binary embeddings which help would be helpful in case I need to downsize on the VM size. 2. Close to 500ms. See [^1]. 3. This [^2] was the reason I went with milvus. I also assumed that more stars would result in a bigger community and hence faster bug discovery and fixes. And better feature support. 4. Yes, I automated the weekly pull here [^3]. Since I am constrained on resources available, I used HuggingFace Spaces to do the automation for me :) Although, the space keeps sleeping and to avoid that, I am planning keep calling the same space using api/gradio_client. Let's see how that goes. | which is more recent, more people might want Absolutely agree. I am planning to add a 'Recency' sorting option for the same. It should balance between similarity and the date published. | also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance. Oh, I will surely look into it. Thank you so much for a detailed response. :D [1]: https://news.ycombinator.com/item?id=42507116#42509636 https://news.ycombinator.com/item?id=42507116#42509636 [2]: https://benchmark.vectorview.ai/vectordbs.html https://benchmark.vectorview.ai/vectordbs.html [3]: https://huggingface.co/spaces/bluuebunny/update_arxiv_embeddings https://huggingface.co/spaces/bluuebunny/update_arxiv_embedd...
  - swyx 2y ago
    my pleasure, thank you for the reply! ive never used milvus or heard of mixbread so this was refreshing.
maCDzP 2y ago
I want to crawl and plug in scihib to this and see what happens.
fasa99 2y ago
For what it's worth, back in the day (a few years ago, before the LLM boom a few years) I found on a similar sized vector database (gensim / doc2vec), it's possible to just brute force a vector search e.g. with SSE or AVX type instructions. You can code it in C and have a python API. Your data appears to be a few gigs so that's feasible for realtime CPU brute force, <200 ms
- Quizzical4230 2y ago
  This is an interesting problem to tackle. Added to TODO list! :D
amelius 2y ago
Great procrastination project :)
- Quizzical4230 2y ago
  hey hey hey! XD
venice_benice 2y ago
interesting project; I’m not really sure how useful it is for field-specific stuff—I'm searching for “image reduction astronomy”, and it shows all sorts of related but not image-reduction work (including noise reduction which is not the same thing). I’m not really familiar with vector search enough to evaluate it well enough. However I can give you the heads-up that the abstracts don't render well because (La)TeX is interpreted as markdown so that Paper~1 shows something and Paper~2 shows something else will strikethrough the text between the tildes (whereas they are meant to be non-breaking spaces). Similarly for the backtick which makes text monospaced in the rendered output but is simply supposed to be the opening quote.
- Quizzical4230 2y ago
  Yes, I think vector search is tricky to navigate at times since now the onus is on the user to explain the problem well. However, you can copy paste full abstracts to get similar papers well enough. I will fix the LaTeX rendering ASAP. Thank you for trying out the site! Happy Holidays :D
underlines 2y ago
hint: 8 days ago txtai released their arxiv embeddings https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv
- Quizzical4230 2y ago
  Yes!
serial_dev 2y ago
I tried a simple search by author and it didn’t work. All the fancy stuff is great, but I’d expect the basics still work, in the end it’s a search engine for papers.
- wodenokoto 2y ago
  Maybe use the right tool for the job? Author names generally don’t have a lot of semantics associated with them and definitely not in the abstract.
zerop 2y ago
This looks great, thanks for building this. Something on similar lines which many may link, Research Rabbit - https://www.researchrabbit.ai/ https://www.researchrabbit.ai/
- Quizzical4230 2y ago
  I am glad you liked it! I wanted PaperMatch to be open-source so that the users can understand the workflow behind it and hack it to their advantage instead of grumbling away when the results aren't to their liking.
OutOfHere 2y ago
Instead of using binarized hamming, why not just use a shorter embedding that you can properly tackle? What good is Milvus if it's not giving you matches using something more proper? Also, this site is not Reddit. You don't have to reply to every comment.
- Quizzical4230 2y ago
  > Also, this site is not Reddit. You don't have to reply to every comment. I am so conflicted whether to reply to this comment or not Xp Jokes apart, Mxbai model + Milvus gives fantastic results in fp32, however it's the latency that is an issue here. I could try chopping the fp32 vectors in half without binarizing to see. Thanks!
curious_cat_163 2y ago
This is great! I just tried some queries and the results were pretty decent, in terms of semantics. But, just thinking of it as a user, if this were to be part of my daily workflow (instead of say something like Google Scholar), I would like: 1. The option to somehow see _how_ the paper was reviewed and/or cited, if at all. There are things like OpenReview, see example [1] 2. The ability to "tell me a story to get up to speed" about a collection of papers. Generative models could help here -- but essentially, I want this thing to be able to write a paragraph for what one might find in the literature review / related work of a paper, with citations. :-) All the best! [1] https://openreview.net/forum?id=jhKbnNhwhc https://openreview.net/forum?id=jhKbnNhwhc
- Quizzical4230 2y ago
  1. I was not aware of OpenReview. I love the transparency and would definitely look into integrating it. 2. This is good feedback, making models write the Introduction section! I was planning to keep this search engine a little more traditional, however if the results are good, then it should be the way forward. Thank you, Happy Holidays! :D
  - odyssey7 2y ago
    I have to second the idea, having hacked together something similar myself, to help me complete a literature review——a literature review that I wasn’t planning to publish. Simply generating summaries or pulling key quotes, paper by paper, wasn’t sufficient to be able to understand the topic in the way I wanted to for writing the literature review. In the end, the system would process a collection of hundreds of PDFs that might be related, generate summaries of what they mentioned about the topic in question, and, importantly, was also prompted to note anything about how the insights built upon or were related to insights from previous research, and the motivations behind developing that insight / the challenge it was attempting to solve and whether it was successful. This worked well enough to reduce what might have been weeks worth of work to just a few hours. Genuinely, I believe that research in the near future could look a lot different from what it looks like today.
deleted 2y ago
[deleted]
kouteiheika 2y ago
Feedback: first thing I tried is searching for "leaky relu" and I got a bunch of results related to fluids, which is... not very relevant. (: Compare that to scholar which returns all relevant results: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leaky+relu https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leak... You might want to retrain/finetune your own embedding model instead of using a general-purpose one.
- Quizzical4230 2y ago
  Thank you for taking the time to try out the site! Google scholar scholar is a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning. Here is an alternative approach: Take one paper that you like, copy the abstract from Google Scholar and paste it in PaperMatch. This should help you find similar papers.
ProofHouse 2y ago
I couuld and really use this, but it didn't work for me. And HAS to have a date filter. That is a must maybe with some time based pre-option defaults like HackerNews. Good luck, want to try again when it works. Good idea
- Quizzical4230 2y ago
  They are definitely planned to be integrated very soon! I probably should have waited to post on HN untill that. I will ping you once the features are live. Thanks for trying out the site!
zzyzek 2y ago
This seems like a cool idea, thanks for creating it! Some feedback: I tried searching for "wave function collapse algorithm", "gumin wave function collapse", "wfc" and "model synthesis" without any relevant hits to the area of research I was interested in. I got a lot of quantum computing and other physics related papers. The "WFC algorithm" overloaded the term (and has nothing to do with quantum mechanics) so it's kind of a bad case for this type of search. Model synthesis is way too generic, so again, might be a bad case for this. The first page of results using "wave function collapse algorithm" from arXiv itself gives relevant results.
- Quizzical4230 2y ago
  Thank you for taking the time to try out the site! arXiv has a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning. Here is an alternative approach: Take one paper that you like, copy the abstract from arXiv (or arXiv ID) and paste it in PaperMatch. This should help you find similar papers.
  - zzyzek 2y ago
    Very nice! Putting in an arXiv ID looks to produce many results that are much more relevant. EDIT: You should provide this in an "information"/"about"/"how to use" dialogue or page to help people use the tool better.
    - Quizzical4230 2y ago
      Thank you! I agree, since this site has the same interface, people expect it to work the same way. Which I was going for but didn't realise the cons of it. I will add an about section!
higty 2y ago
It sounds nice. How do you evaluate the performance of your way against usual embedding?
- Quizzical4230 2y ago
  By assuming "usual embedding" meaning using the default model, which generally is "all-MiniLM-L6-v2", I used MixedBread's embedding model because of this [^1]. You can evaluate how well a model is doing by subjectively going through some search results for papers you have a good grasp on. Another way I look at is to see the 2D "maps" of the embeddings and how well these are segregated, see [^2]. [1]: https://www.mixedbread.ai/blog/binary-mrl https://www.mixedbread.ai/blog/binary-mrl [2]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/heads/main/2024_kde_soft.png https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he...
cryptonector 2y ago
This is really awesome. Thank you!
- Quizzical4230 2y ago
  I am glad you liked it! <3

324 points by Quizzical4230 2y ago

shigeru94 2y ago

Is this similar to https://www.semanticscholar.org https://www.semanticscholar.org (from Allen Institute for AI) ?

triilman 2y ago
I think more like this website https://arxivxplorer.com/ https://arxivxplorer.com/
Quizzical4230 2y ago
It is more like what triilman commented, but with all components open-source. I plan to add filters soon enough with keywords support! (actually waiting for milvus)

lgas 2y ago

This might've saved you some time: https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv

cluckindan 2y ago
The dataset there is almost a year old.
- dmezzetti 2y ago
  It was just updated last week. The dataset page on HF only has the scripts, the raw data resides over on Kaggle.
Quizzical4230 2y ago
Actually, yeah XD

shishy 2y ago

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews includes besides search (used to work at scite.ai)

Quizzical4230 2y ago
Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. This sounds like a real issue in expanding the coverage. | Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? I did, but maybe not thoroughly enough. I will check these and add complementing features. | You might consider what else a dedicated product workflow for lit reviews includes besides search Do you mean a reference management system like Mendeley/Zotero? [1]: https://papermatchbio.mitanshu.tech/ https://papermatchbio.mitanshu.tech/ [2]: https://papermatchmed.mitanshu.tech/ https://papermatchmed.mitanshu.tech/
- eric-burel 2y ago
  Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular). A "dedicated product workflow" could be about taking business use cases like that into account. This is a real business problem, the Google Scholar lock up is annoying and I would pay for something better than what exists.

mrjay42 2y ago

I think you have an encoding problem <3 If you search for "UPC high performance computing evaluation", you'll see paper with buggy characters in the authors name (second results with that search).

Quizzical4230 2y ago
Most definitely. Thank you for pointing this out!

antman 2y ago

Nice work. Any other technical comments, why did you use those embeddings, did you binarzue them, did you use any dpecial prompts?

Quizzical4230 2y ago
At the beginning of the project, MixedBread's embedding model was small and leading the MTEB leaderboard [^1], hence I went with it. Yes, I did binarize them for a faster search experience. However, I think the search quality degrades significantly after the first 10 results, which are same as fp32 search but with a shuffled order. I am planning to add a reranking strategy to boost better results upwards. At the moment, this is plain search with no special prompts. [1]: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard

bubaumba 2y ago

This is cool, but how about local semantic search through tens of thousands articles and books. Sure I'm not the first, there should be some tools already.

Quizzical4230 2y ago
I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?
- bubaumba 2y ago
  Absolutely worth doing. Here is interesting related video, local RAG: https://www.youtube.com/watch?v=bq1Plo2RhYI https://www.youtube.com/watch?v=bq1Plo2RhYI I'm not an expert, but I'll do it for learning. Then open source if it works. As far as I understand this approach requires a vector database and LLM which doesn't have to be big. Technically it can be implemented as local web server. Should be easy to use, just type and get a sorted by relevance list.
  - Quizzical4230 2y ago
    Perfect! Although, atm I am only using retrieval without any LLM involved. Might try integrating if it significantly improves UX without compromising speeds.
ttpphd 2y ago
Try Semantra https://github.com/freedmand/semantra https://github.com/freedmand/semantra

dmezzetti 2y ago

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai (https://github.com/neuml/annotateai https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above. paperai (https://github.com/neuml/paperai https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) paperetl (https://github.com/neuml/paperetl https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.

shishy 2y ago
paperetl is cool, saving that for later, nice! did something similar in-house with grobid in the past (great project by patrice).
- dmezzetti 2y ago
  Grobid is great. paperetl is the workhorse of the projects mentioned above. Good ole programming and multiprocessing to churn through data.
Quizzical4230 2y ago
Thank you for your kind words. These look like great projects, I will surely check them out :D

namanyayg 2y ago

What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp. Some of the current ideas I had: 1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc. I don't know if either are good enough business problems worth solving tho.

bubaumba 2y ago
3. Quick lookup into internal documents. Almost any company needs it. Navigating file-system like hierarchy is slow and limited. That was old way. 4. Quick lookup into the code to find relevant parts even when the wording in comments is different.
- imadethis 2y ago
  For 4, it would be neat to first pass each block of code (function or class or whatever) through an llm to extract meaning, and then embed some combination of llm parsed meaning, docstring and comments, and function name. Then do semantic search against that. That way you’d cover what the human thinks the block is for vs what an LLM “thinks” it’s for. Should cover some amount of drift in names and comments that any codebase sees.
jondwillis 2y ago
Please stop making ad tech better. Someone else might, but you don’t have to.

realvinifabbro 2y ago

[dead]

ukuina 2y ago

Related: emergentmind.com

Quizzical4230 2y ago
Thank you for the link. Would you know any reliable small model to add on top of vanilla search for a similar experience?

tokai 2y ago

Nice but I have to point out that a systematic review cannot be done with semantic search and should never be done in a preprint collection.

Quizzical4230 2y ago
Agreed.
dmezzetti 2y ago
Why?
- Quizzical4230 2y ago
  Not sure about the semantic search, but preprints are peer reviewed and hence not vetted. However, at the current pace of papers on arXiv (5k+/week) peer review alone might halt the progress.
  - dmezzetti 2y ago
    Why not semantic search was the bigger question.
  - OutOfHere 2y ago
    You mean to say that preprints are not peer reviewed.
WolfOliver 2y ago
but it can provide recommendations

omarhaneef 2y ago

For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?

Quizzical4230 2y ago
All benefits depend on the ability of the embedding model. Semantic embeddings understand nuances, so they can match abstracts that align conceptually even if no exact keywords overlap. For example, "neural networks" vs. "deep learning." can and should fetch similar papers. Subjectively, yes. I sent this around my peers and they said it helped them find new authors/papers in the field while preparing their manuscripts. | Is this more useful in certain domains? I don't think I have the capacity to comment on this.
feznyng 2y ago
One of the factors is how users phrase their queries. On some level people are used to full text search but semantic shines when they ask literal questions with terminology that may not match the answer.
- woodson 2y ago
  Query keyword expansion works quite well for that without semantic search (although it can reduce precision).
- Quizzical4230 2y ago
  Exactly. Full text paradigm has it's own pros and I believe we need those tools in the new vector search to take full advantage. I am planning to add keywords feature where if a user enters something in "quotes", the would need to be in the shown results. Just like you can do with a google search.
  - feznyng 2y ago
    You might be interested in hybrid search which issues both a full text and semantic search and then merges the results via reciprocal rank fusion.

gaborme 2y ago

Nice. Why not use a full-text search like self-hosted Typesense?

Quizzical4230 2y ago
Full text search would be redundant as arXiv.org already supports it. For semantic search, Typesense has limited collection of embedding models. [^1] [1]: https://huggingface.co/typesense/models/tree/main https://huggingface.co/typesense/models/tree/main

mskar 2y ago

This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa (https://github.com/Future-House/paper-qa https://github.com/Future-House/paper-qa). Then paper-qa users would be able to use your semantic search as part of its workflow.

Quizzical4230 2y ago
paper-qa looks pretty cool. I will do so!
OutOfHere 2y ago
I advise against it since binarized hamming distance isn't exactly that good unless your vector length is say a million.
- Quizzical4230 2y ago
  I have the fp32 embeddings saved. It is for the website that I use binarised ones to combat latency.

andai 2y ago

Did you notice a difference in performance after binarization? Do you have a way to measure performance?

Quizzical4230 2y ago
Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different indexes used for the search. [^2] [1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/heads/main/search_time_vs_input_type_auto_legend.png https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he... [2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvus-visual-guide https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...
- binarymax 2y ago
  That looks great for speed, but what about recall?
  - Quizzical4230 2y ago
    That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
    - intalentive 2y ago
      Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…) MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
      - Quizzical4230 2y ago

madbutcode 2y ago

This looks great! I have used the biorXiv version of papermatch and it gives pretty good results!

Quizzical4230 2y ago
Thank you for your kind words!

Maro 2y ago

Very cool! Add a "similar papers" link to each paper, that will make this the obvious way to discover topics by clicking along the similar papers.

Quizzical4230 2y ago
Amazing! I will do so :D

swyx 2y ago

1. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 https://arxiv.org/abs/2412.09871 <- which is more recent, more people might want also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance.

Quizzical4230 2y ago
1. The model size was small enough to process the corpus fast-ish using the limited resources I have. They also support MRL and binary embeddings which help would be helpful in case I need to downsize on the VM size. 2. Close to 500ms. See [^1]. 3. This [^2] was the reason I went with milvus. I also assumed that more stars would result in a bigger community and hence faster bug discovery and fixes. And better feature support. 4. Yes, I automated the weekly pull here [^3]. Since I am constrained on resources available, I used HuggingFace Spaces to do the automation for me :) Although, the space keeps sleeping and to avoid that, I am planning keep calling the same space using api/gradio_client. Let's see how that goes. | which is more recent, more people might want Absolutely agree. I am planning to add a 'Recency' sorting option for the same. It should balance between similarity and the date published. | also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance. Oh, I will surely look into it. Thank you so much for a detailed response. :D [1]: https://news.ycombinator.com/item?id=42507116#42509636 https://news.ycombinator.com/item?id=42507116#42509636 [2]: https://benchmark.vectorview.ai/vectordbs.html https://benchmark.vectorview.ai/vectordbs.html [3]: https://huggingface.co/spaces/bluuebunny/update_arxiv_embeddings https://huggingface.co/spaces/bluuebunny/update_arxiv_embedd...
- swyx 2y ago
  my pleasure, thank you for the reply! ive never used milvus or heard of mixbread so this was refreshing.

maCDzP 2y ago

I want to crawl and plug in scihib to this and see what happens.

fasa99 2y ago

For what it's worth, back in the day (a few years ago, before the LLM boom a few years) I found on a similar sized vector database (gensim / doc2vec), it's possible to just brute force a vector search e.g. with SSE or AVX type instructions. You can code it in C and have a python API. Your data appears to be a few gigs so that's feasible for realtime CPU brute force, <200 ms

Quizzical4230 2y ago
This is an interesting problem to tackle. Added to TODO list! :D

amelius 2y ago

Great procrastination project :)

Quizzical4230 2y ago
hey hey hey! XD

venice_benice 2y ago

interesting project; I’m not really sure how useful it is for field-specific stuff—I'm searching for “image reduction astronomy”, and it shows all sorts of related but not image-reduction work (including noise reduction which is not the same thing). I’m not really familiar with vector search enough to evaluate it well enough. However I can give you the heads-up that the abstracts don't render well because (La)TeX is interpreted as markdown so that Paper~1 shows something and Paper~2 shows something else will strikethrough the text between the tildes (whereas they are meant to be non-breaking spaces). Similarly for the backtick which makes text monospaced in the rendered output but is simply supposed to be the opening quote.

Quizzical4230 2y ago
Yes, I think vector search is tricky to navigate at times since now the onus is on the user to explain the problem well. However, you can copy paste full abstracts to get similar papers well enough. I will fix the LaTeX rendering ASAP. Thank you for trying out the site! Happy Holidays :D

underlines 2y ago

hint: 8 days ago txtai released their arxiv embeddings https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv

Quizzical4230 2y ago
Yes!

serial_dev 2y ago

I tried a simple search by author and it didn’t work. All the fancy stuff is great, but I’d expect the basics still work, in the end it’s a search engine for papers.

wodenokoto 2y ago
Maybe use the right tool for the job? Author names generally don’t have a lot of semantics associated with them and definitely not in the abstract.

zerop 2y ago

This looks great, thanks for building this. Something on similar lines which many may link, Research Rabbit - https://www.researchrabbit.ai/ https://www.researchrabbit.ai/

Quizzical4230 2y ago
I am glad you liked it! I wanted PaperMatch to be open-source so that the users can understand the workflow behind it and hack it to their advantage instead of grumbling away when the results aren't to their liking.

OutOfHere 2y ago

Instead of using binarized hamming, why not just use a shorter embedding that you can properly tackle? What good is Milvus if it's not giving you matches using something more proper? Also, this site is not Reddit. You don't have to reply to every comment.

Quizzical4230 2y ago
> Also, this site is not Reddit. You don't have to reply to every comment. I am so conflicted whether to reply to this comment or not Xp Jokes apart, Mxbai model + Milvus gives fantastic results in fp32, however it's the latency that is an issue here. I could try chopping the fp32 vectors in half without binarizing to see. Thanks!

curious_cat_163 2y ago

This is great! I just tried some queries and the results were pretty decent, in terms of semantics. But, just thinking of it as a user, if this were to be part of my daily workflow (instead of say something like Google Scholar), I would like: 1. The option to somehow see _how_ the paper was reviewed and/or cited, if at all. There are things like OpenReview, see example [1] 2. The ability to "tell me a story to get up to speed" about a collection of papers. Generative models could help here -- but essentially, I want this thing to be able to write a paragraph for what one might find in the literature review / related work of a paper, with citations. :-) All the best! [1] https://openreview.net/forum?id=jhKbnNhwhc https://openreview.net/forum?id=jhKbnNhwhc

Quizzical4230 2y ago
1. I was not aware of OpenReview. I love the transparency and would definitely look into integrating it. 2. This is good feedback, making models write the Introduction section! I was planning to keep this search engine a little more traditional, however if the results are good, then it should be the way forward. Thank you, Happy Holidays! :D
- odyssey7 2y ago
  I have to second the idea, having hacked together something similar myself, to help me complete a literature review——a literature review that I wasn’t planning to publish. Simply generating summaries or pulling key quotes, paper by paper, wasn’t sufficient to be able to understand the topic in the way I wanted to for writing the literature review. In the end, the system would process a collection of hundreds of PDFs that might be related, generate summaries of what they mentioned about the topic in question, and, importantly, was also prompted to note anything about how the insights built upon or were related to insights from previous research, and the motivations behind developing that insight / the challenge it was attempting to solve and whether it was successful. This worked well enough to reduce what might have been weeks worth of work to just a few hours. Genuinely, I believe that research in the near future could look a lot different from what it looks like today.

deleted 2y ago

[deleted]

kouteiheika 2y ago

Feedback: first thing I tried is searching for "leaky relu" and I got a bunch of results related to fluids, which is... not very relevant. (: Compare that to scholar which returns all relevant results: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leaky+relu https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leak... You might want to retrain/finetune your own embedding model instead of using a general-purpose one.

Quizzical4230 2y ago
Thank you for taking the time to try out the site! Google scholar scholar is a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning. Here is an alternative approach: Take one paper that you like, copy the abstract from Google Scholar and paste it in PaperMatch. This should help you find similar papers.

ProofHouse 2y ago

I couuld and really use this, but it didn't work for me. And HAS to have a date filter. That is a must maybe with some time based pre-option defaults like HackerNews. Good luck, want to try again when it works. Good idea

Quizzical4230 2y ago
They are definitely planned to be integrated very soon! I probably should have waited to post on HN untill that. I will ping you once the features are live. Thanks for trying out the site!

zzyzek 2y ago

This seems like a cool idea, thanks for creating it! Some feedback: I tried searching for "wave function collapse algorithm", "gumin wave function collapse", "wfc" and "model synthesis" without any relevant hits to the area of research I was interested in. I got a lot of quantum computing and other physics related papers. The "WFC algorithm" overloaded the term (and has nothing to do with quantum mechanics) so it's kind of a bad case for this type of search. Model synthesis is way too generic, so again, might be a bad case for this. The first page of results using "wave function collapse algorithm" from arXiv itself gives relevant results.

Quizzical4230 2y ago
Thank you for taking the time to try out the site! arXiv has a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning. Here is an alternative approach: Take one paper that you like, copy the abstract from arXiv (or arXiv ID) and paste it in PaperMatch. This should help you find similar papers.
- zzyzek 2y ago
  Very nice! Putting in an arXiv ID looks to produce many results that are much more relevant. EDIT: You should provide this in an "information"/"about"/"how to use" dialogue or page to help people use the tool better.
  - Quizzical4230 2y ago
    Thank you! I agree, since this site has the same interface, people expect it to work the same way. Which I was going for but didn't realise the cons of it. I will add an about section!

higty 2y ago

It sounds nice. How do you evaluate the performance of your way against usual embedding?

Quizzical4230 2y ago
By assuming "usual embedding" meaning using the default model, which generally is "all-MiniLM-L6-v2", I used MixedBread's embedding model because of this [^1]. You can evaluate how well a model is doing by subjectively going through some search results for papers you have a good grasp on. Another way I look at is to see the 2D "maps" of the embeddings and how well these are segregated, see [^2]. [1]: https://www.mixedbread.ai/blog/binary-mrl https://www.mixedbread.ai/blog/binary-mrl [2]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/heads/main/2024_kde_soft.png https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he...

cryptonector 2y ago

This is really awesome. Thank you!

Quizzical4230 2y ago
I am glad you liked it! <3