Quizzical4230

7 ms·

61.
by Quizzical4230 2y ago
Thank you! I shall play with it this weekend :D
62.
by Quizzical4230 2y ago
This is pretty cool. The dot product would give the unnormalized cosine similarity from a smaller pool. Thank you so much!
63.
by Quizzical4230 2y ago
Yes, I think vector search is tricky to navigate at times since now the onus is on the user to explain the problem well. However, you can copy paste full abstracts to get similar papers well enough. I will fix the LaTeX rendering ASAP. Than
64.
by Quizzical4230 2y ago
> Recommend reranking. Will explore it thoroughly then! > MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve. Yes, exactly why I went with this model!
65.
by Quizzical4230 2y ago
Exactly. Full text paradigm has it's own pros and I believe we need those tools in the new vector search to take full advantage. I am planning to add keywords feature where if a user enters something in "quotes", the would ne
66.
by Quizzical4230 2y ago
hey hey hey! XD
67.
by Quizzical4230 2y ago
Amazing! I will do so :D
68.
by Quizzical4230 2y ago
This is an interesting problem to tackle. Added to TODO list! :D
69.
by Quizzical4230 2y ago
So true ;-; I was following the Gmail protocol. I will use > from now on. Happy Holidays :D
70.
by Quizzical4230 2y ago
1. The model size was small enough to process the corpus fast-ish using the limited resources I have. They also support MRL and binary embeddings which help would be helpful in case I need to downsize on the VM size. 2. Close to 500ms. See
71.
by Quizzical4230 2y ago
Thank you for your kind words!
72.
by Quizzical4230 2y ago
That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embed
73.
by Quizzical4230 2y ago
Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different in
74.
by Quizzical4230 2y ago
Perfect! Although, atm I am only using retrieval without any LLM involved. Might try integrating if it significantly improves UX without compromising speeds.
75.
by Quizzical4230 2y ago
paper-qa looks pretty cool. I will do so!
76.
by Quizzical4230 2y ago
Not sure about the semantic search, but preprints are peer reviewed and hence not vetted. However, at the current pace of papers on arXiv (5k+/week) peer review alone might halt the progress.
77.
by Quizzical4230 2y ago
This is quite unique. I believe a custom solution might help you better than Google Scholar.
78.
by Quizzical4230 2y ago
All benefits depend on the ability of the embedding model. Semantic embeddings understand nuances, so they can match abstracts that align conceptually even if no exact keywords overlap. For example, "neural networks" vs. "dee
79.
by Quizzical4230 2y ago
Full text search would be redundant as arXiv.org already supports it. For semantic search, Typesense has limited collection of embedding models. [^1] [1]: https://huggingface.co/typesense/models/tree/main
80.
by Quizzical4230 2y ago
Actually, yeah XD
81.
by Quizzical4230 2y ago
Agreed.
82.
by Quizzical4230 2y ago
Thank you for your kind words. These look like great projects, I will surely check them out :D
83.
by Quizzical4230 2y ago
At the beginning of the project, MixedBread's embedding model was small and leading the MTEB leaderboard [^1], hence I went with it. Yes, I did binarize them for a faster search experience. However, I think the search quality degrades
84.
by Quizzical4230 2y ago
Thank you for the link. Would you know any reliable small model to add on top of vanilla search for a similar experience?
85.
by Quizzical4230 2y ago
I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?
86.
by Quizzical4230 2y ago
Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having mul
87.
by Quizzical4230 2y ago
Most definitely. Thank you for pointing this out!
88.
by Quizzical4230 2y ago
It is more like what triilman commented, but with all components open-source. I plan to add filters soon enough with keywords support! (actually waiting for milvus)
89.
Show HN: I made a website to semantically search ArXiv papers(papermatch.mitanshu.tech)
324 points by Quizzical4230 2y ago|104 comments
90.
How does a DLP projector work?(projectorscreen.com)
2 points by Quizzical4230 2y ago|0 comments

More ›