Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv https://huggingface.co/NeuML/txtai-arxiv) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai (https://github.com/neuml/annotateai https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above. paperai (https://github.com/neuml/paperai https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) paperetl (https://github.com/neuml/paperetl https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.