2 ms·
Actually, what I was asking about was how to handle my storage and later retrieval of hundreds of papers. I don't mean just the database tool -- I currently use
by SamBam 3y ago
Actually, what I was asking about was how to handle my storage and later retrieval of hundreds of papers. I don't mean just the database tool -- I currently use Zotero and make sure articles are tagged -- but I'm wondering if I can build a system for my articles where I can ask ChatGPT "are there any studies that show X causes Y" and have it use my articles directly, and not give me bullshit citations like it does 90% of the time now.
(To be clear, I'm pleasantly surprised when it does give me some real, useful citations of articles I didn't know about, but I'd like to get it's accuracy way up, at least for articles I have in my possession that I could feed it for context.)
- llmllmllm 3y agoAs part of my testing of https://flowch.ai https://flowch.ai (which we are developing), I've been uploading multiple PDFs (including scientific papers). I then ask a question and ask it to cite its sources as part of the prompt and it'll give the PDFs it's using for the data / quotes it's using. Works well! It's currently free to use :)
- chaxor 3y agoFirst you just extract all of the text from the PDFs. It's helpful to use arxiv instead, since it gives you latex. Then you can parse to separate content from style. Then store that in a a DuckDB database, with zstd compression since it's great on text. Then, just use some encoder model (decoder models are all the rage but they're not necessary here) to process all of these texts into Qdrant. Then use your Qdrant database happily ever after, with whatever you like, such as Vicuna or Guanaco 30b GPTQ. Sprinkle your langcgain or whatever you like. Galpaca 30b may be appropriate for science text.
- jimmySixDOF 3y agoLitmap is a good surfacer of related papers if you have a starting seed to work from https://www.litmaps.com/ https://www.litmaps.com/