3 ms·
I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing
by rbanffy 2y ago
I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful.
I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything, subtitles from movies and lyrics for songs, and add that to a NAS appliance, it’d be a killer.
- andyfilms1 2y agoEvernote will do this, you can feed it a bunch of PDFs and other documents, it will OCR them and make them all searchable. it's not perfect, but you can also add manual tags for things you know are important.
- rbanffy 2y agoAt some point I'd love to further train an LLM on all my PDFs and be able to ask it questions.
- Jzush 2y agoFor txt based pdfs, doc files etc, this is super easy to do and requires very little technical expertise or configuration. Download the gpt4all client. Pick an LLM the new meta llama3 model works really well then configure the local documents plugin. I wrote a scraper to download all of the California EdCode from the governments site, convert them all to txt docs and I can ask questions about California EdCode in plain English. I work in a shared governance capacity that requires us to refer to the Ed code for contractual negotiations and it’s been extremely helpful.
- kordlessagain 2y agoI have most of this code for doing this - just needs to get rewritten for local storage (I was running it on Google Cloud). Need to pick something that doesn't run Solr as a service for local use. With Ollama, we have function calls running, so should be doable. I was also thinking about using the Open WebUI for use.
- glompers 2y agoDEVONthink 3 [0] (Apple only) will do most of that although I don't keep up at all with its interoperability with LLM extensions. [0] https://www.devontechnologies.com/apps/devonthink https://www.devontechnologies.com/apps/devonthink
- Bluestein 2y ago> extract text descriptions from photos and movies as well as indexing metadata for everything, subtitles from movies and lyrics for songs The big AI players are probably already scraping the bottom of this "barrel" in their search for training data, I am sure ...
- theolivenbaum 2y agoThat's what our app does: curiosity.ai, local index, support for many files types and apps out of the box, and integrated local OCR, STT and even local LLM
- pdw 2y ago> similar thing that can do OCR on scanned docs It's only part of what you want, but ocrmypdf will add a OCRed text layer to PDF files, making the text selectable and indexable
- deleted 2y ago[deleted]