6 ms·
Show HN: Documind – Open-source AI tool to turn documents into structured data
Documind is an open-source tool that turns documents into structured data using AI.
What it does:
- Extracts specific data from PDFs based on your custom schema
- Returns clean, structured JSON that's ready to use
- Works with just a PDF link + your schema definition
Just run npm install documind to get started.
- bob778 2y agoFrom just reading the README, the example is not valid JSON. Is that intentional? Otherwise it seems like a prompt building tool, or am I missing something here?
- assanineass 2y agoOof you’re right LOL
- Tammilore 2y agoThanks for pointing this out. This was an error on my part. I see someone opened an issue for it so will fix now.
- rkuodys 2y agoJust this weekend was solving similar problem. What I've noticed, that on scanned documents, where stamp-text and handwriting is just as important as printed text, Gemini was way better compared to chat gpt. Of course, my prompts might have been an issue, but gemini with very brief and generic queries made significantly better results.
- inexcf 2y agoGot excited about an open-source tool doing this. Alas, i am let down. It is an open-source tool creating the prompt for the OpenAI API and i can't go and send customer data to them. I'm aware of https://github.com/clovaai/donut https://github.com/clovaai/donut so i hoped this would be more like that.
- _joel 2y agoYou can self host OpenAPI compatible models with lmstudio and the like. I've used it with https://anythingllm.com/ https://anythingllm.com/
- turblety 2y agoYou might be able to use Ollama, which has a OpenAI compatible API.
- Zambyte 2y agoNot without chaning the code (should be easy though) https://github.com/DocumindHQ/documind/blob/d91121739df03867ef6642bb3f2b17b51b99dc94/core/src/openAI.ts#L45 https://github.com/DocumindHQ/documind/blob/d91121739df03867...
- Tammilore 2y agoHi. I totally get the concern about sending data to OpenAI. Right now, Documind uses OpenAI's API just so people could quickly get started and see what it is like, but I’m open to adding options and contributions that would be better for privacy.
- inexcf 2y agoThat sounds great.
- deleted 2y ago[deleted]
- sidmo 2y agoI'd recommend checking out vision language models. They generate embeddings of the images themselves (as a collection of patches) and you can see query matching displayed as a heatmap over the document. Picks up text that OCR misses. I built a simple API over it if you want to try it out: https://github.com/DataFog/vlm-api https://github.com/DataFog/vlm-api
- danbruc 2y agoWith such a system, how do you ensure that the extracted data matches the data in the source document? Run the process several times and check that the results are identical? Can it reject inputs for manual processing? Or is it intended to be always checked manually? How good is it, how many errors does it make, say per million extracted values?
- glorpsicle 2y agoPerhaps there's still value in the documents being transformed by this tool and someone reviewing them manually, but obviously the real value would be in reducing manual review. I don't think there's a world–for now–in which this manual review can be completely eliminated. However, if you process, say, 1 million documents, you could sample and review a small percentage of them manually (a power calculation would help here). Assuming your random sample models the "distribution" (which may be tough to define/summarize) of the 1 million documents, you could then extrapolate your accuracy onto the larger set of documents without having to review each and every one.
- danbruc 2y agoYou can sample the result to determine the error rate, but if you find an unacceptable level of errors, then you still have to review everything manually. On the other hand, if you use traditional techniques, pattern matching with regular expressions and things like that, then you can probably get pretty close to perfection for those cases where your patterns match and you can just reject the rest for manual processing. Maybe you could ask a language model to compare the source document and the extracted data and to indicate whether there are errors, but I am not sure if that would help, maybe what tripped up the extraction would also trip up the result evaluation.
- khaki54 2y agoNot sure I would want something non-deterministic in my data pipeline. Maybe if it used GenAI to _develop a ruleset_ that could then be deployed, it would be more practical.
- fredtalty5 2y ago[dead]
- avereveard 2y ago> an interesting open source project enthusiastically setting up a lounge chair > OPENAI_API_KEY=your_openai_api_key carrying it back apathetically
- Tammilore 2y agoThanks for the laugh and your feedback! I know that depending on an OpenAI isn't ideal for everyone. I'm considering ways to make it more self-contained in the future, so it’s great to hear what users are looking for.
- avereveard 2y agolitellm would be a start, then you just pass in a model string that includes the provider, and can default on openai gpts, that removes most of the effort in adapting stuff both from you and other users.
- deleted 2y ago[deleted]
- gibsonf1 2y agoI'm not sure having statistics with fabrication try to extract text from PDF's would result in any mission-critical reliable data?
- eichi 2y agoconst systemPrompt = ` Convert the following PDF page to markdown. Return only the markdown with no explanation text. Do not include deliminators like '''markdown. You must include all information on the page. Do not exclude headers, footers, or subtext. `;
- deleted 2y ago[deleted]
- thor-rodrigues 2y agoVery nice tool! Just last week, I was working on extracting information from PDFs for an automation flow I’m building. I used Unstructured (https://unstructured.io/ https://unstructured.io/), which supports multiple file types, not just PDFs. However, my main issue is that I need to work with confidential client data that cannot be uploaded to a third party. Setting up the open-source, locally hosted version of Unstructured was quite cumbersome due to the numerous additional packages and installation steps required. While I’m open to the idea of parsing content with an LLM that has vision capabilities, data safety and confidentiality are critical for many applications. I think your project would go from good to great if it would be possible to connect to Ollama and run locally, That said, this is an excellent application! I can definitely see myself using it in other projects that don’t demand such stringent data confidentiality.”
- Tammilore 2y agoThank you, I appreciate the feedback! I understand people wanting data confidentiality and I'm considering connecting Ollama for future updates!
- ajith-joseph 2y ago[dead]
- asjfkdlf 2y agoI am looking for a similar service that turns any document (PNG, PDf, DocX) into JSON (preserving the field relationships). I tried with ChatGPT, but hallucinations are common. Does anything exist?
- omk 2y agoThis is also using OpenAI's GPT model. So the same hallucinations are probable here for PDFs.
- cccybernetic 2y agoI built a drag-and-drop document converter that extracts text into custom columns (for CSV) or keys (for JSON). You can schedule it to run at certain times and update a database as well. I haven't had issues with hallucinations. If you're interested, my email is in my bio.
- hirezeeshan 2y agoThat's a valid problem you are solving. I had similar usecase that I solved using PDF[dot]co
- azinman2 2y agoLooking at the source it seems this is just a thin wrapper over OpenAI. Am I missing something?
- emmanueloga_ 2y agoFrom the source, Documind appears to: 1) Install tools like Ghostscript, GraphicsMagick, and LibreOffice with a JS script. 2) Convert document pages to Base64 PNGs and send them to OpenAI for data extraction. 3) Use Supabase for unclear reasons. Some issues with this approach: * OpenAI may retain and use your data for training, raising privacy concerns [1]. * Dependencies should be managed with Docker or package managers like Nix or Pixi, which are more robust. Example: a tool like Parsr [2] provides a Dockerized pdf-to-json solution, complete with OCR support and an HTTP api. * GPT-4 vision seems like a costly, error-prone, and unreliable solution, not really suited for extracting data from sensitive docs like invoices, without review. * Traditional methods (PDF parsers with OCR support) are cheaper, more reliable, and avoid retention risks for this particular use case. Although these tools do require some plumbing... probably LLMs can really help with that! While there are plenty of tools for structured data extraction, I think there’s still room for a streamlined, all-in-one solution. This gap likely explains the abundance of closed-source commercial options tackling this very challenge. --- 1: https://platform.openai.com/docs/models#how-we-use-your-data https://platform.openai.com/docs/models#how-we-use-your-data 2: https://github.com/axa-group/Parsr https://github.com/axa-group/Parsr
- deleted 2y ago[deleted]
- groby_b 2y agoThat's not what [1] says, though? Quoth: "As of March 1, 2023, data sent to the OpenAI API will not be used to train or improve OpenAI models (unless you explicitly opt-in to share data with us, such as by providing feedback in the Playground). " "Traditional methods (PDF parsers with OCR support) are cheaper, more reliable" Not sure on the reliability - the ones I'm using all fail at structured data. You want a table extracted from a PDF, LLMs are your friend. (Recommendations welcome)
- niklasd 2y agoWe found that for extracting tables, OpenAIs LLMs aren't great. What is working well for us is Docling (https://github.com/DS4SD/docling/ https://github.com/DS4SD/docling/)
- infecto 2y agoMultimodal LLM are not the way to do this for a business workflow yet. In my experience your much better of starting with a Azure Doc Intelligence or AWS Textract to first get the structure of the document (PDF). These tools are incredibly robust and do a great job with most of the common cases you can throw at it. From there you can use an LLM to interrogate and structure the data to your hearts delight.
- IndieCoder 2y agoPlus one, using the exact setup to make it scale. If Azure Doc Intelligence gets too expensive, VLMs also work great
- vinothgopi 2y agoWhat is a VLM?
- saharhash 2y agoVision Language Model like Qwen VL https://github.com/QwenLM/Qwen2-VL https://github.com/QwenLM/Qwen2-VL or CoPali https://huggingface.co/blog/manu/colpali https://huggingface.co/blog/manu/colpali
- sidmo 2y agoVLMs are cool - they generate embeddings of the images themselves (as a collection of patches) and you can see query matching displayed as a heatmap over the document. Picks up text that OCR misses. Here's an open-source API demo I built if you want to try it out: https://github.com/DataFog/vlm-api https://github.com/DataFog/vlm-api
- disgruntledphd2 2y ago> AWS Textract to first get the structure of the document (PDF). These tools are incredibly robust and do a great job with most of the common cases you can throw at it. Do they work for Bills of Lading yet? When I tested a sample of these bills a few years back (2022 I think), the results were not good at all. But I honestly wouldn't be surprised if they'd massively improved lately.
- constantinum 2y agoReading from the comments, some of the common questions regarding document extraction are: * Run locally or on premise for security/privacy reasons * Support multiple LLMs and vector DBs - plug and play * Support customisable schemas * Method to check/confirm accuracy with source * Cron jobs for automation There is Unstract that solves the above requirements. https://github.com/Zipstack/unstract https://github.com/Zipstack/unstract
- vr46 2y agoI’ll have to test this against my local Python pipeline which does all this without an LLM in attendance. There are a ton of existing Python libraries which have been doing this for a long time, so let’s take a look..
- thegabriele 2y agoCare to share the best ones for some use cases? Thanks
- vr46 2y agoMinerU PDFQuery PyMuPDF (having more success with older versions, right now)
- IndieCoder 2y ago[dead]
- vunderba 2y agoOP, you've been accused of literally ripping off somebody's more popular repository and posing it as your own. https://news.ycombinator.com/item?id=42178413 https://news.ycombinator.com/item?id=42178413 You may wanna get ahead of this because the evidence is fairly damning. Failing to even give credit to the original project is a pretty gross move.
- Tammilore 2y agoHi. This was definitely not the intention. I made sure to copy and past the MIT license in Zerox exactly as it was into the folder of the code that uses it. I also included it in the main license file as well. If there's anything I could do to make corrections please let me know so I'd change that ASAP.
- ankenyr 2y agoYour initial commit makes it look like you wrote all the code. https://github.com/DocumindHQ/documind/commit/d91121739df03867ef6642bb3f2b17b51b99dc94 https://github.com/DocumindHQ/documind/commit/d91121739df038... This is because you copied and uploaded the code instead of forking. You could do a lot by restoring attribution. Your history would look the same as https://github.com/getomni-ai/zerox/commits/main/ https://github.com/getomni-ai/zerox/commits/main/ and diverge from where you forked. People are getting upset because this is not a nice thing to do. Attribution is significant. No one would care if you replaced all the names with the new ones in a fork because they would see commits that do that.
- Tammilore 2y agoHi. Thank you for pointing this out. I totally understand now that forking would have kept the commit history visible and made the attribution clearer. I have since added a direct note in the repo acknowledging that it is built on the original Zerox project and also linked back to it. If there’s anything else you’d suggest, happy to hear it. Thanks again.
- 2y ago
- deleted 2y ago[deleted]
- slippy 2y agoLegit question: By _removing the MIT license_ from the distribution and replacing it with the AGPL, how are you not violating the copyright and subject to a lawsuit? The MIT license has just 2 conditions. They are pretty easy to read, and the fist one is: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. By replacing the license, you violate this very simple agreement.
- Tammilore 2y agoHi. Thanks for the question. To clarify, the MIT license was never removed or swapped. The license was and still is included in the folder that contains the code from the original project. In the root of the repository, I added the AGPL license for the new code I developed and made sure to explicitly acknowledge that the code in the folder is still under the MIT license. I’ve also added a direct note acknowledging and linking back to the zerox project.