4 ms·
The point isn't to have users chat with documents, it's to automatically parse unstructured data into structured data and store that somewhere for later use. As
by mjr00 3y ago
The point isn't to have users chat with documents, it's to automatically parse unstructured data into structured data and store that somewhere for later use. As a real example, I deal with piles of legal contracts from which I need to extract specific information so I can perform later analysis, to be able to answer questions like "what specific model of widget is under contract here" and "what's the average contract value" and "what's the average contract value of widget XYZ". All stuff that's very easy to answer in SQL[0], once you have the data -- but extracting that data from legal documents, many of which are not in English, previously required a small army of contractors. Now it's been replaced with a local LLM that parses those relevant contract details into JSON which gets stored into a database. The accuracy is suitable for my use case, though not 100%.
[0] and very hard to answer with an LLM, as they are notoriously awful at doing any sort of math.
- akudha 3y agoThank you, I understand your use case better. If possible, could you talk a bit about your process for training your model? It looks like it is specific to legal documents, how easy/hard would it be to do the same for other types of documents? Also, what level of accuracy is good enough, for your use case?
- mjr00 3y agoThe beauty of it is there's no model training involved: it's quite literally a prompt that reads something like "given the following document, output JSON that contains the following information: a field name `"widget"` that contains widget name, etc..." then including the doc right into the prompt. This was written a while ago so it just iterates in a python script dividing the source document up and aggregating to get around context length limits. Extremely simple, but works great. I don't have a really specific accuracy target but since my main interest is in the aggregated results it's not a huge deal if the aggregations are slightly off. (The manual human approach is not 100% accurate either, after all; it's extremely common for humans to make data entry mistakes.)