4 ms·

> Unstructured document parsing If I had to invest in any one area of LLM usage, it would be this. There is so much unstructured data in the world, and convert

by mjr00 3y ago

> Unstructured document parsing If I had to invest in any one area of LLM usage, it would be this. There is so much unstructured data in the world, and converting things like legal contracts or chatlogs into structured, queryable data is absurdly powerful. Nobody wants to talk about this usage for LLMs because they're too busy making TikToks about how GPT4 actually has a soul or whatever, but this will be the lasting legacy of LLMs after the hype around generative AI dies out. > A decent engineer will likely be able to write a slack-like application, definitely good enough to cancel the 500k/year contract, in a couple of months. And this is why generative AI is massively overhyped: the people hyping it don't understand the true value of the products they allegedly replace. Very similar to the crypto/blockchain hype where people who understood nothing about banking or logistics insisted that blockchain would solve all the problems there. If you think a corp is paying Slack $500k/year because it's hard to write a piece of software that can send messages between people in an organization, you're completely off base. (IRC exists, can do this and is free by the way.)

akudha 3y ago
There is so much unstructured data in the world Can you give a couple of specific examples? There are already sites like pdf.ai that let users chat with documents, are you thinking of something different?
- mjr00 3y ago
  The point isn't to have users chat with documents, it's to automatically parse unstructured data into structured data and store that somewhere for later use. As a real example, I deal with piles of legal contracts from which I need to extract specific information so I can perform later analysis, to be able to answer questions like "what specific model of widget is under contract here" and "what's the average contract value" and "what's the average contract value of widget XYZ". All stuff that's very easy to answer in SQL[0], once you have the data -- but extracting that data from legal documents, many of which are not in English, previously required a small army of contractors. Now it's been replaced with a local LLM that parses those relevant contract details into JSON which gets stored into a database. The accuracy is suitable for my use case, though not 100%. [0] and very hard to answer with an LLM, as they are notoriously awful at doing any sort of math.
  - akudha 3y ago
    Thank you, I understand your use case better. If possible, could you talk a bit about your process for training your model? It looks like it is specific to legal documents, how easy/hard would it be to do the same for other types of documents? Also, what level of accuracy is good enough, for your use case?
    - mjr00 3y ago
      The beauty of it is there's no model training involved: it's quite literally a prompt that reads something like "given the following document, output JSON that contains the following information: a field name `"widget"` that contains widget name, etc..." then including the doc right into the prompt. This was written a while ago so it just iterates in a python script dividing the source document up and aggregating to get around context length limits. Extremely simple, but works great. I don't have a really specific accuracy target but since my main interest is in the aggregated results it's not a huge deal if the aggregations are slightly off. (The manual human approach is not 100% accurate either, after all; it's extremely common for humans to make data entry mistakes.)
- bitshiftfaced 3y ago
  An example would be if you had a large article of text recording the history of a given subject, and you wanted a table of years and the number of times a given event happened in each year. It's now possible to do a task in less than a minute that which used to take hours.
- nunez 3y ago
  Real estate's a big one. If you wanted to find publicly-available historical information about who owned a house and how many times it changed hands, the format of the data will change completely from state to state and country to country. An LLM that can rip through millions of these documents (both digital and on paper with GPT-4V) and generate clean, normalized and structured JSON will be absolutely huge. Healthcare's another. Exchanging information between benefits brokers is madness because of inconsistent data formats. A more consumer-facing example: exporting/joining data from one or more apps to migrate into another app. This is a hellish process most of the time that an LLM could, theoretically, reduce to seconds (not counting execution time).
jakderrida 3y ago
> If I had to invest in any one area of LLM usage, it would be this. There is so much unstructured data in the world, and converting things like legal contracts or chatlogs into structured, queryable data is absurdly powerful. I tested poorly OCR'd text from a late 1800s magazine and was pretty impressed with the results from both GPT-4 and even Bard. In addition to making inferences based on the letters, it was able to infer from context the historically accurate terms. However, one prompt asking it to correct words didn't work as well as two prompts with the first one listing candidates for bad OCR'd words. While that might not seem like what you're talking about, it has the benefit of adding to the overall corpus that the future models are trained on. Also, with the NYT lawsuit, I'd presume fixing OCRs of old magazines and articles would be a pretty good way to fill the gap left behind.

by mjr00 3y ago

akudha 3y ago

There is so much unstructured data in the world Can you give a couple of specific examples? There are already sites like pdf.ai that let users chat with documents, are you thinking of something different?

mjr00 3y ago
The point isn't to have users chat with documents, it's to automatically parse unstructured data into structured data and store that somewhere for later use. As a real example, I deal with piles of legal contracts from which I need to extract specific information so I can perform later analysis, to be able to answer questions like "what specific model of widget is under contract here" and "what's the average contract value" and "what's the average contract value of widget XYZ". All stuff that's very easy to answer in SQL[0], once you have the data -- but extracting that data from legal documents, many of which are not in English, previously required a small army of contractors. Now it's been replaced with a local LLM that parses those relevant contract details into JSON which gets stored into a database. The accuracy is suitable for my use case, though not 100%. [0] and very hard to answer with an LLM, as they are notoriously awful at doing any sort of math.
- akudha 3y ago
  Thank you, I understand your use case better. If possible, could you talk a bit about your process for training your model? It looks like it is specific to legal documents, how easy/hard would it be to do the same for other types of documents? Also, what level of accuracy is good enough, for your use case?
  - mjr00 3y ago
    The beauty of it is there's no model training involved: it's quite literally a prompt that reads something like "given the following document, output JSON that contains the following information: a field name `"widget"` that contains widget name, etc..." then including the doc right into the prompt. This was written a while ago so it just iterates in a python script dividing the source document up and aggregating to get around context length limits. Extremely simple, but works great. I don't have a really specific accuracy target but since my main interest is in the aggregated results it's not a huge deal if the aggregations are slightly off. (The manual human approach is not 100% accurate either, after all; it's extremely common for humans to make data entry mistakes.)
bitshiftfaced 3y ago
An example would be if you had a large article of text recording the history of a given subject, and you wanted a table of years and the number of times a given event happened in each year. It's now possible to do a task in less than a minute that which used to take hours.
nunez 3y ago
Real estate's a big one. If you wanted to find publicly-available historical information about who owned a house and how many times it changed hands, the format of the data will change completely from state to state and country to country. An LLM that can rip through millions of these documents (both digital and on paper with GPT-4V) and generate clean, normalized and structured JSON will be absolutely huge. Healthcare's another. Exchanging information between benefits brokers is madness because of inconsistent data formats. A more consumer-facing example: exporting/joining data from one or more apps to migrate into another app. This is a hellish process most of the time that an LLM could, theoretically, reduce to seconds (not counting execution time).

jakderrida 3y ago

> If I had to invest in any one area of LLM usage, it would be this. There is so much unstructured data in the world, and converting things like legal contracts or chatlogs into structured, queryable data is absurdly powerful. I tested poorly OCR'd text from a late 1800s magazine and was pretty impressed with the results from both GPT-4 and even Bard. In addition to making inferences based on the letters, it was able to infer from context the historically accurate terms. However, one prompt asking it to correct words didn't work as well as two prompts with the first one listing candidates for bad OCR'd words. While that might not seem like what you're talking about, it has the benefit of adding to the overall corpus that the future models are trained on. Also, with the NYT lawsuit, I'd presume fixing OCRs of old magazines and articles would be a pretty good way to fill the gap left behind.