4 ms·

I worked on an in-house version of this feature for my employer (turning files into LLM-friendly text). After reading the source code, I can say this is a prett

by irskep 2y ago

I worked on an in-house version of this feature for my employer (turning files into LLM-friendly text). After reading the source code, I can say this is a pretty reasonable implementation of this type of thing. But I would avoid using it for images, since the LLM providers let you just pass images directly, and I would also avoid using it for spreadsheets, since LLMs are very bad at interpreting Markdown tables. There are a lot of random startups and open source projects who try to make this space sound fancy, but I really hope the end state is a simple project like this, easy to understand and easy to deploy. I do wish it had a knob to turn for "how much processing do you want me to do." For PDF specifically, you either have to get a crappy version of the plain text using heuristics in a way that is very sensitive to how the PDF is exported, or you have to go full OCR, and it's annoying when a project locks you into one or the other. I'm also not sure I'd want to use the speech-to-text features here since they might have very different performance characteristics than the text-to-text stuff.

cosmie 2y ago
From your experience, what would be the best way to handle spreadsheets?
- simonw 2y ago
  I don't think tabular data of any sort is a particularly good fit for LLMs at the moment. What are you trying to do with it? If you want to answer questions like "how many students does Everglade High School have?" and you have a spreadsheet of schools where one of the columns is "number of students" I guess you could feed that into an LLM, but it doesn't feel like a great tool for the job. I'd instead use systems like ChatGPT Code Interpreter where the LLM gets to load up that data programatically and answer questions by running code against it. Text-to-SQL systems could work well for that too.
  - btown 2y ago
    This is an active area of research: https://github.com/SpursGoZmy/Awesome-Tabular-LLMs https://github.com/SpursGoZmy/Awesome-Tabular-LLMs is a good starting point!
  - cosmie 2y ago
    For me personally, a lot of times it's for table augmentation purposes. Appending additional columns to a dataset, such as a cleaned/standardized version of another field, extracting a value from another field, or appending categorization attributes (sometimes pre-seeded and sometimes just giving it general direction). Or sometimes I'll manually curate a field like that, and then ask it to generate an Excel function that can be used to produce as similar a result as possible for automated categorization in the future. So in most cases I both want to provide it with tabular data, and also want tabular data back out. In general I've gotten decent results for these sorts of use cases, but when it falls down it's almost always addressable by tinkering with the formatting related instructions – sometimes by tweaking the input and sometimes by tweaking the instructions for the desired output.
- danielmarkbruce 2y ago
  Many LLMs are ok with json and html tables. Not perfect, but not terrible.
  - simonw 2y ago
    I've seen enough examples of an LLM misinterpreting a column or row - resulting in returning the incorrect answer to a question because it was off by one in one of the directions - that I'm nervous about trusting them for this. JSON objects are different - there the key/value relationship is closer in the set of tokens which usually makes it more reliable.
    - danielmarkbruce 2y ago
      yeah... so, you want to two step it. Parse the table into something structured, then answer the question. For a lot of LLM "problems", it's about the same as teaching a kid a multi-step problem in math - if you try to do it in one step, you are going to have a hard time .
- irskep 2y ago
  The only reason I'm not immediately answering is because I need to check whether it's a trade secret. We do our own thing that I haven't seen anywhere else and works super well. Sorry for being mysterious, I'll try to get an OK to share. Edit: yeah I can't talk about it, sorry
- nprateem 2y ago
  Give it the data as separate columns. For each cell give it the row index and the data. That way it's just working with lists but can easily key that eg all this data is in row 3, etc. Tell it to correlate data by the first value in the pair like that.
- layer8 2y ago
  Markdown isn’t suitable for most spreadsheets in the first place, IMO.
- __mharrison__ 2y ago
  LLMs are decent at Pandas. I say "decent" because most of the available training data for Pandas does things in a naive way. OTOH, they are horrible at Polars. (I figure this is mostly a lack of training data.)
  - disgruntledphd2 2y ago
    > I say "decent" because most of the available training data for Pandas does things in a naive way. They're around the level of the median user, which is pretty bad as pandas is a big and complicated API with many different approaches available (as is base R, in case people think I'm just hating on pandas).
themanmaran 2y ago
The reason there's a lot of startups in the OCR space (us being one of them) is the classic 80/20 rule. Any solution that's 80% accurate just doesn't work for most applications. Converting a clean .docx into markdown is 10 lines of python. But what about the same document with a screenshot of an excel file? Or complex table layouts? The .NORM files that people actually use. Definitely agree with having a toggle between rules-based/ocr. But if you're looking at company wide docs, you won't always know which to pick. Example with one of our test files: Input: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/OmniAI+demo+docs.pdf https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/Omni... MarkItDown: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/markitdown.md https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/mark... Ours: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/omnioutput.md https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/omni... The response from MarkItDown seems pretty barebones. I expected it to convert the clean pdf table element into a markdown table, but it just pulls the plaintext, which drops the header/column relationship.
- irskep 2y ago
  > Any solution that's 80% accurate just doesn't work for most applications. And yet people use LLMs, for which "80% accuracy" is still mostly an aspiration. :-) I think it's reasonably likely most people companies end up using open source libraries, at least partly because it lets them avoid adding another GDPR sub-processor. Unstructured.io, one of your competitors, goes as far as having an AWS Marketplace setup so customers can use their own infrastruture but still pay them. LLMs might get better at consuming badly-formatted data, so the data only needs to meet that minimum bar, vs the admittedly very nice output you showed.
  - themanmaran 2y ago
    > LLMs might get better at consuming badly-formatted data Oh agreed. There's definitely a meeting in the middle between better ingestion and smarter models. LLMs are already a great fuzzing layer for that type of interpretation. And even with a perfect WYSIWYG text extraction, you're still limited by how coherent the original document was in the first place.
dragonwriter 2y ago
LLM providers also let you send PDFs directly, too. OTOH, sometimes you are the LLM provider, and you may not be using a multimodal LLM. (Or, even though feeding an LLM is a common use. You may be using the markdown for another purpose.)
Ambix 2y ago
> LLMs are very bad at interpreting Markdown tables Which table format is better for LLMs? Do you have some insights there?

by irskep 2y ago

cosmie 2y ago

From your experience, what would be the best way to handle spreadsheets?

simonw 2y ago
I don't think tabular data of any sort is a particularly good fit for LLMs at the moment. What are you trying to do with it? If you want to answer questions like "how many students does Everglade High School have?" and you have a spreadsheet of schools where one of the columns is "number of students" I guess you could feed that into an LLM, but it doesn't feel like a great tool for the job. I'd instead use systems like ChatGPT Code Interpreter where the LLM gets to load up that data programatically and answer questions by running code against it. Text-to-SQL systems could work well for that too.
- btown 2y ago
  This is an active area of research: https://github.com/SpursGoZmy/Awesome-Tabular-LLMs https://github.com/SpursGoZmy/Awesome-Tabular-LLMs is a good starting point!
- cosmie 2y ago
  For me personally, a lot of times it's for table augmentation purposes. Appending additional columns to a dataset, such as a cleaned/standardized version of another field, extracting a value from another field, or appending categorization attributes (sometimes pre-seeded and sometimes just giving it general direction). Or sometimes I'll manually curate a field like that, and then ask it to generate an Excel function that can be used to produce as similar a result as possible for automated categorization in the future. So in most cases I both want to provide it with tabular data, and also want tabular data back out. In general I've gotten decent results for these sorts of use cases, but when it falls down it's almost always addressable by tinkering with the formatting related instructions – sometimes by tweaking the input and sometimes by tweaking the instructions for the desired output.
danielmarkbruce 2y ago
Many LLMs are ok with json and html tables. Not perfect, but not terrible.
- simonw 2y ago
  I've seen enough examples of an LLM misinterpreting a column or row - resulting in returning the incorrect answer to a question because it was off by one in one of the directions - that I'm nervous about trusting them for this. JSON objects are different - there the key/value relationship is closer in the set of tokens which usually makes it more reliable.
  - danielmarkbruce 2y ago
    yeah... so, you want to two step it. Parse the table into something structured, then answer the question. For a lot of LLM "problems", it's about the same as teaching a kid a multi-step problem in math - if you try to do it in one step, you are going to have a hard time .
irskep 2y ago
The only reason I'm not immediately answering is because I need to check whether it's a trade secret. We do our own thing that I haven't seen anywhere else and works super well. Sorry for being mysterious, I'll try to get an OK to share. Edit: yeah I can't talk about it, sorry
nprateem 2y ago
Give it the data as separate columns. For each cell give it the row index and the data. That way it's just working with lists but can easily key that eg all this data is in row 3, etc. Tell it to correlate data by the first value in the pair like that.
layer8 2y ago
Markdown isn’t suitable for most spreadsheets in the first place, IMO.
__mharrison__ 2y ago
LLMs are decent at Pandas. I say "decent" because most of the available training data for Pandas does things in a naive way. OTOH, they are horrible at Polars. (I figure this is mostly a lack of training data.)
- disgruntledphd2 2y ago
  > I say "decent" because most of the available training data for Pandas does things in a naive way. They're around the level of the median user, which is pretty bad as pandas is a big and complicated API with many different approaches available (as is base R, in case people think I'm just hating on pandas).

themanmaran 2y ago

The reason there's a lot of startups in the OCR space (us being one of them) is the classic 80/20 rule. Any solution that's 80% accurate just doesn't work for most applications. Converting a clean .docx into markdown is 10 lines of python. But what about the same document with a screenshot of an excel file? Or complex table layouts? The .NORM files that people actually use. Definitely agree with having a toggle between rules-based/ocr. But if you're looking at company wide docs, you won't always know which to pick. Example with one of our test files: Input: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/OmniAI+demo+docs.pdf https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/Omni... MarkItDown: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/markitdown.md https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/mark... Ours: https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/omnioutput.md https://omni-demo-data.s3.us-east-1.amazonaws.com/zerox/omni... The response from MarkItDown seems pretty barebones. I expected it to convert the clean pdf table element into a markdown table, but it just pulls the plaintext, which drops the header/column relationship.

irskep 2y ago
> Any solution that's 80% accurate just doesn't work for most applications. And yet people use LLMs, for which "80% accuracy" is still mostly an aspiration. :-) I think it's reasonably likely most people companies end up using open source libraries, at least partly because it lets them avoid adding another GDPR sub-processor. Unstructured.io, one of your competitors, goes as far as having an AWS Marketplace setup so customers can use their own infrastruture but still pay them. LLMs might get better at consuming badly-formatted data, so the data only needs to meet that minimum bar, vs the admittedly very nice output you showed.
- themanmaran 2y ago
  > LLMs might get better at consuming badly-formatted data Oh agreed. There's definitely a meeting in the middle between better ingestion and smarter models. LLMs are already a great fuzzing layer for that type of interpretation. And even with a perfect WYSIWYG text extraction, you're still limited by how coherent the original document was in the first place.

dragonwriter 2y ago

LLM providers also let you send PDFs directly, too. OTOH, sometimes you are the LLM provider, and you may not be using a multimodal LLM. (Or, even though feeding an LLM is a common use. You may be using the markdown for another purpose.)

Ambix 2y ago

> LLMs are very bad at interpreting Markdown tables Which table format is better for LLMs? Do you have some insights there?