3 ms·
This is cool. When dealing with tables, you might want to explore departing from markdown. I’ve found that LLMs tend to struggle with tables that have large num
by mistercow 2y ago
This is cool. When dealing with tables, you might want to explore departing from markdown. I’ve found that LLMs tend to struggle with tables that have large numbers of columns containing similar data types. Correlating a row is easy enough, because the data is all together, but connecting a cell back to its column becomes a counting task, which appears to be pretty rough.
A trick I’ve found seems to work well is leaving some kind of id or coordinate marker on each column, and adding that to each cell. You could probably do that while still having valid markdown if you put the metadata in HTML comments, although it’s hard to say how an LLM will do at understanding that format.
- leroman 2y agoThanks for sharing, will look into adding this as a flag in the options!
- michaelmior 2y agoSpreadsheetLLM[0] might be worth looking into. It's designed for Excel (and similar) spreadsheets, so I'd imagine you could do something far simpler for the majority of HTML tables. [0] https://arxiv.org/abs/2407.09025v1 https://arxiv.org/abs/2407.09025v1
- msnkarthik 2y agoYou're spot on about the challenges LLMs face with complex markdown tables, especially when column counts rise and data types are similar. The "counting task" for column correlation is a real pain point – it's like the LLM loses track of where it is in the data grid. Your ID/coordinate marker idea is clever! It provides explicit context that LLMs seem to crave. Using HTML comments for this metadata is an interesting approach. It keeps the markdown valid for human readability, but I share your uncertainty about how consistently LLMs would parse and utilize it. Some other avenues worth exploring: Alternative Formats: Have you experimented with formats like CSV or JSON for feeding tabular data to LLMs? They might offer a more structured representation that's easier to parse. Pre-processing: Could we pre-process the table to create a more LLM-friendly representation? For example, converting it into a list of dictionaries, where each dictionary represents a row and keys represent column names. Prompt Engineering: Perhaps there are specific prompts or instructions that can guide LLMs to better handle large tables within markdown. It seems like there's room for innovation in how we bridge the gap between human-readable markdown tables and the structured data LLMs thrive on.
- breck 2y agoScrollSets work really well for using LLMs to generate tables: https://sets.scroll.pub/ https://sets.scroll.pub/ ScrollSets are basically "deconstructed CSVs".
- leroman 2y agoThis is now supported, see here- https://github.com/romansky/dom-to-semantic-markdown?tab=readme-ov-file#advanced-features https://github.com/romansky/dom-to-semantic-markdown?tab=rea...
- mattding 2y agoDo you have any numbers re-markdown performance, or is this anecdotal? I'm running a similar experiment right now and would love to hear anything else you've tried.