3 ms·

> where accuracy is paramount > accuracy: 60% Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, bea

> where accuracy is paramount > accuracy: 60% Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, beating even XML. But I guess that’s the new normal because we’re trying to shoehorn conversational AI models to every use case rather than, say, training finetunes that are better at particular tasks. (Yes, of course you can’t train finetunes when the model is a proprietary black box on someone else’s computer.) Something about hammers and nails…

mritchie712 1y ago
they used GPT-4.1 nano, results would be quite different with sonnet or gpt5.
- lyu07282 1y ago
  Or just regular gpt-4.1, it's a quite capable model.
- fnordpiglet 1y ago
  I was looking for the frontier curve where they tested their benchmark across different models since this sort of behavior is highly parameter, architecture, training, and fine tuning sensitive. It’s a practically useful question so I was really disappointed when a) they didn’t publish their code so you could test yourself, b) they didn’t do even a cursory examination of other models and sizes.
- phyzome 1y ago
  trust me bro, the next model bro, it's just way better bro
  - typpilol 1y ago
    To be fair nano was an absolute crap model when it came out
gpt5 1y ago
Isn't the best performing (markdown tables) and the worst (pipe delimited tables) basically the same format?
- simonw 1y ago
  The best performing isn't markdown tables, it's markdown key/value pairs: ## Record 1 ``` id: 1 name: Charlie A0 age: 56 city: New York department: Operations salary: 67896 years_experience: 7 project_count: 1 ``` Which makes sense to me because the problem with formats like CSV and regular markdown tables is that it is too easy for the model to mistakenly associate a value in a row with the wrong header. Explicit key/value formats like this or YAML or JSON objects make that a lot less likely.
  - cwmoore 1y ago
    I was surprised that XML (56%), with closing tags, wasn’t as good as YAMl/KV(60%), though line breaks perform the same kind of grouping function. Then I realized from the table that XML used about 50% more tokens (~75K vs ~50K) for similar accuracy, and for the first time felt a kind of sympathy for the LLM…
  - svachalek 1y ago
    Yeah that was my intuition as well. I think the KV-Markdown format gains additional advantage over JSON and YAML in the special syntax for headers helping to break up records.
mattcollins 1y ago
I'm the person who ran the test. To explain the 60% a bit more... With small amounts of input data, the accuracy is near 100%. As you increase the size of the input data, the accuracy gradually decreases. For this test, I intentionally chose an input data set large enough that the LLM would score in the region of 50% accuracy (with variation between formats) in order to maximise the discriminative power of the test.
- ysleepy 1y ago
  Wouldn't it be more useful to measure the number of rows the model can process while still hitting 100% accuracy?
- rovr138 1y ago
  > As you increase the size of the input data, the accuracy gradually decreases. Interesting. On your section "Limitations and Areas for Further Study", What I'd be curious on future work would be, - changing the order of the data on each table type - changing the order of the questions I'm curious to know if what it fails is the same, if it changes depending on the location, if it's a bias. Is it always a specific question? Is it always a specific value? Is it always question #x (or around question #x?). Does it tend towards x or y on types of questions? Good idea
  - CuriouslyC 1y ago
    LLMs have documented position biases, with skew towards first and last. This is strongest in messages due to system prompt + current question training data, but it's present in list data in general.
    - rovr138 1y ago
      Exactly. But the papers I’ve seen, the tests are done based on answers being multiple choice usually. Where do you eat? A) floor B) table C) dirt In this case, the questions asked have an answer. The bias would then be on the order of the input data. It’s different enough that it triggered my curiosity.

mritchie712 1y ago

they used GPT-4.1 nano, results would be quite different with sonnet or gpt5.

lyu07282 1y ago
Or just regular gpt-4.1, it's a quite capable model.
fnordpiglet 1y ago
I was looking for the frontier curve where they tested their benchmark across different models since this sort of behavior is highly parameter, architecture, training, and fine tuning sensitive. It’s a practically useful question so I was really disappointed when a) they didn’t publish their code so you could test yourself, b) they didn’t do even a cursory examination of other models and sizes.
phyzome 1y ago
trust me bro, the next model bro, it's just way better bro
- typpilol 1y ago
  To be fair nano was an absolute crap model when it came out

gpt5 1y ago

Isn't the best performing (markdown tables) and the worst (pipe delimited tables) basically the same format?

simonw 1y ago
The best performing isn't markdown tables, it's markdown key/value pairs: ## Record 1 ``` id: 1 name: Charlie A0 age: 56 city: New York department: Operations salary: 67896 years_experience: 7 project_count: 1 ``` Which makes sense to me because the problem with formats like CSV and regular markdown tables is that it is too easy for the model to mistakenly associate a value in a row with the wrong header. Explicit key/value formats like this or YAML or JSON objects make that a lot less likely.
- cwmoore 1y ago
  I was surprised that XML (56%), with closing tags, wasn’t as good as YAMl/KV(60%), though line breaks perform the same kind of grouping function. Then I realized from the table that XML used about 50% more tokens (~75K vs ~50K) for similar accuracy, and for the first time felt a kind of sympathy for the LLM…
- svachalek 1y ago
  Yeah that was my intuition as well. I think the KV-Markdown format gains additional advantage over JSON and YAML in the special syntax for headers helping to break up records.

mattcollins 1y ago

I'm the person who ran the test. To explain the 60% a bit more... With small amounts of input data, the accuracy is near 100%. As you increase the size of the input data, the accuracy gradually decreases. For this test, I intentionally chose an input data set large enough that the LLM would score in the region of 50% accuracy (with variation between formats) in order to maximise the discriminative power of the test.

ysleepy 1y ago
Wouldn't it be more useful to measure the number of rows the model can process while still hitting 100% accuracy?
rovr138 1y ago
> As you increase the size of the input data, the accuracy gradually decreases. Interesting. On your section "Limitations and Areas for Further Study", What I'd be curious on future work would be, - changing the order of the data on each table type - changing the order of the questions I'm curious to know if what it fails is the same, if it changes depending on the location, if it's a bias. Is it always a specific question? Is it always a specific value? Is it always question #x (or around question #x?). Does it tend towards x or y on types of questions? Good idea
- CuriouslyC 1y ago
  LLMs have documented position biases, with skew towards first and last. This is strongest in messages due to system prompt + current question training data, but it's present in list data in general.
  - rovr138 1y ago
    Exactly. But the papers I’ve seen, the tests are done based on answers being multiple choice usually. Where do you eat? A) floor B) table C) dirt In this case, the questions asked have an answer. The bias would then be on the order of the input data. It’s different enough that it triggered my curiosity.