4 ms·
Only testing GPT-4.1-nano makes this basically useless. Most people are almost certainly using GPT-5 mini or better. This very poor analysis is like an LLM lite
by dcre 1y ago
Only testing GPT-4.1-nano makes this basically useless. Most people are almost certainly using GPT-5 mini or better. This very poor analysis is like an LLM literacy test for readers.
- grey-area 1y agoPlease go away and do the work for us and let us know what anmazing accuracy you got with whatever version you think is better. Anything below 100% is actually pretty useless when it comes to stats.
- simonw 1y agoIf you want 100% accuracy from these kinds of tasks with LLMs you can get it today, but you need to provide the LLM with the ability to run Python code and tell it to use something like Pandas. You can confirm it's doing the right thing by reviewing the code it wrote.
- grey-area 1y agoOr you can just write the code to do it correctly. Which would be quicker. If you can review it properly you already understand how to do it.
- simonw 1y agoThat would require me to have memorized the pandas API. I've been using pandas on-and-off for over a decade and I still haven't come close to doing that.
- dcre 1y agoSimon is right about using code execution, but many tables one might look at outside of formal data work are small enough for LLMs to be very reliable at, so this format question is practically relevant. I wish they had tested better models.