4 ms·

Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you a

Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you actually care about numbers...

zeitgeistcowboy 1y ago
My sentiments exactly. All the formats were so poorly read that they are all effectively useless.
deleted 1y ago
[deleted]
zaidf 1y ago
I've been stunned by how many smart people talk so casually about LLMs becoming better at math. Do they just forget that a calculator that is wrong 1% of the time is a de facto calculator that doesn't work and should not be used?
- xnx 1y ago
  > I've been stunned by how many smart people talk so casually about LLMs becoming better at math Could they be referring to this? "Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad" https://deepmind.google/discover/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/ https://deepmind.google/discover/blog/advanced-version-of-ge...
- westoncb 1y ago
  Doing math is not the same as calculating. LLMs can be very useful in doing math; for calculating they are the wrong tool (and even there they can be very useful, but you ask them to use calculating tools, not to do the calculations themselves—both Claude and ChatGPT are set up to do this). If you're curious, check out how mathematicians like Robert Ghrist or Terence Tao are using LLMs for math research, both have written about it online repeatedly (along with an increasing number of other researchers). Apart from assisting with research, their ability on e.g. math olympiad problems is periodically measured and objectively rapidly improving, so this isn't just a matter of opinion.
- magicalhippo 1y ago
  The best math lecturers I had at university sucked at mental calculations. Some almost screwed up 2+2 on the blackboard. Yes LLMs suck at calculating stuff. However they can manipulate equations and such, and sometimes impressively so.
- crazygringo 1y ago
  You realize that when typing into a calculator, you probably hit a wrong key more than 1% of the time? Which is why you always type important calculations twice? I've been stunned by how many smart people talk so casually about how because LLMs aren't perfect, they therefore have no value. Do they just forget that nothing in the world is perfect, and the values of things are measured in degrees?
  - Forgeties79 1y ago
    There’s a big difference between mistyping 1% of the time yourself (human error) and a calculator failing 1% of the time (machine error) and I am willing to bet there isn’t a company out there (maybe a handful of less scrupulous ones) that has knowingly shipped a calculator that got it wrong 1% of the time. Especially in previous decades when countless people were using a dedicated calculator dozens of times a day. Hard to imagine a 1% margin of error was acceptable. Not to mention now you have the compounded problem of your mistakes plus the calculator’s mistakes.
    - crazygringo 1y ago
      There isn't a difference in the big picture. Error is error. Even when we have incredibly reliable things, there's error when they interface with humans. Humans have error interfacing with each other. But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;)
      - catlifeonmars 1y ago
        > But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;) Ah, but whose error? ;)
      - Forgeties79 1y ago
        > But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;) You really could’ve done without this bit.
    - 1y ago
mattcollins 1y ago
I'm the person who ran the test. To hopefully clarify a bit... I intentionally chose input data large enough that the LLM would be scoring in the region of 50% accuracy in order to maximise the discriminative power of the test.
- __mharrison__ 1y ago
  Can you expand on how you did this?
  - mattcollins 1y ago
    I did a small test with just a couple of formats and something like 100 records, saw that the accuracy was higher than I wanted, then increased the number of records until the accuracy was down to 50%-ish (e.g. 100 -> 200 -> 500 -> 1000, though I forget the precise numbers.)

Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you actually care about numbers...

zeitgeistcowboy 1y ago

My sentiments exactly. All the formats were so poorly read that they are all effectively useless.

deleted 1y ago

[deleted]

zaidf 1y ago

I've been stunned by how many smart people talk so casually about LLMs becoming better at math. Do they just forget that a calculator that is wrong 1% of the time is a de facto calculator that doesn't work and should not be used?

xnx 1y ago
> I've been stunned by how many smart people talk so casually about LLMs becoming better at math Could they be referring to this? "Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad" https://deepmind.google/discover/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/ https://deepmind.google/discover/blog/advanced-version-of-ge...
westoncb 1y ago
Doing math is not the same as calculating. LLMs can be very useful in doing math; for calculating they are the wrong tool (and even there they can be very useful, but you ask them to use calculating tools, not to do the calculations themselves—both Claude and ChatGPT are set up to do this). If you're curious, check out how mathematicians like Robert Ghrist or Terence Tao are using LLMs for math research, both have written about it online repeatedly (along with an increasing number of other researchers). Apart from assisting with research, their ability on e.g. math olympiad problems is periodically measured and objectively rapidly improving, so this isn't just a matter of opinion.
magicalhippo 1y ago
The best math lecturers I had at university sucked at mental calculations. Some almost screwed up 2+2 on the blackboard. Yes LLMs suck at calculating stuff. However they can manipulate equations and such, and sometimes impressively so.
crazygringo 1y ago
You realize that when typing into a calculator, you probably hit a wrong key more than 1% of the time? Which is why you always type important calculations twice? I've been stunned by how many smart people talk so casually about how because LLMs aren't perfect, they therefore have no value. Do they just forget that nothing in the world is perfect, and the values of things are measured in degrees?
- Forgeties79 1y ago
  There’s a big difference between mistyping 1% of the time yourself (human error) and a calculator failing 1% of the time (machine error) and I am willing to bet there isn’t a company out there (maybe a handful of less scrupulous ones) that has knowingly shipped a calculator that got it wrong 1% of the time. Especially in previous decades when countless people were using a dedicated calculator dozens of times a day. Hard to imagine a 1% margin of error was acceptable. Not to mention now you have the compounded problem of your mistakes plus the calculator’s mistakes.
  - crazygringo 1y ago
    There isn't a difference in the big picture. Error is error. Even when we have incredibly reliable things, there's error when they interface with humans. Humans have error interfacing with each other. But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;)
    - catlifeonmars 1y ago
      > But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;) Ah, but whose error? ;)
    - Forgeties79 1y ago
      > But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;) You really could’ve done without this bit.
  - 1y ago

mattcollins 1y ago

I'm the person who ran the test. To hopefully clarify a bit... I intentionally chose input data large enough that the LLM would be scoring in the region of 50% accuracy in order to maximise the discriminative power of the test.

__mharrison__ 1y ago
Can you expand on how you did this?
- mattcollins 1y ago
  I did a small test with just a couple of formats and something like 100 records, saw that the accuracy was higher than I wanted, then increased the number of records until the accuracy was down to 50%-ish (e.g. 100 -> 200 -> 500 -> 1000, though I forget the precise numbers.)