10 ms·

Quantitative AI progress needs accurate and transparent evaluation

209 points by bertman 1y ago

iloveoof 1y ago
Moore’s Law for AI Progress: AI metrics will double every two years whether the AI gets smarter or not.
ozgrakkurt 1y ago
Out of topic but just opening link and actually being able to read the posts and go to profile on a browser, without an account, feels really good. Opening a mastadon profile, fk twitter
- ipnon 1y ago
  Stallman was right all along.
NitpickLawyer 1y ago
The problem with benchmarks is that they are really useful for honest researchers, but extremely toxic if used for marketing, clout, etc. Something something, every measure that becomes a target sucks. It's really hard to trust anything public (for obvious reasons of dataset contamination), but also some private ones (for the obvious reasons that providers do get most/all of the questions over time, and they can do sneaky things with them). The only true tests are the ones you write yourself, never publish, and only work 100% on open models. If you want to test commercial SotA models from time to time you need to consider them "burned", and come up with more tests.
- antupis 1y ago
  Also, even if you want to be honest, at this point, probably every public or semipublic benchmark is part of CommonCrawl.
  - NitpickLawyer 1y ago
    True. And it's even worse than that, because each test probably gets "talked about" a lot in various places. And people come up with variants. And those variants get ingested. And then the whole thing becomes a mess. This was noticeable with the early Phi models. They were originally trained fully on synthetic data (cool experiment tbh) but the downside was that GPT3 / 4 was "distilling" benchmarks "hacks" into it. It became aparent when new benchmarks were released, after the published date, and there was one that measured "contamination" of about 20+%. Just from distillation.
- rachofsunshine 1y ago
  What makes Goodhart's Law so interesting is that you transition smoothly between two entirely-different problems the more strongly people want to optimize for your metric. One is a measurement problem, a statement about the world as it is: an engineer who can finish such-and-such many steps of this coding task in such-and-such time has such-and-such chance of getting hired. The thing you're measuring isn't running away from you or trying to hide itself, because facts aren't conscious agents with the goal of misleading you. Measurement problems are problems of statistics and optimization, and their goal is a function f: states -> predictions. Your problems are usually problems of inputs, not problems of mathematics. But the larger you get, and the more valuable gaming your test is, the more you leave that measurement problem and find an adversarial problem. Adversarial problems are at least as difficult as your adversary is intelligent, and they can sometimes be even worse by making your adversary the invisible hand of the market. You don't live in the world of gradient descent anymore, because the landscape is no longer fixed. You now live in the world of game theory, and your goal is a function f: (state) x (time) x (adversarial capability) x (history of your function f) -> predictions. It's that last, recursive bit that really makes adversarial problems brutal. Very simple functions can rapidly result in extremely deep chaotic dynamics once you allow even the slightest bit of recursion - even very nice functions like f(x) = 3.5x(1-x) become writhing ergodic masses of confusion.
  - visarga 1y ago
    Well said, the problem with recursion is that it constructs its own context as it goes, rewrites its rules, and you cannot predict it statically, without forward execution. It's why we have the halting problem. Recursion is irreducible. A benchmark is a static dataset, it does not capture the self constructive nature of recursion.
  - bwfan123 1y ago
    nice comment, a reason why ML approaches may struggle in trading markets where other agents are also competing with you possibly using similar algos. or self-driving which involves other agents who could be adversarial. just training on past data is not sufficient as existing edges are competed away and new edges keep arising out of nowhere.
  - 1y ago
- klingon-3 1y ago
  > It's really hard to trust anything public Just feed it into an LLM, unintentionally hint at your bias, and voila, it will use research and the latest or generated metrics to prove whatever you’d like. > The only true tests are the ones you write yourself, never publish, and only work 100% on open models. This may be good enough, and that’s fine if it is. But, if you do it in-house in a closet with open models, you will have your own biases. No tests are valid if all that ever mattered was the argument and perhaps curated evidence. All tests, private and public tests have proved flawed theories historically. Truth has always been elusive and under siege. People will always just believe things. Data is just foundation for pre-existing or fabricated beliefs. It’s the best rationale for faith, because in the end, faith is everything. Without it, there is nothing.
- mmcnl 1y ago
  Yes, I ignore every news article about LLM benchmarks. "GPT 7.3o first to reach >50% score in X2FGT AGI benchmark" - ok thanks for the info?
- crocowhile 1y ago
  There is also a social issue that has to do with accountability. If you claim your model is the best and then it turns out you overfitted the benchmarks and it's actually 68th, your reputation should suffer considerably for cheating. If it does not, we have a deeper problem than the benchmarks.
- ACCount36 1y ago
  Your options for evaluating AI performance are: benchmarks or vibes. Benchmarks are a really good option to have.
ipnon 1y ago
Tao’s commentary is more practical and insightful than all of the “rationalist” doomers put together.
- Quekid5 1y ago
  That seems like a low bar :)
  - ipnon 1y ago
    My priors do not allow the existence of bars. Your move.
    - tempodox 1y ago
      You would have felt right at home in the time of the Prohibition.
- jmmcd 1y ago
  (a) no it's not (b) your comment is miles off-topic, as he is not addressing doom in any sense
- ks2048 1y ago
  I agree about Tao in general, but here, > AI technology is now rapidly approaching the point of transition from qualitative to quantitative achievement. I don't get it. The whole history of deep learning was driven by quantitative achievement on benchmarks. I guess the rest of the post is about adding emphasis on costs in addition to overall performance. But, I don't see how that is a shift from qualitative to quantitative.
  - raincole 1y ago
    He means people in this AI hype trend mostly focused on "now AI can do a task that was impossible mere 5 years ago", but we will gradually change our perception of AI to "how much energy/hardware cost to complete this task and does it really benefit us." (My interpretation, obviously)
kingstnap 1y ago
My own thoughts on it are that it's entirely crazy that we focus so much on "real world" fixed benchmarks. I should write an article on it sometime, but I think the incessant focus on data someone collected from the mystical "real world" over well designed synthetic data from a properly understood algorithm is really damaging to proper understanding.
paradite 1y ago
I believe everyone should run their own evals on their own tasks or use cases. Shameless plug, but I made a simple app for anyone to create their own evals locally: https://eval.16x.engineer/ https://eval.16x.engineer/
pu_pe 1y ago
> For instance, if a cutting-edge AI tool can expend $1000 worth of compute resources to solve an Olympiad-level problem, but its success rate is only 20%, then the actual cost required to solve the problem (assuming for simplicity that success is independent across trials) becomes $5000 on the average (with significant variability). If only the 20% of trials that were successful were reported, this would give a highly misleading impression of the actual cost required (which could be even higher than this, if the expense of verifying task completion is also non-trivial, or if the failures to solve the goal were correlated across iterations). This is a very valid point. Google and ChatGPT announced they got the gold medal with specialized models, but what exactly does that entail? If one of them used a billion dollars in compute and the other a fraction of that, we should know about it. Error rates are equally important. Since there are conflicts of interest here, academia would be best suited for producing reliable benchmarks, but they would need access to closed models.
- JohnKemeny 1y ago
  Don't put Google and ChatGPT in the same category here. Google cooperated with the organizers, at least.
  - spuz 1y ago
    Could you clarify what you mean by this?
    - raincole 1y ago
      Google's answers were judged by IMO. OpenAI's were judged by themselves internally. Whether it matters is up to the reader.
    - EnnEmmEss 1y ago
      TheZvi had a summarization of this here: https://thezvi.substack.com/i/168895545/not-announcing-so-fast https://thezvi.substack.com/i/168895545/not-announcing-so-fa... In short (there is nuance), Google cooperated with the IMO team while OpenAI didn't which is why OpenAI announced before Google.
BrenBarn 1y ago
I like Tao, but it's always so sad to me to see people talk in this detached rational way about "how" to do AI without even mentioning the ethical and social issues involved. It's like pondering what's the best way to burn down the Louvre.
- spuz 1y ago
  Do you not think social and ethical issues can be approached rationally? To me it sounds like Tao is concerned about the cost of running AI powered solutions and I can quite easily see how the ethical and social costs fit under that umbrella along with monetary and environmental costs.
- bubblyworld 1y ago
  I don't think everybody has to pay lip service to this stuff every time they talk about AI. Many people (myself included) acknowledge these issues but have nothing to add to the conversation that hasn't been said a million times already. Tao is a mathematician - I think it's completely fine that he's focused on the quantitative aspects of this stuff, as that is where his expertise is most relevant.
- Karrot_Kream 1y ago
  I feel like your comment could be more clear and less hyperbolic or inflammatory by saying something like: “I like Tao but the ethical and social issues surrounding AI are much more important to me than discussing its specifics.”
  - rolandog 1y ago
    I don't agree; portraying it as an opinion has the risk of continuing to erode the world with moral relativism. The tech — despite being sometimes impresaive — is objectively inefficient, expensive, and harmful to the environment (excessive use if energy and water for cooling), to the people located near the data centers (by stochastic leeching of coolants to the waterbed IIRC), and the economic harm done to hundreds of millions of people whose data was involuntarily used for training.
fsh 1y ago
I believe that it may be misguided to focus on compute that much, and it would be more instructive to consider the effort that went into curating the training set. The easiest way of solving math problems with an LLM is to make sure that very similar problems are included in the training set. Many of the AI achievements would probably look a lot less miraculous if one could check the training data. The most crass example is OpenAI paying off the FrontierMath creators last year to get exclusive secret access to the problems before the evaluation [1]. Even without resorting to cheating, competition formats are vulnerable to this. It is extremely difficult to come up with truly original questions, so by spending significant resources on re-hashing all kinds of permutations of previous question, one will probably end up very close to the actual competition set. The first rule I learned about training neural networks is to make damn sure there is no overlap between the training and validation sets. It it interesting that this rule has gone completely out of the window in the age of LLMs. [1] https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso...
- OtherShrezzing 1y ago
  > The easiest way of solving math problems with an LLM is to make sure that very similar problems are included in the training set. Many of the AI achievements would probably look a lot less miraculous if one could check the training data I'm fairly certain this phenomenon is responsible for LLM capabilities on GeoGuesser type games. They have unreasonably good performance. For example, being able to identify obscure locations from featureless/foggy pictures of a bench. GeoGuesser's entire dataset, including GPS metadata, is definitely included in all of the frontier model training datasets - so it should be unsurprising that they have excellent performance in that domain.
  - YetAnotherNick 1y ago
    > GeoGuesser's entire dataset No, it is not included, however there must be quite a lot of pictures on internet for most cities.. Geoguesser data is same as Google's street view data and it probably contains billions of 360 degree photos.
mhl47 1y ago
Side note: What is going on with these comments on Mathstodon? From moon landing denials, to insults, allegations that he used AI to write this ... almost all of them are to some capacity insane.
- nurettin 1y ago
  That is how peak humanity looks like.
- Karrot_Kream 1y ago
  I find the same kind of behavior on bigger Bluesky AI threads. I don't use Mathstodon (or actively follow folks on it) but I certainly feel sad to see similar replies there too. I speculate that folks opposed to AI are angry and take it out by writing these sorts of comments, but this is just my hunch. That's as much as I feel I should write about this without feeling guilty for derailing the discussion.
  - ACCount36 1y ago
    No wonder. Bluesky is where insane Twitter people go when they get too insane for Twitter.
- dash2 1y ago
  Almost everywhere on the internet is like this. It's hn that is (mostly!) exceptional.
  - f1shy 1y ago
    The “mostly” there is so important! But also HN suffers from other problems (see in this thread the discussion about over policing comments, and calling fast hyperbolic and inflammatory). And don’t get me started in the decline on depth in technical topics and soaring in political discussions. I came to HN for the first, not the second. So we are humans, there will never be a perfect forum.
    - 1y ago
stared 1y ago
I agree that after a challenge is something can be done at all (heavier-than-air flight, Moon landing, Gold medal at the IMO) then next question is makes sense economically. I like ARC-AGI approach for the reason that it shows both axes - score and price, and place human benchmark on these. https://arcprize.org/leaderboard https://arcprize.org/leaderboard
pama 1y ago
This sounds very reasonable to me. When considering top tier labs that optimize inference and own the GPUs: the electricity cost of USD 5000 at a data center with 4 cents per kWh (which may be possible to arrange or beat in some counties in the US with special industrial contracts) can produce about 2 trillion tokens for the R1-0528 model using 120kW draw for the B200 NVL72 hardware and the (still to be fully optimized) sglang inference pipeline: https://lmsys.org/blog/2025-06-16-gb200-part-1/ https://lmsys.org/blog/2025-06-16-gb200-part-1/ Although 2T tokens is not unreasonable for being able to get high precision answers to challenging math questions, such a very high token number would strongly suggest there are lots of unknown techniques deployed at these labs. If one adds the cost of GPU ownership or rental, say 2 USD/h/GPU, then the number of tokens for 5k USD shrinks dramatically to only 66B tokens, which is still high for usual techniques that try to optimize for a best single answer in the end, but perhaps plausible if the vast majority of these are intermediate thinking tokens and a lot of the value comes from LLM-based verification.
akomtu 1y ago
The benchmarks should really add the test of data compression. Intelligence is mostly about discovering the underlying principles, the ability to see simple rules behind complex behaviors, and data compression captures this well. For example, if you can look at a dataset of planetary and stellar motions and compress it into a simple equation, you'd be considered wildly intelligent. If you can't remember and reproduce a simple checkerboard pattern, you'd be considered dumb. Another example is drawing a duck in SVG - another form of data compression. Data extrapolation, on the other hand, is the opposite problem, which can be solved by imitation or by understanding the rules producing the data. Only the latter deserves to be called intelligence. Note, though, that understanding the rules isn't always a superior method. When we are driving, we drive by imitation based on our extensive experience with similar situations, hardly understanding the physics of driving.
js8 1y ago
LLMs could be very useful in formalizing the problem and assumptions (conversion from natural language), but once problem is described in a formal way (it can be described in some fuzzy logic), then more reliable AI techniques should be applied. Interestingly, Tao mentions https://teorth.github.io/equational_theories/ https://teorth.github.io/equational_theories/, and I believe this is better progress than LLMs doing math. I believe enhancing Lean with more tactics and formalizing those in Lean itself is a more fruitful avenue for AI in math.
- agentcoops 1y ago
  I used to work quite extensively with Isabelle and as a developer on Sledgehammer [1]. There are well-known results, most obviously the halting problem, that mean fully-automated logical methods applied to a formalism with any expressive capability, i.e. that can be used to formalize non-trivial problems, simply can never fulfill the role you seem to be suggesting. The proofs that are actually generated in that way are, anyway, horrendous -- in fact, the problem I used to work on was using graph algorithms to try and simplify computer-generated proofs for human comprehension. That's the very reason that all the serious work has previously been on proof /assistants/ and formal validation. LLMs, especially in /conjunction/ with Lean for formal validation, are really an exciting new frontier in mathematics and it's a mistake to see that as just "unreliable" versus "reliable" symbolic AI etc. The OP Terence Tao has been pushing the edge here since day one and providing, I think, the most unbiased perspective on where things stand today, strengths as much as limitations. [1] https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehammer.html https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehamm...
  - js8 1y ago
    LLMs (as well as humans) are algorithms like anything else and so they are also subject to halting problem. I don't see what LLMs do that couldn't be in principle formalized as a Lean tactic. (IMHO LLMs are just learning rules - theorems of some kind of fuzzy logic - and then try to apply them using heuristic search to satisfy the goal. Unfortunately the rules learned are likely not fully consistent and so you get reasoning errors.)
PontingClarke 1y ago
[dead]
data_maan 1y ago
The concept of pre-registered eval (an analogy to pre-registered study) will go a long way towards fixing this. More information https://mathstodon.xyz/@friederrr/114881863146859839 https://mathstodon.xyz/@friederrr/114881863146859839
kristianp 1y ago
It's going to take a large step up in transparency for AI companies to do this. It was back in gpt 4 days that openai stopped reporting model size for example and the others followed suit.
deleted 1y ago
[deleted]

209 points by bertman 1y ago

Moore’s Law for AI Progress: AI metrics will double every two years whether the AI gets smarter or not.

Out of topic but just opening link and actually being able to read the posts and go to profile on a browser, without an account, feels really good. Opening a mastadon profile, fk twitter

ipnon 1y ago
Stallman was right all along.

NitpickLawyer 1y ago

The problem with benchmarks is that they are really useful for honest researchers, but extremely toxic if used for marketing, clout, etc. Something something, every measure that becomes a target sucks. It's really hard to trust anything public (for obvious reasons of dataset contamination), but also some private ones (for the obvious reasons that providers do get most/all of the questions over time, and they can do sneaky things with them). The only true tests are the ones you write yourself, never publish, and only work 100% on open models. If you want to test commercial SotA models from time to time you need to consider them "burned", and come up with more tests.

antupis 1y ago
Also, even if you want to be honest, at this point, probably every public or semipublic benchmark is part of CommonCrawl.
- NitpickLawyer 1y ago
  True. And it's even worse than that, because each test probably gets "talked about" a lot in various places. And people come up with variants. And those variants get ingested. And then the whole thing becomes a mess. This was noticeable with the early Phi models. They were originally trained fully on synthetic data (cool experiment tbh) but the downside was that GPT3 / 4 was "distilling" benchmarks "hacks" into it. It became aparent when new benchmarks were released, after the published date, and there was one that measured "contamination" of about 20+%. Just from distillation.
rachofsunshine 1y ago
What makes Goodhart's Law so interesting is that you transition smoothly between two entirely-different problems the more strongly people want to optimize for your metric. One is a measurement problem, a statement about the world as it is: an engineer who can finish such-and-such many steps of this coding task in such-and-such time has such-and-such chance of getting hired. The thing you're measuring isn't running away from you or trying to hide itself, because facts aren't conscious agents with the goal of misleading you. Measurement problems are problems of statistics and optimization, and their goal is a function f: states -> predictions. Your problems are usually problems of inputs, not problems of mathematics. But the larger you get, and the more valuable gaming your test is, the more you leave that measurement problem and find an adversarial problem. Adversarial problems are at least as difficult as your adversary is intelligent, and they can sometimes be even worse by making your adversary the invisible hand of the market. You don't live in the world of gradient descent anymore, because the landscape is no longer fixed. You now live in the world of game theory, and your goal is a function f: (state) x (time) x (adversarial capability) x (history of your function f) -> predictions. It's that last, recursive bit that really makes adversarial problems brutal. Very simple functions can rapidly result in extremely deep chaotic dynamics once you allow even the slightest bit of recursion - even very nice functions like f(x) = 3.5x(1-x) become writhing ergodic masses of confusion.
- visarga 1y ago
  Well said, the problem with recursion is that it constructs its own context as it goes, rewrites its rules, and you cannot predict it statically, without forward execution. It's why we have the halting problem. Recursion is irreducible. A benchmark is a static dataset, it does not capture the self constructive nature of recursion.
- bwfan123 1y ago
  nice comment, a reason why ML approaches may struggle in trading markets where other agents are also competing with you possibly using similar algos. or self-driving which involves other agents who could be adversarial. just training on past data is not sufficient as existing edges are competed away and new edges keep arising out of nowhere.
- 1y ago
klingon-3 1y ago
> It's really hard to trust anything public Just feed it into an LLM, unintentionally hint at your bias, and voila, it will use research and the latest or generated metrics to prove whatever you’d like. > The only true tests are the ones you write yourself, never publish, and only work 100% on open models. This may be good enough, and that’s fine if it is. But, if you do it in-house in a closet with open models, you will have your own biases. No tests are valid if all that ever mattered was the argument and perhaps curated evidence. All tests, private and public tests have proved flawed theories historically. Truth has always been elusive and under siege. People will always just believe things. Data is just foundation for pre-existing or fabricated beliefs. It’s the best rationale for faith, because in the end, faith is everything. Without it, there is nothing.
mmcnl 1y ago
Yes, I ignore every news article about LLM benchmarks. "GPT 7.3o first to reach >50% score in X2FGT AGI benchmark" - ok thanks for the info?
crocowhile 1y ago
There is also a social issue that has to do with accountability. If you claim your model is the best and then it turns out you overfitted the benchmarks and it's actually 68th, your reputation should suffer considerably for cheating. If it does not, we have a deeper problem than the benchmarks.
ACCount36 1y ago
Your options for evaluating AI performance are: benchmarks or vibes. Benchmarks are a really good option to have.

ipnon 1y ago

Tao’s commentary is more practical and insightful than all of the “rationalist” doomers put together.

Quekid5 1y ago
That seems like a low bar :)
- ipnon 1y ago
  My priors do not allow the existence of bars. Your move.
  - tempodox 1y ago
    You would have felt right at home in the time of the Prohibition.
jmmcd 1y ago
(a) no it's not (b) your comment is miles off-topic, as he is not addressing doom in any sense
ks2048 1y ago
I agree about Tao in general, but here, > AI technology is now rapidly approaching the point of transition from qualitative to quantitative achievement. I don't get it. The whole history of deep learning was driven by quantitative achievement on benchmarks. I guess the rest of the post is about adding emphasis on costs in addition to overall performance. But, I don't see how that is a shift from qualitative to quantitative.
- raincole 1y ago
  He means people in this AI hype trend mostly focused on "now AI can do a task that was impossible mere 5 years ago", but we will gradually change our perception of AI to "how much energy/hardware cost to complete this task and does it really benefit us." (My interpretation, obviously)

kingstnap 1y ago

My own thoughts on it are that it's entirely crazy that we focus so much on "real world" fixed benchmarks. I should write an article on it sometime, but I think the incessant focus on data someone collected from the mystical "real world" over well designed synthetic data from a properly understood algorithm is really damaging to proper understanding.

paradite 1y ago

I believe everyone should run their own evals on their own tasks or use cases. Shameless plug, but I made a simple app for anyone to create their own evals locally: https://eval.16x.engineer/ https://eval.16x.engineer/

pu_pe 1y ago

> For instance, if a cutting-edge AI tool can expend $1000 worth of compute resources to solve an Olympiad-level problem, but its success rate is only 20%, then the actual cost required to solve the problem (assuming for simplicity that success is independent across trials) becomes $5000 on the average (with significant variability). If only the 20% of trials that were successful were reported, this would give a highly misleading impression of the actual cost required (which could be even higher than this, if the expense of verifying task completion is also non-trivial, or if the failures to solve the goal were correlated across iterations). This is a very valid point. Google and ChatGPT announced they got the gold medal with specialized models, but what exactly does that entail? If one of them used a billion dollars in compute and the other a fraction of that, we should know about it. Error rates are equally important. Since there are conflicts of interest here, academia would be best suited for producing reliable benchmarks, but they would need access to closed models.

JohnKemeny 1y ago
Don't put Google and ChatGPT in the same category here. Google cooperated with the organizers, at least.
- spuz 1y ago
  Could you clarify what you mean by this?
  - raincole 1y ago
    Google's answers were judged by IMO. OpenAI's were judged by themselves internally. Whether it matters is up to the reader.
  - EnnEmmEss 1y ago
    TheZvi had a summarization of this here: https://thezvi.substack.com/i/168895545/not-announcing-so-fast https://thezvi.substack.com/i/168895545/not-announcing-so-fa... In short (there is nuance), Google cooperated with the IMO team while OpenAI didn't which is why OpenAI announced before Google.

BrenBarn 1y ago

I like Tao, but it's always so sad to me to see people talk in this detached rational way about "how" to do AI without even mentioning the ethical and social issues involved. It's like pondering what's the best way to burn down the Louvre.

spuz 1y ago
Do you not think social and ethical issues can be approached rationally? To me it sounds like Tao is concerned about the cost of running AI powered solutions and I can quite easily see how the ethical and social costs fit under that umbrella along with monetary and environmental costs.
bubblyworld 1y ago
I don't think everybody has to pay lip service to this stuff every time they talk about AI. Many people (myself included) acknowledge these issues but have nothing to add to the conversation that hasn't been said a million times already. Tao is a mathematician - I think it's completely fine that he's focused on the quantitative aspects of this stuff, as that is where his expertise is most relevant.
Karrot_Kream 1y ago
I feel like your comment could be more clear and less hyperbolic or inflammatory by saying something like: “I like Tao but the ethical and social issues surrounding AI are much more important to me than discussing its specifics.”
- rolandog 1y ago
  I don't agree; portraying it as an opinion has the risk of continuing to erode the world with moral relativism. The tech — despite being sometimes impresaive — is objectively inefficient, expensive, and harmful to the environment (excessive use if energy and water for cooling), to the people located near the data centers (by stochastic leeching of coolants to the waterbed IIRC), and the economic harm done to hundreds of millions of people whose data was involuntarily used for training.

fsh 1y ago

I believe that it may be misguided to focus on compute that much, and it would be more instructive to consider the effort that went into curating the training set. The easiest way of solving math problems with an LLM is to make sure that very similar problems are included in the training set. Many of the AI achievements would probably look a lot less miraculous if one could check the training data. The most crass example is OpenAI paying off the FrontierMath creators last year to get exclusive secret access to the problems before the evaluation [1]. Even without resorting to cheating, competition formats are vulnerable to this. It is extremely difficult to come up with truly original questions, so by spending significant resources on re-hashing all kinds of permutations of previous question, one will probably end up very close to the actual competition set. The first rule I learned about training neural networks is to make damn sure there is no overlap between the training and validation sets. It it interesting that this rule has gone completely out of the window in the age of LLMs. [1] https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso...

OtherShrezzing 1y ago
> The easiest way of solving math problems with an LLM is to make sure that very similar problems are included in the training set. Many of the AI achievements would probably look a lot less miraculous if one could check the training data I'm fairly certain this phenomenon is responsible for LLM capabilities on GeoGuesser type games. They have unreasonably good performance. For example, being able to identify obscure locations from featureless/foggy pictures of a bench. GeoGuesser's entire dataset, including GPS metadata, is definitely included in all of the frontier model training datasets - so it should be unsurprising that they have excellent performance in that domain.
- YetAnotherNick 1y ago
  > GeoGuesser's entire dataset No, it is not included, however there must be quite a lot of pictures on internet for most cities.. Geoguesser data is same as Google's street view data and it probably contains billions of 360 degree photos.

mhl47 1y ago

Side note: What is going on with these comments on Mathstodon? From moon landing denials, to insults, allegations that he used AI to write this ... almost all of them are to some capacity insane.

nurettin 1y ago
That is how peak humanity looks like.
Karrot_Kream 1y ago
I find the same kind of behavior on bigger Bluesky AI threads. I don't use Mathstodon (or actively follow folks on it) but I certainly feel sad to see similar replies there too. I speculate that folks opposed to AI are angry and take it out by writing these sorts of comments, but this is just my hunch. That's as much as I feel I should write about this without feeling guilty for derailing the discussion.
- ACCount36 1y ago
  No wonder. Bluesky is where insane Twitter people go when they get too insane for Twitter.
dash2 1y ago
Almost everywhere on the internet is like this. It's hn that is (mostly!) exceptional.
- f1shy 1y ago
  The “mostly” there is so important! But also HN suffers from other problems (see in this thread the discussion about over policing comments, and calling fast hyperbolic and inflammatory). And don’t get me started in the decline on depth in technical topics and soaring in political discussions. I came to HN for the first, not the second. So we are humans, there will never be a perfect forum.
  - 1y ago

stared 1y ago

I agree that after a challenge is something can be done at all (heavier-than-air flight, Moon landing, Gold medal at the IMO) then next question is makes sense economically. I like ARC-AGI approach for the reason that it shows both axes - score and price, and place human benchmark on these. https://arcprize.org/leaderboard https://arcprize.org/leaderboard

pama 1y ago

This sounds very reasonable to me. When considering top tier labs that optimize inference and own the GPUs: the electricity cost of USD 5000 at a data center with 4 cents per kWh (which may be possible to arrange or beat in some counties in the US with special industrial contracts) can produce about 2 trillion tokens for the R1-0528 model using 120kW draw for the B200 NVL72 hardware and the (still to be fully optimized) sglang inference pipeline: https://lmsys.org/blog/2025-06-16-gb200-part-1/ https://lmsys.org/blog/2025-06-16-gb200-part-1/ Although 2T tokens is not unreasonable for being able to get high precision answers to challenging math questions, such a very high token number would strongly suggest there are lots of unknown techniques deployed at these labs. If one adds the cost of GPU ownership or rental, say 2 USD/h/GPU, then the number of tokens for 5k USD shrinks dramatically to only 66B tokens, which is still high for usual techniques that try to optimize for a best single answer in the end, but perhaps plausible if the vast majority of these are intermediate thinking tokens and a lot of the value comes from LLM-based verification.

akomtu 1y ago

The benchmarks should really add the test of data compression. Intelligence is mostly about discovering the underlying principles, the ability to see simple rules behind complex behaviors, and data compression captures this well. For example, if you can look at a dataset of planetary and stellar motions and compress it into a simple equation, you'd be considered wildly intelligent. If you can't remember and reproduce a simple checkerboard pattern, you'd be considered dumb. Another example is drawing a duck in SVG - another form of data compression. Data extrapolation, on the other hand, is the opposite problem, which can be solved by imitation or by understanding the rules producing the data. Only the latter deserves to be called intelligence. Note, though, that understanding the rules isn't always a superior method. When we are driving, we drive by imitation based on our extensive experience with similar situations, hardly understanding the physics of driving.

js8 1y ago

LLMs could be very useful in formalizing the problem and assumptions (conversion from natural language), but once problem is described in a formal way (it can be described in some fuzzy logic), then more reliable AI techniques should be applied. Interestingly, Tao mentions https://teorth.github.io/equational_theories/ https://teorth.github.io/equational_theories/, and I believe this is better progress than LLMs doing math. I believe enhancing Lean with more tactics and formalizing those in Lean itself is a more fruitful avenue for AI in math.

agentcoops 1y ago
I used to work quite extensively with Isabelle and as a developer on Sledgehammer [1]. There are well-known results, most obviously the halting problem, that mean fully-automated logical methods applied to a formalism with any expressive capability, i.e. that can be used to formalize non-trivial problems, simply can never fulfill the role you seem to be suggesting. The proofs that are actually generated in that way are, anyway, horrendous -- in fact, the problem I used to work on was using graph algorithms to try and simplify computer-generated proofs for human comprehension. That's the very reason that all the serious work has previously been on proof /assistants/ and formal validation. LLMs, especially in /conjunction/ with Lean for formal validation, are really an exciting new frontier in mathematics and it's a mistake to see that as just "unreliable" versus "reliable" symbolic AI etc. The OP Terence Tao has been pushing the edge here since day one and providing, I think, the most unbiased perspective on where things stand today, strengths as much as limitations. [1] https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehammer.html https://isabelle.in.tum.de/website-Isabelle2009-1/sledgehamm...
- js8 1y ago
  LLMs (as well as humans) are algorithms like anything else and so they are also subject to halting problem. I don't see what LLMs do that couldn't be in principle formalized as a Lean tactic. (IMHO LLMs are just learning rules - theorems of some kind of fuzzy logic - and then try to apply them using heuristic search to satisfy the goal. Unfortunately the rules learned are likely not fully consistent and so you get reasoning errors.)

PontingClarke 1y ago

[dead]

data_maan 1y ago

The concept of pre-registered eval (an analogy to pre-registered study) will go a long way towards fixing this. More information https://mathstodon.xyz/@friederrr/114881863146859839 https://mathstodon.xyz/@friederrr/114881863146859839

kristianp 1y ago

It's going to take a large step up in transparency for AI companies to do this. It was back in gpt 4 days that openai stopped reporting model size for example and the others followed suit.

deleted 1y ago

[deleted]