6 ms·

Noam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient

Noam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient [than o1 or o3] with its thinking. And there’s a lot of room to push the test-time compute and efficiency further. > As fast as recent AI progress has been, I fully expect the trend to continue. Importantly, I think we’re close to AI substantially contributing to scientific discovery. I thought progress might be slowing down, but this is clear evidence to the contrary. Not the result itself, but the claims that it is a fully general model and has a clear path to improved efficiency. https://x.com/polynoamial/status/1946478249187377206 https://x.com/polynoamial/status/1946478249187377206

YeGoblynQueenne 1y ago
How is a claim, "clear evidence" to anything?
- kelipso 1y ago
  [flagged]
- modeless 1y ago
  Most evidence you have about the world is claims from other people, not direct experiment. There seems to be a thought-terminating cliche here on HN, dismissing any claim from employees of large tech companies. Unlike seemingly most here on HN, I judge people's trustworthiness individually and not solely by the organization they belong to. Noam Brown is a well known researcher in the field and I see no reason to doubt these claims other than a vague distrust of OpenAI or big tech employees generally which I reject.
  - emp17344 1y ago
    OpenAI have already shown us they aren’t trustworthy. Remember the FrontierMath debacle?
    - modeless 1y ago
      It's only a "debacle" if you already assume OpenAI isn't trustworthy, because they said they don't train on the test set. I hope you can see that presenting your belief that they lied about training on the test set as evidence of them being untrustworthy is a circular argument. You're assuming the thing you're trying to prove. The one OpenAI "scandal" that I did agree with was the thing where they threatened to cancel people's vested equity if they didn't sign a non-disparagement agreement. They did apologize for that one and make changes. But it doesn't have a lot to do with their research claims. I'm open to actual evidence that OpenAI's research claims are untrustworthy, but again, I also judge people individually, not just by the organization they belong to.
      - emp17344 1y ago
        They funded the entire benchmark and didn’t disclose their involvement. They then proceeded to make use of the benchmark while pretending like they weren’t affiliated with EpochAI. That’s a huge omission and more than enough reason to distrust their claims.
        modeless 1y ago
        IMO their involvement is only an issue if they gained an advantage on the benchmark by it. If they didn't train on the test set then their gained advantage is minimal and I don't see a big problem with it nor do I see an obligation to disclose. Especially since there is a hold-out set that OpenAI doesn't have access to, which can detect any malfeasance.
        emp17344 1y ago
        There’s no way to figure out whether they gained an advantage. We have to trust their claims, which again, is an issue for me after finding out they already lied.
        modeless 1y ago
  - sealeck 1y ago
    > I judge people's trustworthiness individually and not solely by the organization they belong to This is certainly a courageous viewpoint – I imagine this makes it very hard for you to engage in the modern world? Most of us are very bound by institutions we operate in!
    - modeless 1y ago
      Over the years my viewpoint has led to great success in predicting the direction and speed of development of many technologies, among other things. As a result, by objective metrics of professional and financial success, I have done very well. I think your imagination is misleading you.
  - zer00eyz 1y ago
    > dismissing any claim from employees of large tech companies Me: I have a way to turn lead into gold. You: Show me!!! Me: NO (and then spends the rest of my life in poverty). Cold Fusion (physics not the programing language) is the best example of why you "Show your work". This is the Valley we're talking about. It's the thudnderdome of technology and companies. If you have a meaningful breakthrough you don't talk about it you drop it on the public and flex.
    - mlyle 1y ago
      I don't think this is a reasonable take. Some people/organizations send signals about things that we're not ready to fully drop it on the world. Others consider those signals in context (reputation of sender, prior probability of being true, reasons for sender to be honest vs. deceptive, etc). When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence or disbelieve the existence of the pie. And I start to believe that it'll probably be a particularly good pie. This is from OpenAI. Here they've not been so great with public communications in the past, and they have a big incentive in a crowded marketplace to exaggerate claims. On the other hand, it seems like a dumb thing to say unless they're really going to deliver that soon.
      - zer00eyz 1y ago
        > Some people/organizations send signals about things that we're not ready to fully drop it on the world. This is called marketing. > When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence Because you have evidence, it smells. And if later your ask your wife "where is the pie" and she says "I sprayed pie scent in the air, I was just singling" how are you going to feel? Open AI spent its "fool us once" card already. Doing things this way does not earn back trust, failure to deliver (and they have done that more than once) ... See staff non disparagement, see the math fiasco, see open weights.
        mlyle 1y ago
        > This is called marketing. Many signals are marketing, but the purpose of signals is not purely to develop markets. We all have to determine what we think will happen next and how others will act. > Because you have evidence, it smells. I think you read that differently than what I intended to write -- she claims it smells good. > Open AI spent its "fool us once" card already. > > This is from OpenAI. Here they've not been so great with public communications in the past, and they have a big incentive in a crowded marketplace to exaggerate claims.
  - YeGoblynQueenne 1y ago
    A thought-terminating cliché? Not at all, certainly not when it comes to claims of technological or scientific breakthroughs. After all, that's partly why we have peer review and an emphasis on reproducibility. Until such a claim has been scrutinised by experts or reproduced by the community at large, it remains an unverified claim. >> Unlike seemingly most here on HN, I judge people's trustworthiness individually and not solely by the organization they belong to. That has nothing to do with anything I said. A claim can be false without it being fraudulent, in fact most false claims are probably not fraudulent; though, still, false. Claims are also very often contested. See e.g. the various claims of Quantum Superiority and the debate they have generated. Science is a debate. If we believe everything anyone says automatically, then there is no debate.
    - famouswaffles 1y ago
      They don't give a lot of details but they give enough for it to be pretty hard to say the claim is false but unfraudulent. Some researchers got a breakthrough and decided to share right then rather than the months later it would take for a viable product. It happens, researchers are humans after all and i'm generally glad to take a peek at the actual frontier rather than what's behind by many months. You can and it's fair to ignore such claims until that part but i think anything more than that is fairly uncharitable for the situation.
      - YeGoblynQueenne 1y ago
        I'm not ignoring it. I'm waiting to see evidence of it. Is that uncharitable?
- mitthrowaway2 1y ago
  I read the GP's comment as "but [assuming this claim is correct], this is clear evidence to the contrary."
strangeloops85 1y ago
I assume there was tool use in the fine tuning?
- nmca 1y ago
  There wasn’t in the CoT for these problems.
lossolo 1y ago
> it’s also more efficient [than o1 or o3] with its thinking. "So under his saturate response, he never loses. For her to win, must make him unable at some even -> would need Q_{even-1}>even, i.e. some a_j> sqrt2. but we just showed always a_j<=c< sqrt2. So she can never cause his loss. So against this fixed response of his, she never wins (outcomes: may be infinite or she may lose by sum if she picks badly; but no win). So she does NOT have winning strategy at λ=c. So at equality, neither player has winning strategy."[1] Why use lot word when few word do trick? 1. https://github.com/aw31/openai-imo-2025-proofs/blob/main/problem_5.txt https://github.com/aw31/openai-imo-2025-proofs/blob/main/pro...
aprilthird2021 1y ago
What's the clear path to improved efficiency now that we've reached peak data?
- NitpickLawyer 1y ago
  > now that we've reached peak data? A) that's not clear B) now we have "reasoning" models that can be used to analyse the data, create n rollouts for each data piece, and "argue" for / against / neutral on every piece of data going into the model. Imagine having every page of a "short story book" + 10 best "how to write" books, and do n x n on them. Huge compute, but basically infinite data as well. We went from "a bunch of data" to "even more data" to "basically everything we got" to "ok, maybe use a previous model to sort through everything we got and only keep quality data" to "ok, maybe we can augment some data with synthetic datasets from tools etc" to "RL goes brrr" to (point B from above) "let's mix the data with quality sources on best practices".
  - jononor 1y ago
    A) We are out of the Internet-scale-for-free data. Of course the companies deploying LLM based systems at massive scale are of course ingesting a lot of human data from their users, that they are seeking to use to further improve their models. B) Has learning though "self-play" (like with AlphaZero etc) been demonstrated working for improving LLMs? What is the latest key research on this?
    - CamperBob2 1y ago
      Certainly the models have orders of magnitude more data available to them than the smartest human being who ever lived does/did. So we can assume that if the goal is "merely" superhuman intelligence, data is not a problem. It might be a constraint on the evolution of godlike intelligence, or AGI. But at that point we're so far out in bong-hit territory that it will be impossible to say who's right or wrong about what's coming. Has learning though "self-play" (like with AlphaZero etc) been demonstrated working for improving LLMs? My understanding (which might be incorrect) is that this amounts to RLHF without the HF part, and is basically how DeepSeek-R1 was trained. I recall reading about OpenAI being butthurt^H^H^H^H^H^H^H^H concerned that their API might have been abused by the Chinese to train their own model.
wds 1y ago
> I think we’re close to AI substantially contributing to scientific discovery. The new "Full Self-Driving next year"?
- Velorivox 1y ago
  "AI" already contributes "substantially" to "scientific discovery". It's a very safe statement to make, whereas "full self-driving" has some concrete implications.
  - Aperocky 1y ago
    Well I also think full self-driving contribute substantially to navigating the car on the street..
  - mort96 1y ago
    "AI" here means language models. Machine learning has been contributing to scientific discovery for ages, but this new wave of hype that marketing departments are calling "AI" are language models.
- oceanplexian 1y ago
  I know it’s a meme but there actually are fully self driving cars, they make thousands of trips every day in a couple US cities.
  - bigyabai 1y ago
    > in a couple US cities FWIW, when you get this reductive with your criterion there were technically self-driving cars in 2008 too.
    - skybrian 1y ago
      To be a bit more specific: no, they were not routinely making thousands of taxi rides with paying customers every day in 2008.
    - 1y ago
nightski 1y ago
That's a big leap from "answering test questions" to "contributing to scientific discovery".
- transcriptase 1y ago
  Having spent tens of thousands of hours contributing to scientific discovery by reading dense papers for a single piece of information, reverse engineering code written by biologists, and tweaking graphics to meet journal requirements… I can say with certainty it’s already contributing by allowing scientists to spend time on science versus spending an afternoon figuring out which undocumented argument in a R package from 2008 changes chart labels.
  - highwaylights 1y ago
    This. Even if LLM’s ultimately hit some hard ceiling as substantially-better-Googling-automatons they would already accelerate all thought-based work across the board, and that’s the level they’re already at now (arguably they’re beyond that). We’re already at the point where these tools are removing repetitive/predictable tasks from researchers (and everyone else), so clearly they’re already accelerating research.
    - 13years 1y ago
      Not sure how you get around the contamination problems. I use these everyday and they are extremely problematic about making errors that are hard to perceive. They are not reliable tools for any tasks that require accurate data.
  - j2kun 1y ago
    That is not what they mean by contributing to scientific discovery.
    - transcriptase 1y ago
      Perhaps not, but my point stands from personal experience and knowing what’s going on in labs right now that AI is greatly contributing to research even if it’s not doing the parts that most people think of when they think science. A sufficiently advanced AI in the near term isn’t going to start churning out novel hypotheses and being able to collect non-online data without first being able to secure funding to hire grad students or whatever robots can replace those.
wouldbecouldbe 1y ago
Yeah that’s the dream, but same as with the bar exams, they are fine tuning the models for specific tests. Which probably the model even has been trained on previous version of those tests
torginus 1y ago
Thing is, for example, all of classical physics can be derived from Newton's laws, Maxwell's equations and the laws of Thermodynamics, all of which can be written on a slip of paper. A sufficiently brilliant and determined human can invent or explain everything armed only with this knowledge. There's no need to train him on a huge corpus of text, like they do with ChatGPT. Not sure what this model's like, but I'm quite certain it's not trained on terabytes of Internet and book dumps, but rather is trained for abstract problem solving in some way, and is likely much smaller than these trillion parameter SOTA transformers, hence is much faster.
- ctoth 1y ago
  And the billions of years of evolution and the language that you use to explain the task to him and and the schooling he needs to understand what you're saying it and... and and and?
- tim333 1y ago
  If you look at the history of physics I don't think it really worked like that. It took about three centuries from Newton to Maxwell because it's hard to just deduce everything from basic principles.
  - torginus 1y ago
    I think you misundertand me, I'm making some pie in the sky statement about AI being able to discover the laws of nature in an afternoon. I'm just making the observation that if you know the basic equiations, and enough math (which is about multivariate calc), you can derive every single formula in your Physics textbook (and most undergrads do as part of their education). Since smart people can derive a lot of knowledge from a tiny set of axioms, smart AIs should be able to as well, which means you don't need to rely on a huge volume of curated information. Which means that needing to invest the internet and training on a terabyte of text might not be how these newer models are trained, and since they don't need to learn that much raw information, they might be smaller and faster.
    - 1y ago

YeGoblynQueenne 1y ago

How is a claim, "clear evidence" to anything?

kelipso 1y ago
[flagged]
modeless 1y ago
Most evidence you have about the world is claims from other people, not direct experiment. There seems to be a thought-terminating cliche here on HN, dismissing any claim from employees of large tech companies. Unlike seemingly most here on HN, I judge people's trustworthiness individually and not solely by the organization they belong to. Noam Brown is a well known researcher in the field and I see no reason to doubt these claims other than a vague distrust of OpenAI or big tech employees generally which I reject.
- emp17344 1y ago
  OpenAI have already shown us they aren’t trustworthy. Remember the FrontierMath debacle?
  - modeless 1y ago
    It's only a "debacle" if you already assume OpenAI isn't trustworthy, because they said they don't train on the test set. I hope you can see that presenting your belief that they lied about training on the test set as evidence of them being untrustworthy is a circular argument. You're assuming the thing you're trying to prove. The one OpenAI "scandal" that I did agree with was the thing where they threatened to cancel people's vested equity if they didn't sign a non-disparagement agreement. They did apologize for that one and make changes. But it doesn't have a lot to do with their research claims. I'm open to actual evidence that OpenAI's research claims are untrustworthy, but again, I also judge people individually, not just by the organization they belong to.
    - emp17344 1y ago
      They funded the entire benchmark and didn’t disclose their involvement. They then proceeded to make use of the benchmark while pretending like they weren’t affiliated with EpochAI. That’s a huge omission and more than enough reason to distrust their claims.
      - modeless 1y ago
        IMO their involvement is only an issue if they gained an advantage on the benchmark by it. If they didn't train on the test set then their gained advantage is minimal and I don't see a big problem with it nor do I see an obligation to disclose. Especially since there is a hold-out set that OpenAI doesn't have access to, which can detect any malfeasance.
        emp17344 1y ago
        There’s no way to figure out whether they gained an advantage. We have to trust their claims, which again, is an issue for me after finding out they already lied.
        modeless 1y ago
- sealeck 1y ago
  > I judge people's trustworthiness individually and not solely by the organization they belong to This is certainly a courageous viewpoint – I imagine this makes it very hard for you to engage in the modern world? Most of us are very bound by institutions we operate in!
  - modeless 1y ago
    Over the years my viewpoint has led to great success in predicting the direction and speed of development of many technologies, among other things. As a result, by objective metrics of professional and financial success, I have done very well. I think your imagination is misleading you.
- zer00eyz 1y ago
  > dismissing any claim from employees of large tech companies Me: I have a way to turn lead into gold. You: Show me!!! Me: NO (and then spends the rest of my life in poverty). Cold Fusion (physics not the programing language) is the best example of why you "Show your work". This is the Valley we're talking about. It's the thudnderdome of technology and companies. If you have a meaningful breakthrough you don't talk about it you drop it on the public and flex.
  - mlyle 1y ago
    I don't think this is a reasonable take. Some people/organizations send signals about things that we're not ready to fully drop it on the world. Others consider those signals in context (reputation of sender, prior probability of being true, reasons for sender to be honest vs. deceptive, etc). When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence or disbelieve the existence of the pie. And I start to believe that it'll probably be a particularly good pie. This is from OpenAI. Here they've not been so great with public communications in the past, and they have a big incentive in a crowded marketplace to exaggerate claims. On the other hand, it seems like a dumb thing to say unless they're really going to deliver that soon.
    - zer00eyz 1y ago
      > Some people/organizations send signals about things that we're not ready to fully drop it on the world. This is called marketing. > When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence Because you have evidence, it smells. And if later your ask your wife "where is the pie" and she says "I sprayed pie scent in the air, I was just singling" how are you going to feel? Open AI spent its "fool us once" card already. Doing things this way does not earn back trust, failure to deliver (and they have done that more than once) ... See staff non disparagement, see the math fiasco, see open weights.
      - mlyle 1y ago
        > This is called marketing. Many signals are marketing, but the purpose of signals is not purely to develop markets. We all have to determine what we think will happen next and how others will act. > Because you have evidence, it smells. I think you read that differently than what I intended to write -- she claims it smells good. > Open AI spent its "fool us once" card already. > > This is from OpenAI. Here they've not been so great with public communications in the past, and they have a big incentive in a crowded marketplace to exaggerate claims.
- YeGoblynQueenne 1y ago
  A thought-terminating cliché? Not at all, certainly not when it comes to claims of technological or scientific breakthroughs. After all, that's partly why we have peer review and an emphasis on reproducibility. Until such a claim has been scrutinised by experts or reproduced by the community at large, it remains an unverified claim. >> Unlike seemingly most here on HN, I judge people's trustworthiness individually and not solely by the organization they belong to. That has nothing to do with anything I said. A claim can be false without it being fraudulent, in fact most false claims are probably not fraudulent; though, still, false. Claims are also very often contested. See e.g. the various claims of Quantum Superiority and the debate they have generated. Science is a debate. If we believe everything anyone says automatically, then there is no debate.
  - famouswaffles 1y ago
    They don't give a lot of details but they give enough for it to be pretty hard to say the claim is false but unfraudulent. Some researchers got a breakthrough and decided to share right then rather than the months later it would take for a viable product. It happens, researchers are humans after all and i'm generally glad to take a peek at the actual frontier rather than what's behind by many months. You can and it's fair to ignore such claims until that part but i think anything more than that is fairly uncharitable for the situation.
    - YeGoblynQueenne 1y ago
      I'm not ignoring it. I'm waiting to see evidence of it. Is that uncharitable?
mitthrowaway2 1y ago
I read the GP's comment as "but [assuming this claim is correct], this is clear evidence to the contrary."

strangeloops85 1y ago

I assume there was tool use in the fine tuning?

nmca 1y ago
There wasn’t in the CoT for these problems.

lossolo 1y ago

> it’s also more efficient [than o1 or o3] with its thinking. "So under his saturate response, he never loses. For her to win, must make him unable at some even -> would need Q_{even-1}>even, i.e. some a_j> sqrt2. but we just showed always a_j<=c< sqrt2. So she can never cause his loss. So against this fixed response of his, she never wins (outcomes: may be infinite or she may lose by sum if she picks badly; but no win). So she does NOT have winning strategy at λ=c. So at equality, neither player has winning strategy."[1] Why use lot word when few word do trick? 1. https://github.com/aw31/openai-imo-2025-proofs/blob/main/problem_5.txt https://github.com/aw31/openai-imo-2025-proofs/blob/main/pro...

aprilthird2021 1y ago

What's the clear path to improved efficiency now that we've reached peak data?

NitpickLawyer 1y ago
> now that we've reached peak data? A) that's not clear B) now we have "reasoning" models that can be used to analyse the data, create n rollouts for each data piece, and "argue" for / against / neutral on every piece of data going into the model. Imagine having every page of a "short story book" + 10 best "how to write" books, and do n x n on them. Huge compute, but basically infinite data as well. We went from "a bunch of data" to "even more data" to "basically everything we got" to "ok, maybe use a previous model to sort through everything we got and only keep quality data" to "ok, maybe we can augment some data with synthetic datasets from tools etc" to "RL goes brrr" to (point B from above) "let's mix the data with quality sources on best practices".
- jononor 1y ago
  A) We are out of the Internet-scale-for-free data. Of course the companies deploying LLM based systems at massive scale are of course ingesting a lot of human data from their users, that they are seeking to use to further improve their models. B) Has learning though "self-play" (like with AlphaZero etc) been demonstrated working for improving LLMs? What is the latest key research on this?
  - CamperBob2 1y ago
    Certainly the models have orders of magnitude more data available to them than the smartest human being who ever lived does/did. So we can assume that if the goal is "merely" superhuman intelligence, data is not a problem. It might be a constraint on the evolution of godlike intelligence, or AGI. But at that point we're so far out in bong-hit territory that it will be impossible to say who's right or wrong about what's coming. Has learning though "self-play" (like with AlphaZero etc) been demonstrated working for improving LLMs? My understanding (which might be incorrect) is that this amounts to RLHF without the HF part, and is basically how DeepSeek-R1 was trained. I recall reading about OpenAI being butthurt^H^H^H^H^H^H^H^H concerned that their API might have been abused by the Chinese to train their own model.

wds 1y ago

> I think we’re close to AI substantially contributing to scientific discovery. The new "Full Self-Driving next year"?

Velorivox 1y ago
"AI" already contributes "substantially" to "scientific discovery". It's a very safe statement to make, whereas "full self-driving" has some concrete implications.
- Aperocky 1y ago
  Well I also think full self-driving contribute substantially to navigating the car on the street..
- mort96 1y ago
  "AI" here means language models. Machine learning has been contributing to scientific discovery for ages, but this new wave of hype that marketing departments are calling "AI" are language models.
oceanplexian 1y ago
I know it’s a meme but there actually are fully self driving cars, they make thousands of trips every day in a couple US cities.
- bigyabai 1y ago
  > in a couple US cities FWIW, when you get this reductive with your criterion there were technically self-driving cars in 2008 too.
  - skybrian 1y ago
    To be a bit more specific: no, they were not routinely making thousands of taxi rides with paying customers every day in 2008.
  - 1y ago

nightski 1y ago

That's a big leap from "answering test questions" to "contributing to scientific discovery".

transcriptase 1y ago
Having spent tens of thousands of hours contributing to scientific discovery by reading dense papers for a single piece of information, reverse engineering code written by biologists, and tweaking graphics to meet journal requirements… I can say with certainty it’s already contributing by allowing scientists to spend time on science versus spending an afternoon figuring out which undocumented argument in a R package from 2008 changes chart labels.
- highwaylights 1y ago
  This. Even if LLM’s ultimately hit some hard ceiling as substantially-better-Googling-automatons they would already accelerate all thought-based work across the board, and that’s the level they’re already at now (arguably they’re beyond that). We’re already at the point where these tools are removing repetitive/predictable tasks from researchers (and everyone else), so clearly they’re already accelerating research.
  - 13years 1y ago
    Not sure how you get around the contamination problems. I use these everyday and they are extremely problematic about making errors that are hard to perceive. They are not reliable tools for any tasks that require accurate data.
- j2kun 1y ago
  That is not what they mean by contributing to scientific discovery.
  - transcriptase 1y ago
    Perhaps not, but my point stands from personal experience and knowing what’s going on in labs right now that AI is greatly contributing to research even if it’s not doing the parts that most people think of when they think science. A sufficiently advanced AI in the near term isn’t going to start churning out novel hypotheses and being able to collect non-online data without first being able to secure funding to hire grad students or whatever robots can replace those.

wouldbecouldbe 1y ago

Yeah that’s the dream, but same as with the bar exams, they are fine tuning the models for specific tests. Which probably the model even has been trained on previous version of those tests

torginus 1y ago

Thing is, for example, all of classical physics can be derived from Newton's laws, Maxwell's equations and the laws of Thermodynamics, all of which can be written on a slip of paper. A sufficiently brilliant and determined human can invent or explain everything armed only with this knowledge. There's no need to train him on a huge corpus of text, like they do with ChatGPT. Not sure what this model's like, but I'm quite certain it's not trained on terabytes of Internet and book dumps, but rather is trained for abstract problem solving in some way, and is likely much smaller than these trillion parameter SOTA transformers, hence is much faster.

ctoth 1y ago
And the billions of years of evolution and the language that you use to explain the task to him and and the schooling he needs to understand what you're saying it and... and and and?
tim333 1y ago
If you look at the history of physics I don't think it really worked like that. It took about three centuries from Newton to Maxwell because it's hard to just deduce everything from basic principles.
- torginus 1y ago
  I think you misundertand me, I'm making some pie in the sky statement about AI being able to discover the laws of nature in an afternoon. I'm just making the observation that if you know the basic equiations, and enough math (which is about multivariate calc), you can derive every single formula in your Physics textbook (and most undergrads do as part of their education). Since smart people can derive a lot of knowledge from a tiny set of axioms, smart AIs should be able to as well, which means you don't need to rely on a huge volume of curated information. Which means that needing to invest the internet and training on a terabyte of text might not be how these newer models are trained, and since they don't need to learn that much raw information, they might be smaller and faster.
  - 1y ago