4 ms·

Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

117 points by matt_d 1mo ago

akshay_akula 1mo ago
Evals on actual research workflows is the right direction, most agent benches are toy tasks.
- deleted 1mo ago
  [deleted]
- quietbuilds_jp 1mo ago
  [dead]
rubslopes 1mo ago
I'm glad to see that GPT Sol beats Opus at least in Mathematical Sciences, because that's my need right now, and I much prefer GPT's prose style.
vatsachak 1mo ago
Damn. These things aren't AGI... but I don't care. Luna is good enough for me to give a parser spec and have it write one.
- daveguy 1mo ago
  How is an llm parser_spec to parser better than something like lex?
  - vatsachak 1mo ago
    Because sometimes parsers can have weird inputs, like structured Excel files
mlmonkey 1mo ago
Sad to see no mention of Gemini ...
- alansaber 1mo ago
  Do they especially train for scientific performance?
  - WithinReason 1mo ago
    They seem to, from my experience
jerpint 1mo ago
The fact that opus 5 is outperforming fable is odd to me From personal experience, opus 5 feels net inferior to fable on almost every aspect (for coding tasks)
- jhbadger 1mo ago
  This is measuring on scientific tasks though. I haven't used Fable lately but when I was playing with it when it was new its "safety" features made it practically impossible to use for biomedical science -- it once refused to work on a pipeline of mine that was analyzing pathogenicity islands in bacteria (presumably because it had guardrails to that effect to stop potential bioterrorists and the like)
- ianjbutler 1mo ago
  IMHO it's roughly task depth (fable) vs breadth (opus). Fable is great at tracing and debugging sometimes, but otherwise shorthand for confabulation. It's persistent but ungovernable, struggles to switch contexts, and goes insane with too much freedom to explore. Don't point it at anything that looks like a "system" for actual work (but mapping or planning might be ok). Opus is maybe not as creative, but it's more stable and more trustworthy. Opus driving Fable could be awesome, but Fable unleashed/unsupervised on longer horizon tasks or things that require more methodical changes on lots of components seems like a disaster every time I try it. Since this kind of thing is always down to harness, project-type, and other structural constraints, of course your mileage may vary. Fable is probably great for pen-testing, or as a decision-making kernel of other kinds of applications, and way better than Opus at those things. Probably fine for code-review or changing a codebase of a few thousand lines in any language. Actually building that codebase or changing an even bigger one? Woof. How this fits in with science? IDK but I bet other existing causal reasoning benchmarks might tell the whole story and this is back to stability again. Sometimes having a smart idea is really important! But more often it's important to just not forget what you were doing. What was I talking about? Oh look a squirrel
johnnyApplePRNG 1mo ago
Not surprised to see Claude significantly higher in scientific intelligence than Sol. You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. That's the feel I get from the both of them anyways and I've used both on the 20x plan for the past week at length. Don't get me wrong, codex is great at finding bugs and building games. It's great.
- saithound 1mo ago
  > You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. If you have time, can you elaborate or give some examples of mathematical nuances? I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantially incorrect proofs in a larger body of text), and Sol is saturating the benchmark, while Fable is below 50% even with the most generous grading. I don't work in the natural sciences, so I suspect you mean something different by "mathematical nuance".
  - overdrive110 1mo ago
    Is this benchmark public? Anecdotally I have had decent results asking Sol to nitpick my proofs (mostly probability theory but nothing super dense). I have never tried Claude seriously, so I am very curious about what the failures look like with Fable.
    - anmolkabra 1mo ago
      Yes! All tasks are on github: https://github.com/harbor-framework/terminal-bench-science https://github.com/harbor-framework/terminal-bench-science. They were contributed through PRs so the discussion and reviewing (before tasks were accepted) is also fully public.
- deleted 1mo ago
  [deleted]
a2ff6eeb0 1mo ago
I wonder how long it's going to be before self improvement encompasses hardware and materials science, not just code. It's exciting, soon we'll be able to fully hand off scientific, mathematical, and technical progress over to the machines, and then we can fully lay back.
- alansaber 1mo ago
  I love it when people handwave tech developments of a truly gargantuan scale
  - Xmd5a 1mo ago
    When ChatGPT4 was released I was like: "shit we're going to have programmable matter way earlier than I thought. Ferrari! Materialize! But in yellow this time!".
boorang 1mo ago
I got pretty good mileage out of context engineering, adding my personal coding heuristics to my AGENTS.md and referencing subdocuments on a "when doing X, consult Y" pattern. I assume others are doing similar things, but I was pretty surprised when I was able to get it to generate code that is pretty close to what I would do if I was doing it manually. I'm curious if scientists and mathematicians are doing things like that. "When I see X, I typically immediately check Y" or whatever their domain heuristics look like.
respectattentio 1mo ago
Like you were reading my mind. I was waiting for such benchmark to land. This will improve models for such scientific research workflows. AI should have started with science from the beginning, not after 4 years. I am building on top of it with agents to improve scientific workflows.
- latentsea 1mo ago
  >AI should have started with science from the beginning, not after 4 years. DeepMind has entered the chat.
CJefferson 1mo ago
I worry this doesn’t check correctness. I’ve been finding Claude is lately awful at folllowing instructions, I’ll ask it to implement the algorithm from a paper and it will do something simpler and slower and when challenged do it’s stupid apology thing. It can’t be trusted with anything I’d put in a paper, it lies too much. 4.6 couldn’t do as complex tasks, but it would do what was asked.
- j_maffe 1mo ago
  > I worry this doesn’t check correctness Then it's not a valid benchmark. I agree though they're not reliable enough to just put results in a paper.
- anmolkabra 1mo ago
  Benchmark does check correctness. Team is working to write a paper that will likely contain failure mode analysis; checking for instruction following could be a good idea. Agents are given instructions in markdown format, allowed to read data and libraries sandboxed in a Docker container, and evaluated on deterministic pytests on the outcome. Two things the team aimed to enforce to add tasks we could trust: - Scientific workflows are often simulations that are correct upto numerical tolerances (the scientist decides what's reasonable), so task verifiers' evaluate results of agent-written code within the tolerances. - There could be multiple solution codes to a scientific workflow, and the team tried to ensure the verifier tests accommodate those. Not overfit to the oracle reference code, written by the scientist. Instruction following is implicitly assumed, if the model gives up and doesn't complete the task it counts as a failure because the verifier tests fail.
  - mick01255 1mo ago
    [flagged]
PeterStuer 1mo ago
Strange to see Opus 5 outperform Fable 5 by that much. It can't be refusals as they share the same governor AFAIK.
jubilanti 1mo ago
No. Please no. I don't want science vibecoded. Software can rely on layers of testing and verification and most code is applying decades-old patterns to a customer's donain and gluing libraries together until they click. That simply don't work when you're on the frontier of something entirely new. There was already a glut of slop science before 2024. We don't need even more science-shaped slop clogging the peer review pipelines. Trust in science is at an all time low. This is a terrible thing to benchmark for and optimize.
- jampekka 1mo ago
  I generally trust vibe code a lot more than researcher code.
- frozenseven 1mo ago
  "Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.
  - mullenba 1mo ago
    I'm sure there will be some golden nuggets in the petabytes of slop that are about to be unleashed. The signal to noise ratio will be so bad that all of science will suffer.
    - frozenseven 1mo ago
      Every doomer claim of this sort has not only failed to come true, but the exact opposite has happened. For instance, see the tsunami of new incredible results in mathematics.
j_maffe 1mo ago
The tasks are the thing to really look at here: https://github.com/harbor-framework/terminal-bench-science/tree/main/tasks https://github.com/harbor-framework/terminal-bench-science/t...
- walrus01 1mo ago
  I am hoping someone with more free time than myself can contribute some things in the RF engineering domain in the 'engineering-sciences' section. There's some problems out there that will definitely stump even a smart LLM.
  - daveguy 1mo ago
    Looks like most things definitely stump even a "smart LLM"... Best score on this is 30%. Which is what you should assume for tasks you give an LLM if they aren't exactly the same as an existing benchmarked task. They're just not that good for the purposes people seem to think they are. Very limited application space.
seobotaicom 1mo ago
[dead]
cbg0 1mo ago
I wish these benchmarks were more closed off. The next model releases will have a huge jump in scores on this benchmark and it will be pretty much useless.
alikhater30000 1mo ago
[flagged]

117 points by matt_d 1mo ago

akshay_akula 1mo ago

Evals on actual research workflows is the right direction, most agent benches are toy tasks.

deleted 1mo ago
[deleted]
quietbuilds_jp 1mo ago
[dead]

rubslopes 1mo ago

I'm glad to see that GPT Sol beats Opus at least in Mathematical Sciences, because that's my need right now, and I much prefer GPT's prose style.

vatsachak 1mo ago

Damn. These things aren't AGI... but I don't care. Luna is good enough for me to give a parser spec and have it write one.

daveguy 1mo ago
How is an llm parser_spec to parser better than something like lex?
- vatsachak 1mo ago
  Because sometimes parsers can have weird inputs, like structured Excel files

mlmonkey 1mo ago

Sad to see no mention of Gemini ...

alansaber 1mo ago
Do they especially train for scientific performance?
- WithinReason 1mo ago
  They seem to, from my experience

jerpint 1mo ago

The fact that opus 5 is outperforming fable is odd to me From personal experience, opus 5 feels net inferior to fable on almost every aspect (for coding tasks)

jhbadger 1mo ago
This is measuring on scientific tasks though. I haven't used Fable lately but when I was playing with it when it was new its "safety" features made it practically impossible to use for biomedical science -- it once refused to work on a pipeline of mine that was analyzing pathogenicity islands in bacteria (presumably because it had guardrails to that effect to stop potential bioterrorists and the like)
ianjbutler 1mo ago
IMHO it's roughly task depth (fable) vs breadth (opus). Fable is great at tracing and debugging sometimes, but otherwise shorthand for confabulation. It's persistent but ungovernable, struggles to switch contexts, and goes insane with too much freedom to explore. Don't point it at anything that looks like a "system" for actual work (but mapping or planning might be ok). Opus is maybe not as creative, but it's more stable and more trustworthy. Opus driving Fable could be awesome, but Fable unleashed/unsupervised on longer horizon tasks or things that require more methodical changes on lots of components seems like a disaster every time I try it. Since this kind of thing is always down to harness, project-type, and other structural constraints, of course your mileage may vary. Fable is probably great for pen-testing, or as a decision-making kernel of other kinds of applications, and way better than Opus at those things. Probably fine for code-review or changing a codebase of a few thousand lines in any language. Actually building that codebase or changing an even bigger one? Woof. How this fits in with science? IDK but I bet other existing causal reasoning benchmarks might tell the whole story and this is back to stability again. Sometimes having a smart idea is really important! But more often it's important to just not forget what you were doing. What was I talking about? Oh look a squirrel

johnnyApplePRNG 1mo ago

Not surprised to see Claude significantly higher in scientific intelligence than Sol. You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. That's the feel I get from the both of them anyways and I've used both on the 20x plan for the past week at length. Don't get me wrong, codex is great at finding bugs and building games. It's great.

saithound 1mo ago
> You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. If you have time, can you elaborate or give some examples of mathematical nuances? I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantially incorrect proofs in a larger body of text), and Sol is saturating the benchmark, while Fable is below 50% even with the most generous grading. I don't work in the natural sciences, so I suspect you mean something different by "mathematical nuance".
- overdrive110 1mo ago
  Is this benchmark public? Anecdotally I have had decent results asking Sol to nitpick my proofs (mostly probability theory but nothing super dense). I have never tried Claude seriously, so I am very curious about what the failures look like with Fable.
  - anmolkabra 1mo ago
    Yes! All tasks are on github: https://github.com/harbor-framework/terminal-bench-science https://github.com/harbor-framework/terminal-bench-science. They were contributed through PRs so the discussion and reviewing (before tasks were accepted) is also fully public.
deleted 1mo ago
[deleted]

a2ff6eeb0 1mo ago

I wonder how long it's going to be before self improvement encompasses hardware and materials science, not just code. It's exciting, soon we'll be able to fully hand off scientific, mathematical, and technical progress over to the machines, and then we can fully lay back.

alansaber 1mo ago
I love it when people handwave tech developments of a truly gargantuan scale
- Xmd5a 1mo ago
  When ChatGPT4 was released I was like: "shit we're going to have programmable matter way earlier than I thought. Ferrari! Materialize! But in yellow this time!".

boorang 1mo ago

I got pretty good mileage out of context engineering, adding my personal coding heuristics to my AGENTS.md and referencing subdocuments on a "when doing X, consult Y" pattern. I assume others are doing similar things, but I was pretty surprised when I was able to get it to generate code that is pretty close to what I would do if I was doing it manually. I'm curious if scientists and mathematicians are doing things like that. "When I see X, I typically immediately check Y" or whatever their domain heuristics look like.

respectattentio 1mo ago

Like you were reading my mind. I was waiting for such benchmark to land. This will improve models for such scientific research workflows. AI should have started with science from the beginning, not after 4 years. I am building on top of it with agents to improve scientific workflows.

latentsea 1mo ago
>AI should have started with science from the beginning, not after 4 years. DeepMind has entered the chat.

CJefferson 1mo ago

I worry this doesn’t check correctness. I’ve been finding Claude is lately awful at folllowing instructions, I’ll ask it to implement the algorithm from a paper and it will do something simpler and slower and when challenged do it’s stupid apology thing. It can’t be trusted with anything I’d put in a paper, it lies too much. 4.6 couldn’t do as complex tasks, but it would do what was asked.

j_maffe 1mo ago
> I worry this doesn’t check correctness Then it's not a valid benchmark. I agree though they're not reliable enough to just put results in a paper.
anmolkabra 1mo ago
Benchmark does check correctness. Team is working to write a paper that will likely contain failure mode analysis; checking for instruction following could be a good idea. Agents are given instructions in markdown format, allowed to read data and libraries sandboxed in a Docker container, and evaluated on deterministic pytests on the outcome. Two things the team aimed to enforce to add tasks we could trust: - Scientific workflows are often simulations that are correct upto numerical tolerances (the scientist decides what's reasonable), so task verifiers' evaluate results of agent-written code within the tolerances. - There could be multiple solution codes to a scientific workflow, and the team tried to ensure the verifier tests accommodate those. Not overfit to the oracle reference code, written by the scientist. Instruction following is implicitly assumed, if the model gives up and doesn't complete the task it counts as a failure because the verifier tests fail.
- mick01255 1mo ago
  [flagged]

PeterStuer 1mo ago

Strange to see Opus 5 outperform Fable 5 by that much. It can't be refusals as they share the same governor AFAIK.

jubilanti 1mo ago

No. Please no. I don't want science vibecoded. Software can rely on layers of testing and verification and most code is applying decades-old patterns to a customer's donain and gluing libraries together until they click. That simply don't work when you're on the frontier of something entirely new. There was already a glut of slop science before 2024. We don't need even more science-shaped slop clogging the peer review pipelines. Trust in science is at an all time low. This is a terrible thing to benchmark for and optimize.

jampekka 1mo ago
I generally trust vibe code a lot more than researcher code.
frozenseven 1mo ago
"Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.
- mullenba 1mo ago
  I'm sure there will be some golden nuggets in the petabytes of slop that are about to be unleashed. The signal to noise ratio will be so bad that all of science will suffer.
  - frozenseven 1mo ago
    Every doomer claim of this sort has not only failed to come true, but the exact opposite has happened. For instance, see the tsunami of new incredible results in mathematics.

j_maffe 1mo ago

The tasks are the thing to really look at here: https://github.com/harbor-framework/terminal-bench-science/tree/main/tasks https://github.com/harbor-framework/terminal-bench-science/t...

walrus01 1mo ago
I am hoping someone with more free time than myself can contribute some things in the RF engineering domain in the 'engineering-sciences' section. There's some problems out there that will definitely stump even a smart LLM.
- daveguy 1mo ago
  Looks like most things definitely stump even a "smart LLM"... Best score on this is 30%. Which is what you should assume for tasks you give an LLM if they aren't exactly the same as an existing benchmarked task. They're just not that good for the purposes people seem to think they are. Very limited application space.

seobotaicom 1mo ago

[dead]

cbg0 1mo ago

I wish these benchmarks were more closed off. The next model releases will have a huge jump in scores on this benchmark and it will be pretty much useless.

alikhater30000 1mo ago

[flagged]