3 ms·

AGI should be able to do every job a human can do using a computer at least as well as the average human.

by UltraSane 4mo ago

LordDragonfang 4mo ago
That's already been true for a while, you're overestimating the average human. They just have different failure modes.
- UltraSane 4mo ago
  It isn't even close to true. The biggest problem is that humans performance improves over time. https://www.linkedin.com/pulse/announcing-aa-briefcase-bench https://www.linkedin.com/pulse/announcing-aa-briefcase-bench... AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work. Tasks with many messy input files, conflicting information, and complex deliverables remain difficult for all models. Under a strict all-or-nothing grading scheme per task, Claude Fable 5 leads overall, but achieves a perfect task score on only 3% of tasks. On 31 of 91 tasks, no model scores above 50%.
Davidzheng 4mo ago
And what is it worse at than an average human today that can be done on a computer?
- UltraSane 4mo ago
  almost everything? AGI has to be able to completely replace a human in any information worker role indefinitely.
  - virgildotcodes 4mo ago
    I think you're speeding past the word "average" in the sentence. I'd argue that current frontier models already exceed the abilities of average humans across the majority of tasks you can do on a computer, although you might be able to argue that they tend to be a bit slower? That latter part is debatable though - have you seen a non-technical person try to figure out something new on a computer?
    - UltraSane 4mo ago
      " I'd argue that current frontier models already exceed the abilities of average humans " for things that fit in their context window sure but LLMs can't learn over time the way humans can. One example is LLMs are very good at writing a few thousands line of code but they absolutely cannot write coherent million line codebases. By average human I meant the average skill level for the job. AGI would need to be able to pass a interview and get hired and the perform well enough to not get fired.
      - Davidzheng 4mo ago
        Yeah it's not true that for every job, it is better than median worker of that job. But it is conceivable that for almost all jobs it is already better than the median human (not just workers of that job).
        isomorphic_duck 4mo ago
        You have to understand that the median human is terrible at (almost) everything. Humans, the only examples of general intelligence we know, are economically valuable precisely because they can train themselves to specialise at a (relatively) narrow task over time. You don’t measure how good a coding model is by how well it programs relative to Doctors, or how well it can prove theorems relative to baristas, or how well it can write coherent novels relative to programmers. That would be a dumb metric.
        4mo ago
- UltraSane 4mo ago
  https://www.linkedin.com/pulse/announcing-aa-briefcase-bench https://www.linkedin.com/pulse/announcing-aa-briefcase-bench... AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work. Tasks with many messy input files, conflicting information, and complex deliverables remain difficult for all models. Under a strict all-or-nothing grading scheme per task, Claude Fable 5 leads overall, but achieves a perfect task score on only 3% of tasks. On 31 of 91 tasks, no model scores above 50%.

AGI should be able to do every job a human can do using a computer at least as well as the average human.

LordDragonfang 4mo ago

That's already been true for a while, you're overestimating the average human. They just have different failure modes.

UltraSane 4mo ago
It isn't even close to true. The biggest problem is that humans performance improves over time. https://www.linkedin.com/pulse/announcing-aa-briefcase-bench https://www.linkedin.com/pulse/announcing-aa-briefcase-bench... AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work. Tasks with many messy input files, conflicting information, and complex deliverables remain difficult for all models. Under a strict all-or-nothing grading scheme per task, Claude Fable 5 leads overall, but achieves a perfect task score on only 3% of tasks. On 31 of 91 tasks, no model scores above 50%.

Davidzheng 4mo ago

And what is it worse at than an average human today that can be done on a computer?

UltraSane 4mo ago
almost everything? AGI has to be able to completely replace a human in any information worker role indefinitely.
- virgildotcodes 4mo ago
  I think you're speeding past the word "average" in the sentence. I'd argue that current frontier models already exceed the abilities of average humans across the majority of tasks you can do on a computer, although you might be able to argue that they tend to be a bit slower? That latter part is debatable though - have you seen a non-technical person try to figure out something new on a computer?
  - UltraSane 4mo ago
    " I'd argue that current frontier models already exceed the abilities of average humans " for things that fit in their context window sure but LLMs can't learn over time the way humans can. One example is LLMs are very good at writing a few thousands line of code but they absolutely cannot write coherent million line codebases. By average human I meant the average skill level for the job. AGI would need to be able to pass a interview and get hired and the perform well enough to not get fired.
    - Davidzheng 4mo ago
      Yeah it's not true that for every job, it is better than median worker of that job. But it is conceivable that for almost all jobs it is already better than the median human (not just workers of that job).
      - isomorphic_duck 4mo ago
        You have to understand that the median human is terrible at (almost) everything. Humans, the only examples of general intelligence we know, are economically valuable precisely because they can train themselves to specialise at a (relatively) narrow task over time. You don’t measure how good a coding model is by how well it programs relative to Doctors, or how well it can prove theorems relative to baristas, or how well it can write coherent novels relative to programmers. That would be a dumb metric.
        4mo ago
UltraSane 4mo ago
https://www.linkedin.com/pulse/announcing-aa-briefcase-bench https://www.linkedin.com/pulse/announcing-aa-briefcase-bench... AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work. Tasks with many messy input files, conflicting information, and complex deliverables remain difficult for all models. Under a strict all-or-nothing grading scheme per task, Claude Fable 5 leads overall, but achieves a perfect task score on only 3% of tasks. On 31 of 91 tasks, no model scores above 50%.