4 ms·

Can someone tell me where your average every day human that’s walking around and has a regular job and kids and a mortgage would land on this leaderboard? That’

Can someone tell me where your average every day human that’s walking around and has a regular job and kids and a mortgage would land on this leaderboard? That’s who we should be comparing against. The fact that the only formal comparisons for AI systems that are ever done are explicitly based on the highest performing narrowly focused humans, tells me how unprepared society is for what’s happening. Appreciate that: at the point in which there is unambiguous demonstration of superhuman level performance across all human tasks by a machine, (and make no mistake, that *is the bar that this blog post and every other post about AI sets*) it’s completely over for the human race; unless someone figures out an entirely new economic system.

raincole 1y ago
> average every day human Average math major can't get Brozne.
pphysch 1y ago
Machines have always had superhuman capabilities in narrow domains. The LLM domain is quite broad but it's still just a LLM, beholden to its training. The average everyday human does not have the time to read all available math texts. LLMs do, but they still can't get bronze. What does that say about them?
zdragnar 1y ago
The average person is bad at literally almost everything. If I want something done, I'll seek out someone with a skill set that matches the problem. I don't want AI to be as good as an average person. I want AI to be better than the person I would go to for help. A person can talk with me, understand where I've misunderstood my own problem, can point out faulty assumptions, and may even tell me that the problem isn't even a problem that needs solving. A person can suggest a variety of options and let me decide what trade-offs I want to make. If I don't trust the AI to do that, then I'm not sure why I'd use it for anything other than things that don't need to be done at all, unless I can justify the chance that maybe it'll be done right, and I can afford the time lost getting it done right without the AI afterwards.
- SirFatty 1y ago
  "The average person is bad at literally almost everything." Wow... that's quite a generalization. And not my experience at all.
  - Retric 1y ago
    The average person can’t play 99% of all musical instruments, speak 99% of all languages, do 99% of surgeries, recite 99% of all poems from memory etc. We don’t ask the average person to do most things, either finding a specialist or providing training beforehand.
    - krapp 1y ago
      One cannot be bad at the things one doesn't even do. None of this demonstrates that humans are bad at "literally almost everything."
      - Retric 1y ago
        > One cannot be bad at the things one doesn't even do. ??? If you don’t know how to do something you’re really bad at it. I’m not sure what that sentence is even trying to convey.
        krapp 1y ago
        > Obviously you could train someone to recite the The Raven from memory, but they can’t do it now. That doesn't make them bad at reciting The Raven from memory. Being trained to recite The Raven from memory and still being unable to do so would be a proper application of the term. There is an obvious difference between the two states of being and conflating them is specious. If you want to take seriously the premise that humans are bad at almost everything because most humans haven't been trained at doing almost everything humans can do, then you must apply the same rubric to LLMs, which are only capable of expressions within their specific dataset (and thus not the entire corpus of data on which they haven't been trained) and even then which tend to confabulate far more frequently than human beings at even simple tasks. edit: never mind, I guess you aren't willing to take this conversation on good faith.
baobabKoodaa 1y ago
Average human would score exactly 0 at IMO.
bgwalter 1y ago
Average humans, no. Mathematicians with enough time and a well indexed database of millions of similar problems, probably. We don't allow chess players to access a Syzygy tablebase in a tournament.
pragmatic 1y ago
That’s not how modern societies/economies work. We have specialists everywhere.
- AndrewKemendo 1y ago
  My literal last sentence addresses this

raincole 1y ago

> average every day human Average math major can't get Brozne.

pphysch 1y ago

Machines have always had superhuman capabilities in narrow domains. The LLM domain is quite broad but it's still just a LLM, beholden to its training. The average everyday human does not have the time to read all available math texts. LLMs do, but they still can't get bronze. What does that say about them?

zdragnar 1y ago

The average person is bad at literally almost everything. If I want something done, I'll seek out someone with a skill set that matches the problem. I don't want AI to be as good as an average person. I want AI to be better than the person I would go to for help. A person can talk with me, understand where I've misunderstood my own problem, can point out faulty assumptions, and may even tell me that the problem isn't even a problem that needs solving. A person can suggest a variety of options and let me decide what trade-offs I want to make. If I don't trust the AI to do that, then I'm not sure why I'd use it for anything other than things that don't need to be done at all, unless I can justify the chance that maybe it'll be done right, and I can afford the time lost getting it done right without the AI afterwards.

SirFatty 1y ago
"The average person is bad at literally almost everything." Wow... that's quite a generalization. And not my experience at all.
- Retric 1y ago
  The average person can’t play 99% of all musical instruments, speak 99% of all languages, do 99% of surgeries, recite 99% of all poems from memory etc. We don’t ask the average person to do most things, either finding a specialist or providing training beforehand.
  - krapp 1y ago
    One cannot be bad at the things one doesn't even do. None of this demonstrates that humans are bad at "literally almost everything."
    - Retric 1y ago
      > One cannot be bad at the things one doesn't even do. ??? If you don’t know how to do something you’re really bad at it. I’m not sure what that sentence is even trying to convey.
      - krapp 1y ago
        > Obviously you could train someone to recite the The Raven from memory, but they can’t do it now. That doesn't make them bad at reciting The Raven from memory. Being trained to recite The Raven from memory and still being unable to do so would be a proper application of the term. There is an obvious difference between the two states of being and conflating them is specious. If you want to take seriously the premise that humans are bad at almost everything because most humans haven't been trained at doing almost everything humans can do, then you must apply the same rubric to LLMs, which are only capable of expressions within their specific dataset (and thus not the entire corpus of data on which they haven't been trained) and even then which tend to confabulate far more frequently than human beings at even simple tasks. edit: never mind, I guess you aren't willing to take this conversation on good faith.

baobabKoodaa 1y ago

Average human would score exactly 0 at IMO.

bgwalter 1y ago

Average humans, no. Mathematicians with enough time and a well indexed database of millions of similar problems, probably. We don't allow chess players to access a Syzygy tablebase in a tournament.

pragmatic 1y ago

That’s not how modern societies/economies work. We have specialists everywhere.

AndrewKemendo 1y ago
My literal last sentence addresses this