7 ms·

I have proposed[1] a pragmatic rule of thumb for defining superintelligence: A system is superintelligent if in a remote hiring process it can get any non-phys

by Satam 3y ago

I have proposed[1] a pragmatic rule of thumb for defining superintelligence: A system is superintelligent if in a remote hiring process it can get any non-physical job, earn a salary and perform its duties to the satisfaction of the employer. This should be true for all positions and for all levels of seniority, from an intern to senior staff and PHDs. The employer must believe they are employing a human (it's ok if doubts arise due to system vastly outperforming its peers). [1] https://builtnotfound.proseful.com/pragmatic-superintelligence https://builtnotfound.proseful.com/pragmatic-superintelligen...

spywaregorilla 3y ago
why do you use the term superintelligence
- nothrowaways 3y ago
  Above basic Intelligence
  - pmontra 3y ago
    Are some of us on HN superintelligent according to your definition?
- Satam 3y ago
  This might go against the standard definition of AGI, but I think LLMs have already achieved general intelligence (but not superintelligence). It is not human level and it succeeds and fails in ways different from us. However, it certainly displays general intelligence the likes of which we have literally never seen in an artificial system.
  - poisonborz 3y ago
    Even the most advanced LLMs currently will tell you blatant falsehoods on a whim. More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights.
    - famouswaffles 3y ago
      >More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
  - deleted 3y ago
    [deleted]
  - enasterosophes 3y ago
    I felt like that too, at first. I was really excited by what LLMs seem capable of. But when I started implementing one from scratch I realized that it's just linear algebra picking out the most probable match to a prompt. I'm afraid I had to come to the conclusion that LLMs, although being fun to play around with sometimes and seeming surprising at first, it's a similar kind of fun and surprise to sleight-of-hand. Like sleight-of-hand, and unlike real general intelligences (i.e. other people), it seems less and less fun and surprising the more exposed to it I become. I guess this won't change your mind though. Maybe you've understood more about them than I have, and maybe I'm just grumpy and cynical.
    - famouswaffles 3y ago
      I used to think planes flew. I was really excited by what they seemed capable of when I heard about them. But when I saw one in person, I was disappointed. I saw no flapping. I saw no feathers. Sure I saw it soar through the clouds but that's just sleight of hand I'm sure. A trick. Planes surely do not fly.
      - enasterosophes 3y ago
        Is this meant to be a refutation? I didn't say that I had only "heard about" LLMs. I said I played around with them. If you were being faithful to what I wrote, you would take in the full thing, including the part about how LLMs become less fun and surprising the more you play with them. Planes do not fly less effectively the more experienced you become at piloting them. So I can only conclude that you did not come here to have a discussion in good faith. So what are you really after? Do you think you'll change my mind with your poetry?
        famouswaffles 3y ago
        It's an analogy for your flawed thinking. The situation is even worse than my story outlined because you understand neither the workings of the brain and how it is intelligent nor what the computation that preceeded a prediction of a language model represents. You don't know x and you don't y but you are convinced x=/y. >the part about how LLMs become less fun and surprising the more you play with them. Do they ? I don't think so. >Planes do not fly less effectively the more experienced you become at piloting them. How well a plane flies is an objective measure. How "fun" and "surprising" something is is largely subjective and can change on grounds that have nothing to do with what is being evaluated. Your refutation boils down to "It's just linear algebra!". Setting aside the fact that you can be reductionist about anything including biology, Okay? So what? Results matter not vague and meaningless philosophical ramblings.
  - ilaksh 3y ago
    Sure, we definitely are seeing some level/type of general intelligence, but we still want to distinguish between general intelligence and superintelligence. That's a big part of the problem, people thinking imprecisely and using "AGI" and "ASI" interchangeably.
    - RetroTechie 3y ago
      GI is a concept, SI is a matter of degree imho. What sets humans apart from AI right now, is specific types of out-of-the-box thinking. Let's say there's a bridge, an AI gets to control a robot, and is awarded points for crossing that bridge. I think you can be pretty sure that over time, the AI will figure out how to control the robot's limbs, move toward opposite side of the bridge, and find its way around obstacles. Meanwhile, humans may walk back & forth, because they enjoy crossing the bridge. Or stand in the middle for an hour, watching boats pass underneath. Some asshats may climb it. On a hot sunny day, others may go halfway & then jump off for a refreshing swim. Yet others may do some protest, and use the bridge as a scaffold to hang a banner from. Etc etc. Much such behavior defies logic. But yet, the bridge will primarily be used as intended, to cross from side to side. And humans go about in this fashion in many if not all their activities. When I see AIs that show similar creative / out-of-box behavior, while remaining mostly goal-oriented, and learn from its mistakes without humans nudging it every step of the way, then I think we'll have achieved AGI. Beyond that, SI is just a matter of scale & efficiency.
- bdb7 3y ago
  They are proposing whoever can get a job in corporate wonderland is super intelligent. When there is enough evidence corporate wonderland is overflowing with one dimensional intelligence cause their goals and what gets rewarded is extremely narrow. Einstein and Newton will fail the test 100%.
abetusk 3y ago
I think this is a pretty solid test, thanks. Maybe we can call it the Bartleby test?
- mensetmanusman 3y ago
  If it passes so many jobs go poof. Capital owners will then rejoice in their golden swimming pools.
thiht 3y ago
I think there's something to say about how we tend to define an intelligent system as... able to work and satisfy an employer. Not judging, this is actually my answer as well, but it's interesting. We could define AGI as "someone" (for lack of a better word) fun to talk to, or interesting on many subjects, or with a good personality, or we could use it to teach kids, or even all students, or as a shrink, or as a replacement for a friend. But we choose "able to please a boss" for this definition of "intelligent".

by Satam 3y ago

spywaregorilla 3y ago

why do you use the term superintelligence

nothrowaways 3y ago
Above basic Intelligence
- pmontra 3y ago
  Are some of us on HN superintelligent according to your definition?
Satam 3y ago
This might go against the standard definition of AGI, but I think LLMs have already achieved general intelligence (but not superintelligence). It is not human level and it succeeds and fails in ways different from us. However, it certainly displays general intelligence the likes of which we have literally never seen in an artificial system.
- poisonborz 3y ago
  Even the most advanced LLMs currently will tell you blatant falsehoods on a whim. More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights.
  - famouswaffles 3y ago
    >More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
- deleted 3y ago
  [deleted]
- enasterosophes 3y ago
  I felt like that too, at first. I was really excited by what LLMs seem capable of. But when I started implementing one from scratch I realized that it's just linear algebra picking out the most probable match to a prompt. I'm afraid I had to come to the conclusion that LLMs, although being fun to play around with sometimes and seeming surprising at first, it's a similar kind of fun and surprise to sleight-of-hand. Like sleight-of-hand, and unlike real general intelligences (i.e. other people), it seems less and less fun and surprising the more exposed to it I become. I guess this won't change your mind though. Maybe you've understood more about them than I have, and maybe I'm just grumpy and cynical.
  - famouswaffles 3y ago
    I used to think planes flew. I was really excited by what they seemed capable of when I heard about them. But when I saw one in person, I was disappointed. I saw no flapping. I saw no feathers. Sure I saw it soar through the clouds but that's just sleight of hand I'm sure. A trick. Planes surely do not fly.
    - enasterosophes 3y ago
      Is this meant to be a refutation? I didn't say that I had only "heard about" LLMs. I said I played around with them. If you were being faithful to what I wrote, you would take in the full thing, including the part about how LLMs become less fun and surprising the more you play with them. Planes do not fly less effectively the more experienced you become at piloting them. So I can only conclude that you did not come here to have a discussion in good faith. So what are you really after? Do you think you'll change my mind with your poetry?
      - famouswaffles 3y ago
        It's an analogy for your flawed thinking. The situation is even worse than my story outlined because you understand neither the workings of the brain and how it is intelligent nor what the computation that preceeded a prediction of a language model represents. You don't know x and you don't y but you are convinced x=/y. >the part about how LLMs become less fun and surprising the more you play with them. Do they ? I don't think so. >Planes do not fly less effectively the more experienced you become at piloting them. How well a plane flies is an objective measure. How "fun" and "surprising" something is is largely subjective and can change on grounds that have nothing to do with what is being evaluated. Your refutation boils down to "It's just linear algebra!". Setting aside the fact that you can be reductionist about anything including biology, Okay? So what? Results matter not vague and meaningless philosophical ramblings.
- ilaksh 3y ago
  Sure, we definitely are seeing some level/type of general intelligence, but we still want to distinguish between general intelligence and superintelligence. That's a big part of the problem, people thinking imprecisely and using "AGI" and "ASI" interchangeably.
  - RetroTechie 3y ago
    GI is a concept, SI is a matter of degree imho. What sets humans apart from AI right now, is specific types of out-of-the-box thinking. Let's say there's a bridge, an AI gets to control a robot, and is awarded points for crossing that bridge. I think you can be pretty sure that over time, the AI will figure out how to control the robot's limbs, move toward opposite side of the bridge, and find its way around obstacles. Meanwhile, humans may walk back & forth, because they enjoy crossing the bridge. Or stand in the middle for an hour, watching boats pass underneath. Some asshats may climb it. On a hot sunny day, others may go halfway & then jump off for a refreshing swim. Yet others may do some protest, and use the bridge as a scaffold to hang a banner from. Etc etc. Much such behavior defies logic. But yet, the bridge will primarily be used as intended, to cross from side to side. And humans go about in this fashion in many if not all their activities. When I see AIs that show similar creative / out-of-box behavior, while remaining mostly goal-oriented, and learn from its mistakes without humans nudging it every step of the way, then I think we'll have achieved AGI. Beyond that, SI is just a matter of scale & efficiency.
bdb7 3y ago
They are proposing whoever can get a job in corporate wonderland is super intelligent. When there is enough evidence corporate wonderland is overflowing with one dimensional intelligence cause their goals and what gets rewarded is extremely narrow. Einstein and Newton will fail the test 100%.

abetusk 3y ago

I think this is a pretty solid test, thanks. Maybe we can call it the Bartleby test?

mensetmanusman 3y ago
If it passes so many jobs go poof. Capital owners will then rejoice in their golden swimming pools.

thiht 3y ago

I think there's something to say about how we tend to define an intelligent system as... able to work and satisfy an employer. Not judging, this is actually my answer as well, but it's interesting. We could define AGI as "someone" (for lack of a better word) fun to talk to, or interesting on many subjects, or with a good personality, or we could use it to teach kids, or even all students, or as a shrink, or as a replacement for a friend. But we choose "able to please a boss" for this definition of "intelligent".