4 ms·

The Table -https://imgur.com/a/Ta848Lu https://imgur.com/a/Ta848Lu Instead of one acronym that now means a million different things to a million different peop

The Table -https://imgur.com/a/Ta848Lu https://imgur.com/a/Ta848Lu Instead of one acronym that now means a million different things to a million different people, there are "levels" that correspond to what percentage of skilled human workers it can replace.

School-Cotton 3y ago
Computers have been far better than the best human at chess for well over ten years. It’s weird to see “Stockfish (2023)” cited here.
TeMPOraL 3y ago
I need to read the full paper now, but going from just the table you posted, I see a problem with their "Narrow" classification, in that it doesn't correct for task characteristics and obscurity - which means the "Narrow" rating isn't useful without also giving the specific task to which it applies, and overall it feels off. I mean, they put GPT-4 ("SOTA LLMs") for a subset of tasks at Level 2N, a spell checker at Level 3N, and chess and Go solvers at Levels 4N and 5N. It feels to me that any useful grading scheme would give the opposite ordering. Sure, the current classification meets their definitions - "outperforms X% humans" - but at least with chess and Go, the rating is dominated entirely by the fact that very few people play these games at non-joke level, and the activity itself is super specific, and super narrowly scoped, mathematically speaking. Feels almost like calling CPUs Level 5N at "adding numbers together", since they clearly outperform 100% of humans at that. Or, much less jokingly, one could convincingly argue a PID controller is Level 5 Narrow, because not only it's going to be outperforming 100% of humans at its task, it's also learning and adapting (within its operational envelope).
- famouswaffles 3y ago
  Computer Chess and Go is so far ahead of human ability now that it doesn't matter how many humans decide to take up the game seriously. I see your broader point but it doesn't really matter here. Yes what is designated a task is a fair bit arbitrary but game playing AI's have been popularly designated narrow (well basically anything besides LLMs are narrow) for a long time now.
  - TeMPOraL 3y ago
    Sure. My point is that this classification is useful only within its category. Say I have a specific problem, like "play chess" or "play StarCraft" or "do maths" or "spellcheck essays". It would be great to have a task-specific list of algorithms ranked by their "Narrow" levels. It would help with use cases like: "oh, model X here spellchecks at level 5N, but is super complex, meanwhile algorithm Y is only Level 3N, but it fits on a napkin; Y is more than sufficient for my MVP". But this rating doesn't let us compare between tasks. That's fine by itself, but then it doesn't make sense to use the same rating for both specialized and general algorithms.
- usaar333 3y ago
  I don't even understand what the "AI" means in "narrow" AI. Why is a compiler at Level 0 rather than Level 5?
  - famouswaffles 3y ago
    Compilers aren't thought of as being intelligent. That's why they're level 0
    - usaar333 3y ago
      What's intelligence though? GOFAI (level 1) is just logic and search, which are properties of modern optimizing compilers as well. Even Deep Blue (level 4) is just logic and search.
      - famouswaffles 3y ago
        Yes what's defined as intelligent is a fair bit arbitrary.
ben_w 3y ago
It's a start. I think "general" needs to be more than a boolean though: The standard "wide range of non-physical tasks, including metacognitive abilities like learning new skills" is met by ChatGPT (as in the chart), which is much better in some languages than others, and at some tasks than others. But how wide does it have to be to count as "wide"? A brain upload of any human ought to count, but most of us have much less breadth than any LLM.
adampwells 3y ago
I think they make a category error by putting ChatGPT etc in the General column. As far as I can tell we only have narrow definitions of 'intelligence' and ChatGPT falls into one of those. I don't know of a general agreement on what 'General Intelligence' is in people, so how can we categorise anything is AGI? Knowing a bit about how ChatGPT works I feel it is a lot more like a chess program than a human.
- famouswaffles 3y ago
  That's not a category error. GPT is general. It is able to perform many tasks (Creative Writing, playing Chess, Poker and other games, language translation, Code, robot piloting) etc
  - seanhunter 3y ago
    But not make toast. That's a general task that very nearly any human (intelligent or otherwise) can do. If we define "general" as tasks with text input and output we are severely restricting the domain.
    - famouswaffles 3y ago
      There are millions of humans who can't make toast and will never be able to. At any rate, no it's not really limiting. https://tidybot.cs.princeton.edu/ https://tidybot.cs.princeton.edu/ https://deepmind.google/discover/blog/rt-2-new-model-translates-vision-and-language-into-action/ https://deepmind.google/discover/blog/rt-2-new-model-transla... https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-...
    - ben_w 3y ago
      Arguments like yours are why I regard "generality" (certainly in the context of AGI) as a continuum rather than a boolean. AlphaZero is more general than AlphaGo Zero, as the former can do three games and the latter only one. All LLMs are much more general than those game playing models, even if they aren't so wildly superhuman on any specific skill, and `gpt-4-vision-preview` is more general than any 3.5 model as 4 can take image inputs while the 3.5's can't.
gumballindie 3y ago
> to what percentage of skilled human workers it can replace Any levels of insanity that makes people believe such statements?
- Philpax 3y ago
  Here’s an article by the Financial Times demonstrating its real-world impact today: https://archive.is/hlcL2 https://archive.is/hlcL2 More capable systems will have a larger impact still.
  - gumballindie 3y ago
    Paid marketing content isn’t really relevant. Jobs are being lost left and right due to the overall economy. The only valid takeaway is that BCG consultants are such low quality that a chat bot can improve their productivity. I’d avoid such consultants, but that’s already known. Additionally, chatgpt is the running joke - full of mistakes and misinformation. Probably appealing to that market segment that’s easily gullible and falls for whatever the running conspiracy theory is popular at the time.

School-Cotton 3y ago

Computers have been far better than the best human at chess for well over ten years. It’s weird to see “Stockfish (2023)” cited here.

TeMPOraL 3y ago

I need to read the full paper now, but going from just the table you posted, I see a problem with their "Narrow" classification, in that it doesn't correct for task characteristics and obscurity - which means the "Narrow" rating isn't useful without also giving the specific task to which it applies, and overall it feels off. I mean, they put GPT-4 ("SOTA LLMs") for a subset of tasks at Level 2N, a spell checker at Level 3N, and chess and Go solvers at Levels 4N and 5N. It feels to me that any useful grading scheme would give the opposite ordering. Sure, the current classification meets their definitions - "outperforms X% humans" - but at least with chess and Go, the rating is dominated entirely by the fact that very few people play these games at non-joke level, and the activity itself is super specific, and super narrowly scoped, mathematically speaking. Feels almost like calling CPUs Level 5N at "adding numbers together", since they clearly outperform 100% of humans at that. Or, much less jokingly, one could convincingly argue a PID controller is Level 5 Narrow, because not only it's going to be outperforming 100% of humans at its task, it's also learning and adapting (within its operational envelope).

famouswaffles 3y ago
Computer Chess and Go is so far ahead of human ability now that it doesn't matter how many humans decide to take up the game seriously. I see your broader point but it doesn't really matter here. Yes what is designated a task is a fair bit arbitrary but game playing AI's have been popularly designated narrow (well basically anything besides LLMs are narrow) for a long time now.
- TeMPOraL 3y ago
  Sure. My point is that this classification is useful only within its category. Say I have a specific problem, like "play chess" or "play StarCraft" or "do maths" or "spellcheck essays". It would be great to have a task-specific list of algorithms ranked by their "Narrow" levels. It would help with use cases like: "oh, model X here spellchecks at level 5N, but is super complex, meanwhile algorithm Y is only Level 3N, but it fits on a napkin; Y is more than sufficient for my MVP". But this rating doesn't let us compare between tasks. That's fine by itself, but then it doesn't make sense to use the same rating for both specialized and general algorithms.
usaar333 3y ago
I don't even understand what the "AI" means in "narrow" AI. Why is a compiler at Level 0 rather than Level 5?
- famouswaffles 3y ago
  Compilers aren't thought of as being intelligent. That's why they're level 0
  - usaar333 3y ago
    What's intelligence though? GOFAI (level 1) is just logic and search, which are properties of modern optimizing compilers as well. Even Deep Blue (level 4) is just logic and search.
    - famouswaffles 3y ago
      Yes what's defined as intelligent is a fair bit arbitrary.

ben_w 3y ago

It's a start. I think "general" needs to be more than a boolean though: The standard "wide range of non-physical tasks, including metacognitive abilities like learning new skills" is met by ChatGPT (as in the chart), which is much better in some languages than others, and at some tasks than others. But how wide does it have to be to count as "wide"? A brain upload of any human ought to count, but most of us have much less breadth than any LLM.

adampwells 3y ago

I think they make a category error by putting ChatGPT etc in the General column. As far as I can tell we only have narrow definitions of 'intelligence' and ChatGPT falls into one of those. I don't know of a general agreement on what 'General Intelligence' is in people, so how can we categorise anything is AGI? Knowing a bit about how ChatGPT works I feel it is a lot more like a chess program than a human.

famouswaffles 3y ago
That's not a category error. GPT is general. It is able to perform many tasks (Creative Writing, playing Chess, Poker and other games, language translation, Code, robot piloting) etc
- seanhunter 3y ago
  But not make toast. That's a general task that very nearly any human (intelligent or otherwise) can do. If we define "general" as tasks with text input and output we are severely restricting the domain.
  - famouswaffles 3y ago
    There are millions of humans who can't make toast and will never be able to. At any rate, no it's not really limiting. https://tidybot.cs.princeton.edu/ https://tidybot.cs.princeton.edu/ https://deepmind.google/discover/blog/rt-2-new-model-translates-vision-and-language-into-action/ https://deepmind.google/discover/blog/rt-2-new-model-transla... https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-...
  - ben_w 3y ago
    Arguments like yours are why I regard "generality" (certainly in the context of AGI) as a continuum rather than a boolean. AlphaZero is more general than AlphaGo Zero, as the former can do three games and the latter only one. All LLMs are much more general than those game playing models, even if they aren't so wildly superhuman on any specific skill, and `gpt-4-vision-preview` is more general than any 3.5 model as 4 can take image inputs while the 3.5's can't.

gumballindie 3y ago

> to what percentage of skilled human workers it can replace Any levels of insanity that makes people believe such statements?

Philpax 3y ago
Here’s an article by the Financial Times demonstrating its real-world impact today: https://archive.is/hlcL2 https://archive.is/hlcL2 More capable systems will have a larger impact still.
- gumballindie 3y ago
  Paid marketing content isn’t really relevant. Jobs are being lost left and right due to the overall economy. The only valid takeaway is that BCG consultants are such low quality that a chat bot can improve their productivity. I’d avoid such consultants, but that’s already known. Additionally, chatgpt is the running joke - full of mistakes and misinformation. Probably appealing to that market segment that’s easily gullible and falls for whatever the running conspiracy theory is popular at the time.