5 ms·

One thought that occurred to me as I was reading this — and I appreciated the logical rigor — is it seems there's generally an implicit assumption that you are

One thought that occurred to me as I was reading this — and I appreciated the logical rigor — is it seems there's generally an implicit assumption that you are comparing the cognitive abilities of a single typical human with a single typical AI. In the very least it's unclear what's being assumed. That is, we talk about the "abilities" of an AI system surpassing human intelligence, but it's a little unclear what's meant by the latter. Are we comparing an AI system against a single person, or against humans collectively? Usually there's a statement like Hypothesis 2 — "A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages" — but then also "although the capacity of a human brain is huge, its input/output channels are bandwidth-limited". But is the relevant comparison a single human brain, or a sort of collective system of humans? It seems a little misleading in a logical-comparative sense to talk about an AI system built on a collective human system like wikipedia, or the internet, in compared to a single person. I don't actually mean to question whether a superintelligent system surpassing all human abilities, individually or collectively, could be built. But it seems to be that the concerns about bandwidth and whatnot are a function of the problem definition, and not necessarily a single human system, when a task-solving exercise involves a collective output in both cases. I think going forward we're going to need much more clarity about what "superintelligence" really means. Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source. If/when AI starts focusing more on reasoning per se, it will probably also be necessary to better define whether there are some limits to discovery and reasoning, in the sense of bounds due to information about a problem domain versus reasoning ability limits. That is, sometimes we know the possible options but just don't have enough empirical information to decide between them; other times we lack explanatory options. Maybe that endeavor also is constrained by human abilities but it seems worthwhile to explore. Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else. That is, AI sort of proceeded in a more experimental way without mathematical theory driving it, so we have some catching up to do.

ilaksh 3y ago
Whether the limits are approximately human intelligence or not, we know that AI can operate orders of magnitude faster than humans. It is actually easy to see how something like GPT-4 could be accelerated by a factor of 50 in the next few years. Without any completely new ideas.
adamsmith143 3y ago
It's very clear that a strong AI/AGI would vastly surpass all humans combined. We've already seen shades of this with Chess AIs(all humans combined still can't beat a 3000 elo chess engine e.g.), hell even Kasparov vs the world showed showed us that the rest of the world combined can't even match a single high end human. >I think going forward we're going to need much more clarity about what "superintelligence" really means. This was already well defined by Bostrom in 1997: "An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills" >Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source. There are no indications at all that it is constrained to that level.
- NumberWangMan 3y ago
  I would add the caveat that Chess is a game where the nature of the rules mostly eliminate any advantage of numbers, which is not necessarily the case for other sorts of competition. I'm still quite concerned about the risk of rogue AI, as it's not even necessarily true that humans will have the advantage of numbers, or be able to bring our numbers to bear. And a smart AI will try to divide us and conquer, if it comes down to a conflict, including recruiting humans to its side.
pixl97 3y ago
>Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else Then I would say your expectations are a failure of your own imagination. The problem with humans being the smartest things on the block is we think we are optimized for intelligence. One problem with wanting to clarify what 'super intelligence' is, is we really have piss poor definitions of what intelligence is on a multidimensional gradient scale. The bounds and limits of intelligence are poorly defined, but I scenerly ask, why would you think that humanity would be anywhere near the upper bound of those limitations? Why would evolution choose 'the smartest possible global maxima' versus 'the smartest possible with 2 pounds of jelly in a 20 watt power envelope?'

ilaksh 3y ago

Whether the limits are approximately human intelligence or not, we know that AI can operate orders of magnitude faster than humans. It is actually easy to see how something like GPT-4 could be accelerated by a factor of 50 in the next few years. Without any completely new ideas.

adamsmith143 3y ago

It's very clear that a strong AI/AGI would vastly surpass all humans combined. We've already seen shades of this with Chess AIs(all humans combined still can't beat a 3000 elo chess engine e.g.), hell even Kasparov vs the world showed showed us that the rest of the world combined can't even match a single high end human. >I think going forward we're going to need much more clarity about what "superintelligence" really means. This was already well defined by Bostrom in 1997: "An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills" >Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source. There are no indications at all that it is constrained to that level.

NumberWangMan 3y ago
I would add the caveat that Chess is a game where the nature of the rules mostly eliminate any advantage of numbers, which is not necessarily the case for other sorts of competition. I'm still quite concerned about the risk of rogue AI, as it's not even necessarily true that humans will have the advantage of numbers, or be able to bring our numbers to bear. And a smart AI will try to divide us and conquer, if it comes down to a conflict, including recruiting humans to its side.

pixl97 3y ago

>Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else Then I would say your expectations are a failure of your own imagination. The problem with humans being the smartest things on the block is we think we are optimized for intelligence. One problem with wanting to clarify what 'super intelligence' is, is we really have piss poor definitions of what intelligence is on a multidimensional gradient scale. The bounds and limits of intelligence are poorly defined, but I scenerly ask, why would you think that humanity would be anywhere near the upper bound of those limitations? Why would evolution choose 'the smartest possible global maxima' versus 'the smartest possible with 2 pounds of jelly in a 20 watt power envelope?'