9 ms·

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level a

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coordinated-human-action-example-superhuman-intelligence/ https://aiimpacts.org/coordinated-human-action-example-super... In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware. If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be. The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them. However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.

Retric 9y ago
This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.
- nopinsight 9y ago
  I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1] Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage. [1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.
  - coolaliasbro 9y ago
    > I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it. Assuming an AGI's consciousness would be anything like a human's, changes to its moral core might be largely dictated by environment, in which case it would be a good idea to be pals.
  - s1dechnl 9y ago
    [AGI Developer] A controller/overseer can easily limit/block this sufficiently and securely. Were talking about hardware/software. There are systems/standardized approaches to solving this problem. The 'Control/Safety' problem for AI are lauded as theoretical and new. However, they are not. They are solved by industry standard approaches day in and out. Any seasoned/experienced engineer in this field could solve this with known approaches. > Since an AGI can copy itself and be available at a multitude of access points at once Same comment above applies. This can only occur if done by a controller/overseer. Real-life isn't a sci-fi movie... There's engineering involved. > AGI changing x,y,z Not possible unless it is given access. Solved easily in industry standard ways.
    - nopinsight 9y ago
      Has the industry always been able to prevent smart, persistent actors from breaking access locks? Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks?
      - s1dechnl 9y ago
        > Has the industry always been able to prevent smart, persistent actors from breaking the access locks? Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks? Yes, the industry has persistently been able to do this. It's why the whole world isn't falling apart as we speak. What limits the locks most often is cost not capability. As such, you are possibly mistaking one's business decision not to use a more costly lock for the lack of capability to create a capable lock. Furthermore, you mistakingly attributing the actor in this case. The actor in the case of AGI is in a carefully controlled/monitored box. Actors in the real-world are not. As such, please tell me how an absolutely monitored/restricted actor has the ability to go playing with locks that aren't within its reach? I have a more fundamental question even : Have you been able pick 'your locks' yet? Do you even know what they are? Where they are? Those capable of 'creation' hold certain things close to their chest .. The act of creation necessitates it and is [built in]. > Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks? Show me how you're able to break your 'locks' and you'll have an argument for how AGI can break its locks. I don't think you're grasping the level of 'locks' that I'm speaking about. Humans have been around for how long and still don't even know what their [locks] even are... or where they are. It's quite easy to example at a certain level of visibility how your scenario is unwarranted. I can draw direct parallels to eons of human history.
- coolaliasbro 9y ago
  > The smartest player possible would still lose poker to someone that can read their hand. And let's not forget those cases where the smartest player is the one that can read your hand. https://en.m.wikipedia.org/wiki/Stu_Ungar https://en.m.wikipedia.org/wiki/Stu_Ungar
- red75prime 9y ago
  Why the smartest player can't cheat too? Do you assume that there cannot be such a thing as generally smarter player?
mrfusion 9y ago
That counterargument makes a lot of sense. I wonder why I never see it discussed anywhere? It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq. Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.
- nopinsight 9y ago
  Thanks. I came up with that myself although I would like to see if someone can poke holes on it [1]. I am aware that many NP-hard problems, for example, usually have good approximate solutions in non-pathological cases. So that could be a big hole in that argument. Could a theoretical computer scientist give more insight on this? In particular, what is the hardness of approximate solutions to most problems that could have strong impact in the real world? [1] I did, in passing, find a similar idea discussed somewhere afterwards. If someone has a reference for that, I would be glad to have it.
  - s1dechnl 9y ago
    [AGI Developer] I'll instead give you something to consider : * The totality of mathematics is not fully defined. As such, there are can be completely new mathematics that destroys previous conceptions especially as it relates to the limits of computability. Theoretical Limits often reflect one's limited scope of perception. Your limits or the limits someone theorized ages ago aren't necessarily my limits if I find a path beyond them. Information theory and computational complexity theory are theories... Solving fundamental aspects of intelligence will invalidate a host of theories, create new mathematics, new algorithms, and new theories. Continuing to frame something as ground breaking as an understanding/implementation of intelligence in yester-years theoretical limits is flawed.
    - xamuel 9y ago
      In a phrase like "information theory", "theory" means "body of literature", not "unproven hypothesis". Information theory isn't really something that can be invalidated, no more than the Pythagorean Theorem can be invalidated.
s1dechnl 9y ago
[AGI Developer here] > the real barrier for AGI is software and not hardware. This is indeed true. It has actually been true for some time. AGI is computable on present day hardware if one has enough knowledge on how to correctly structure it. A fundamental understanding of intelligence is the first step. How one crafts this understanding into software is the second step. Hardware reached capability in recent years. > exploding intelligence There is no such thing. Time is still required like with all things. Teaching/learning/interaction is still required. Furthermore, A controller/overseer of the system can more than adequately limit progress they are not comfortable with. I find the idea of exploding intelligence/overnight super AI to be pure fantasy not at all aware as to the structure of AGI. > If an organization can marginally predict stock price movements better than the rest of the world.. The problem is this kind of thinking... AGI is achieved and people rush to apply it to games to get rich. Sorry, that will not occur. It will not occur because the stock market is fundamentally a [game]. A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. One could make all of the accurate predictions they wanted, if the game changes underneath you or before you can act, your lofty predictions have no real world value and that's exactly how the market behaves.
- nopinsight 9y ago
  From what I gather from your comment above and another one, the AGI you talk about does not include general intent or free will to act differently from what its creators anticipate. An AGI, perhaps a different variety from yours, can behave outside of our predictions in multiple ways [1], unless we truly solve the problem of constraining its will to a range that is acceptable to us. (I do not believe anyone has solved that. Ref: A talk by Prof. Stuart Russell, AAAI fellow and author of the standard textbook on AI: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai https://www.ted.com/talks/stuart_russell_3_principles_for_cr...) [1] Note that there are incentives for at least some groups to develop a highly capable AGI with the characteristics I describe. > Time is still required like with all things. I agree that time is required but an AGI can multiply itself and collect all requisite information from the world quite quickly. There is so much available to learn just from the Internet if it knows how to learn independently like humans do. Computational resources might be a bottleneck but presumably a human-level AGI can at least do online work at a minimum wage (e.g. translating documents, simple accounting, ...). It can parallelly execute many 'brains' to accomplish more work to acquire more resources, to do more work profitably.... Humans are limited by 24 hours a day. An AGI can over a fairly short amount of time (months) accumulate sufficient resources to make thousands or millions of its copies perhaps with variations to specialize for different kinds of work. Over time, it should gain experience to perform more and more highly-valued work as well. > A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. A smart AGI can form alliance and share benefits with inside players and execute any cunning strategies not available to humans who at least need to take into account law enforcement. There are many other advantages an AGI has over human organizations (some of which I mentioned above).
mannykannot 9y ago
I am not entirely convinced by the 'most problems are exponentially hard' argument, as problem-solving often has synergistic effects - e.g. consider calculus, developed in solving the problem of planetary orbits, but useful for so much more. On the other hand, trying to discern a general rule from historical data is complicated by the fact that the number of problem-solvers has been growing exponentially.
- whereismypwd 9y ago
  I think the better argument is that most "breakthroughs" are exponentially hard, or we wouldn't consider them breakthroughs!
  - mannykannot 9y ago
    'Exponential' means something more specific than 'big' or 'largest in class'.
    - whereismypwd 9y ago
      Exponential is probably the easiest way to say "a search without usable gradients in a high-dimensional problem space."
      - mannykannot 9y ago
        'Breakthrough' means something less specific than 'a solution to a problem having a search without usable gradients in a high-dimensional problem space.' This is rather beside the point, however, as the the claim that kicked off this thread (most problems are exponentially hard) is about the difficulty of problem-solving in general, not the subset of problems that are pretty much hard by definition, in that their solution was/would be a breakthrough.

Retric 9y ago

This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.

nopinsight 9y ago
I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1] Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage. [1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.
- coolaliasbro 9y ago
  > I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it. Assuming an AGI's consciousness would be anything like a human's, changes to its moral core might be largely dictated by environment, in which case it would be a good idea to be pals.
- s1dechnl 9y ago
  [AGI Developer] A controller/overseer can easily limit/block this sufficiently and securely. Were talking about hardware/software. There are systems/standardized approaches to solving this problem. The 'Control/Safety' problem for AI are lauded as theoretical and new. However, they are not. They are solved by industry standard approaches day in and out. Any seasoned/experienced engineer in this field could solve this with known approaches. > Since an AGI can copy itself and be available at a multitude of access points at once Same comment above applies. This can only occur if done by a controller/overseer. Real-life isn't a sci-fi movie... There's engineering involved. > AGI changing x,y,z Not possible unless it is given access. Solved easily in industry standard ways.
  - nopinsight 9y ago
    Has the industry always been able to prevent smart, persistent actors from breaking access locks? Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks?
    - s1dechnl 9y ago
      > Has the industry always been able to prevent smart, persistent actors from breaking the access locks? Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks? Yes, the industry has persistently been able to do this. It's why the whole world isn't falling apart as we speak. What limits the locks most often is cost not capability. As such, you are possibly mistaking one's business decision not to use a more costly lock for the lack of capability to create a capable lock. Furthermore, you mistakingly attributing the actor in this case. The actor in the case of AGI is in a carefully controlled/monitored box. Actors in the real-world are not. As such, please tell me how an absolutely monitored/restricted actor has the ability to go playing with locks that aren't within its reach? I have a more fundamental question even : Have you been able pick 'your locks' yet? Do you even know what they are? Where they are? Those capable of 'creation' hold certain things close to their chest .. The act of creation necessitates it and is [built in]. > Why should we assume that an AGI which can accumulate experience over time, gain more knowledge, and make connections with others, including human actors, will not ever be able to break the locks? Show me how you're able to break your 'locks' and you'll have an argument for how AGI can break its locks. I don't think you're grasping the level of 'locks' that I'm speaking about. Humans have been around for how long and still don't even know what their [locks] even are... or where they are. It's quite easy to example at a certain level of visibility how your scenario is unwarranted. I can draw direct parallels to eons of human history.
coolaliasbro 9y ago
> The smartest player possible would still lose poker to someone that can read their hand. And let's not forget those cases where the smartest player is the one that can read your hand. https://en.m.wikipedia.org/wiki/Stu_Ungar https://en.m.wikipedia.org/wiki/Stu_Ungar
red75prime 9y ago
Why the smartest player can't cheat too? Do you assume that there cannot be such a thing as generally smarter player?

mrfusion 9y ago

That counterargument makes a lot of sense. I wonder why I never see it discussed anywhere? It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq. Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.

nopinsight 9y ago
Thanks. I came up with that myself although I would like to see if someone can poke holes on it [1]. I am aware that many NP-hard problems, for example, usually have good approximate solutions in non-pathological cases. So that could be a big hole in that argument. Could a theoretical computer scientist give more insight on this? In particular, what is the hardness of approximate solutions to most problems that could have strong impact in the real world? [1] I did, in passing, find a similar idea discussed somewhere afterwards. If someone has a reference for that, I would be glad to have it.
- s1dechnl 9y ago
  [AGI Developer] I'll instead give you something to consider : * The totality of mathematics is not fully defined. As such, there are can be completely new mathematics that destroys previous conceptions especially as it relates to the limits of computability. Theoretical Limits often reflect one's limited scope of perception. Your limits or the limits someone theorized ages ago aren't necessarily my limits if I find a path beyond them. Information theory and computational complexity theory are theories... Solving fundamental aspects of intelligence will invalidate a host of theories, create new mathematics, new algorithms, and new theories. Continuing to frame something as ground breaking as an understanding/implementation of intelligence in yester-years theoretical limits is flawed.
  - xamuel 9y ago
    In a phrase like "information theory", "theory" means "body of literature", not "unproven hypothesis". Information theory isn't really something that can be invalidated, no more than the Pythagorean Theorem can be invalidated.

s1dechnl 9y ago

[AGI Developer here] > the real barrier for AGI is software and not hardware. This is indeed true. It has actually been true for some time. AGI is computable on present day hardware if one has enough knowledge on how to correctly structure it. A fundamental understanding of intelligence is the first step. How one crafts this understanding into software is the second step. Hardware reached capability in recent years. > exploding intelligence There is no such thing. Time is still required like with all things. Teaching/learning/interaction is still required. Furthermore, A controller/overseer of the system can more than adequately limit progress they are not comfortable with. I find the idea of exploding intelligence/overnight super AI to be pure fantasy not at all aware as to the structure of AGI. > If an organization can marginally predict stock price movements better than the rest of the world.. The problem is this kind of thinking... AGI is achieved and people rush to apply it to games to get rich. Sorry, that will not occur. It will not occur because the stock market is fundamentally a [game]. A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. One could make all of the accurate predictions they wanted, if the game changes underneath you or before you can act, your lofty predictions have no real world value and that's exactly how the market behaves.

nopinsight 9y ago
From what I gather from your comment above and another one, the AGI you talk about does not include general intent or free will to act differently from what its creators anticipate. An AGI, perhaps a different variety from yours, can behave outside of our predictions in multiple ways [1], unless we truly solve the problem of constraining its will to a range that is acceptable to us. (I do not believe anyone has solved that. Ref: A talk by Prof. Stuart Russell, AAAI fellow and author of the standard textbook on AI: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai https://www.ted.com/talks/stuart_russell_3_principles_for_cr...) [1] Note that there are incentives for at least some groups to develop a highly capable AGI with the characteristics I describe. > Time is still required like with all things. I agree that time is required but an AGI can multiply itself and collect all requisite information from the world quite quickly. There is so much available to learn just from the Internet if it knows how to learn independently like humans do. Computational resources might be a bottleneck but presumably a human-level AGI can at least do online work at a minimum wage (e.g. translating documents, simple accounting, ...). It can parallelly execute many 'brains' to accomplish more work to acquire more resources, to do more work profitably.... Humans are limited by 24 hours a day. An AGI can over a fairly short amount of time (months) accumulate sufficient resources to make thousands or millions of its copies perhaps with variations to specialize for different kinds of work. Over time, it should gain experience to perform more and more highly-valued work as well. > A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. A smart AGI can form alliance and share benefits with inside players and execute any cunning strategies not available to humans who at least need to take into account law enforcement. There are many other advantages an AGI has over human organizations (some of which I mentioned above).

mannykannot 9y ago

I am not entirely convinced by the 'most problems are exponentially hard' argument, as problem-solving often has synergistic effects - e.g. consider calculus, developed in solving the problem of planetary orbits, but useful for so much more. On the other hand, trying to discern a general rule from historical data is complicated by the fact that the number of problem-solvers has been growing exponentially.

whereismypwd 9y ago
I think the better argument is that most "breakthroughs" are exponentially hard, or we wouldn't consider them breakthroughs!
- mannykannot 9y ago
  'Exponential' means something more specific than 'big' or 'largest in class'.
  - whereismypwd 9y ago
    Exponential is probably the easiest way to say "a search without usable gradients in a high-dimensional problem space."
    - mannykannot 9y ago
      'Breakthrough' means something less specific than 'a solution to a problem having a search without usable gradients in a high-dimensional problem space.' This is rather beside the point, however, as the the claim that kicked off this thread (most problems are exponentially hard) is about the difficulty of problem-solving in general, not the subset of problems that are pretty much hard by definition, in that their solution was/would be a breakthrough.