2 ms·

For instance, go to the bottom of page 8 and see this: > Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among oth

For instance, go to the bottom of page 8 and see this: > Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things: > 1. There cannot exist any efficient real-world algorithm with goal G. > 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail. > ...4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm). > ...6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G. > All of these seem extraordinarily strong claims to make! The OP doesn't consider any of these.

wzdd 8y ago
The OP's point is: 1. The "superintelligence" crowd are capable of making very convincing arguments which he simply cannot refute on their own terms (I guess he'd say your example is one of them) 2. Since he can't refute them, if he engages with them, he will probably end up accepting them 3. However other groups exist which a) present equally-convincing (to the OP) arguments; and b) believe in a future which contradicts that espoused by the superintelligence crowd 4. With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set. This is not a conclusion which sits well with tech people, because they like to assume they are capable of engaging with each set of arguments and deciding on the most likely set. OP's point is that because each group is really good at making these arguments, it's easy to get this decision-making process wrong. I think that's a pretty great point, but it requires accepting the fact that you may be less than perfect at picking apart 'superingelligence' arguments and premises.
- jessriedel 8y ago
  > However other groups exist which a) present equally-convincing (to the OP) arguments If he thinks those convincing argument exist, he should present or link to them. It doesn't make sense to link to one side and then say "equally convincing counterarguments exist". > With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set. The proper response to this, if you have given up on assessing the correct conclusion, would be to retain uncertainty, not, as the OP has done, exude confidence that the superintelligence arguments are wrong.
aqsalose 8y ago
The claim amounts to saying that the space of definable goals and algorithms for achieving is large, and it would be a strong claim to say some particular corner of that space is impossible. However, from "improbably thing X is not impossible" does not follow that "X is probable". The paper says nothing conclusive on how the mass of probable algorithms is distributed among that space; what it does is to present some arguments formalish-appearing manner which entice reader's intuitions to answer some questions in a way that they would not otherwise. It does not make their claims sound. When presented with abstract forms of argument, human intuitions are often way off. This is why freshmen in mathematics programs usually spend their first year or so proving calculus and many other theorems from scratch and quite laboriously considering how obvious they are when sounded out ( https://en.wikipedia.org/wiki/Intermediate_value_theorem https://en.wikipedia.org/wiki/Intermediate_value_theorem ). The reason is that that with tools of mathematical analysis many unintuitive things ( https://en.wikipedia.org/wiki/Weierstrass_function https://en.wikipedia.org/wiki/Weierstrass_function , https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox ) may also be said, and so the correct answer is not to trust intuition but proof until student has developed intuition about what really makes calculus to tick. For example, to paraphrase the argument Armstrong presents on p. 16 "consider all superintelligences that we theoretically could build, is it likely that them having some particular goal would be impossible"? Frankly, I don't know: what is the typical goal, what is the typical path of superintelligence that could be built? Maybe the answer is yes, it would be impossible, and it is only wording that makes it sound unlikely because it invites us to think about large spaces ("all possible X") and small portions of them ("particular goal G") in a certain way. Thinking about space of all possible algorithms and goals, especially about the subset of algorithms that include all kind of intelligent behavior is bound to be unintuitive. The set to which they would "converge to" may not be small, but it still could exclude vast amount of goals, because why would a research team would want to (or even be capable of creating) a creature with blatantly orthogonal goals even if one can draw a hypothetical space of such goals, abstracting away all the important details? (Secondly, rereading, I believe Armstrong misrepresents the counterarguments by presenting them in extremely strong-looking forms -- convergence thesis, incompleteness thesis -- and tearing them down by arguing that surely it is not totally impossible that something could happen.) Translated to slightly more formal and mathematical-sounding argument, the author of the talk linked above claims that likely paths for complex minds will not involve them desiring orthogonal goals, because chances of a non-human artificial complex mind arising uniformly randomly from the space of potential algorithms with the vast space of potential goals is negligible: if such being will created, it will be precisely created (or evolved or whatever) following similar principles as the other complex minds on the planet (or by having them as a starting point). In other words, the space of potential algorithms we should concern ourselves with is severely restricted. (All this assuming that agents and goals is even a sensible framework to model how creatures we call intelligent operate.)

wzdd 8y ago

The OP's point is: 1. The "superintelligence" crowd are capable of making very convincing arguments which he simply cannot refute on their own terms (I guess he'd say your example is one of them) 2. Since he can't refute them, if he engages with them, he will probably end up accepting them 3. However other groups exist which a) present equally-convincing (to the OP) arguments; and b) believe in a future which contradicts that espoused by the superintelligence crowd 4. With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set. This is not a conclusion which sits well with tech people, because they like to assume they are capable of engaging with each set of arguments and deciding on the most likely set. OP's point is that because each group is really good at making these arguments, it's easy to get this decision-making process wrong. I think that's a pretty great point, but it requires accepting the fact that you may be less than perfect at picking apart 'superingelligence' arguments and premises.

jessriedel 8y ago
> However other groups exist which a) present equally-convincing (to the OP) arguments If he thinks those convincing argument exist, he should present or link to them. It doesn't make sense to link to one side and then say "equally convincing counterarguments exist". > With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set. The proper response to this, if you have given up on assessing the correct conclusion, would be to retain uncertainty, not, as the OP has done, exude confidence that the superintelligence arguments are wrong.

aqsalose 8y ago

The claim amounts to saying that the space of definable goals and algorithms for achieving is large, and it would be a strong claim to say some particular corner of that space is impossible. However, from "improbably thing X is not impossible" does not follow that "X is probable". The paper says nothing conclusive on how the mass of probable algorithms is distributed among that space; what it does is to present some arguments formalish-appearing manner which entice reader's intuitions to answer some questions in a way that they would not otherwise. It does not make their claims sound. When presented with abstract forms of argument, human intuitions are often way off. This is why freshmen in mathematics programs usually spend their first year or so proving calculus and many other theorems from scratch and quite laboriously considering how obvious they are when sounded out ( https://en.wikipedia.org/wiki/Intermediate_value_theorem https://en.wikipedia.org/wiki/Intermediate_value_theorem ). The reason is that that with tools of mathematical analysis many unintuitive things ( https://en.wikipedia.org/wiki/Weierstrass_function https://en.wikipedia.org/wiki/Weierstrass_function , https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox ) may also be said, and so the correct answer is not to trust intuition but proof until student has developed intuition about what really makes calculus to tick. For example, to paraphrase the argument Armstrong presents on p. 16 "consider all superintelligences that we theoretically could build, is it likely that them having some particular goal would be impossible"? Frankly, I don't know: what is the typical goal, what is the typical path of superintelligence that could be built? Maybe the answer is yes, it would be impossible, and it is only wording that makes it sound unlikely because it invites us to think about large spaces ("all possible X") and small portions of them ("particular goal G") in a certain way. Thinking about space of all possible algorithms and goals, especially about the subset of algorithms that include all kind of intelligent behavior is bound to be unintuitive. The set to which they would "converge to" may not be small, but it still could exclude vast amount of goals, because why would a research team would want to (or even be capable of creating) a creature with blatantly orthogonal goals even if one can draw a hypothetical space of such goals, abstracting away all the important details? (Secondly, rereading, I believe Armstrong misrepresents the counterarguments by presenting them in extremely strong-looking forms -- convergence thesis, incompleteness thesis -- and tearing them down by arguing that surely it is not totally impossible that something could happen.) Translated to slightly more formal and mathematical-sounding argument, the author of the talk linked above claims that likely paths for complex minds will not involve them desiring orthogonal goals, because chances of a non-human artificial complex mind arising uniformly randomly from the space of potential algorithms with the vast space of potential goals is negligible: if such being will created, it will be precisely created (or evolved or whatever) following similar principles as the other complex minds on the planet (or by having them as a starting point). In other words, the space of potential algorithms we should concern ourselves with is severely restricted. (All this assuming that agents and goals is even a sensible framework to model how creatures we call intelligent operate.)