9 ms·

OP here, nonplussed that his obscure blog suddenly is near the top of Hacker News. First, I don't think that corporations are nothing but parasites, and said a

by mtraven 13y ago

OP here, nonplussed that his obscure blog suddenly is near the top of Hacker News. First, I don't think that corporations are nothing but parasites, and said as much in the article. I did say that finance was parasitical. That was probably too strong. Allocation of capital is a useful function; the problem is that the allocators are helping themselves to a far larger cut of the proceedings than the value they produce (and I don't particularly want to argue that here, just wanted to clarify what I meant). Second, the more central point about whether the Singularity Institute types are wasting their time and should refocus their thinking. It is true that not many people are thinking about the effects of superhuman AIs, and there's no harm and possibly some good in a few smart weirdos focusing on that, especially since corporations have been around and critiqued for so long. However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. But I take believe they are seriously interested in world improvement, so my suggestion to them in their work on "benevolent goal architectures" is to study how existing goal architectures work, that is to look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing. For my part, I am in the process of trying to understand their friendly AI work better so I critique it from a more informed standpoint.

topynate 13y ago
> Allocation of capital is a useful function; the problem is that the allocators are helping themselves to a far larger cut of the proceedings than the value they produce Yeah, I find this (that financial actors are acting in a hostile manner) more tenable than the impression I took from your blog post, which was that the financial system itself was acting with intentionality. > look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action I hope you can bring that perspective to the table; I've noticed that a lot of the public work is about preserving goals while modifying structure, while actually mediating between conflicting goals/values is a bit handwavey. And yeah, I hope we're far away from human level AI, but we're talking about our uncertainty on conceptual distance to an undiscovered theoretical framework, here. It's not quite as bad as trying to guess when the Riemann Hypothesis will be resolved, but it has that quality.
derefr 13y ago
> However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. Why do you think this? The idea behind the Singularity is that even an AI that starts far below human-level, but is capable of recursive self-modification, can bootstrap itself to superintelligence. We don't need to understand the human brain to create such an AI; we need only to: A. create a the simplest of "digital animals", and B. give it the capability to push itself into self-directed Lamarckian evolution to satisfy its preferences; C. and have this all happen on electronic time-scales (e.g. successive "generations" being created in milliseconds or less), and we'll likely see a Singularity. > look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing. "Existing economic and social systems" allow humans to cooperate with humans. The whole point of FAI research is to figure out how to make an AI that is "human"-enough that it would have cause to participate in such systems with us. A paperclip-maximizer has nothing to gain from participating in a stock market, or an election. People do not want to be turned into paperclips, and it does not want paperclip-resources turned to human ends. Its goals are not disparate, but rather incompatible with our own. A non-Friendly AI should not be conceptualized by your intuition as a certain kind of human that we can possibly learn to "live-and-let-live." It can, at best, be conceptualized as something like a Baby-Eating Alien[1]. Usually, though, it would seem to us something more akin to a supermassive black hole: an unlimited force that consumes many things we want and outputs nothing useful to us, that we must keep away from, and especially must avoid creating in the laboratory. [1] http://lesswrong.com/lw/y4/three_worlds_collide_08/ http://lesswrong.com/lw/y4/three_worlds_collide_08/
- mtraven 13y ago
  The fact that it hasn't happened yet is good evidence that it isn't as easy as all that. Also, evolution optimizes for survival and reproduction, not intelligence. While it may be possible to design an artificial evolutionary system that optimizes for superintelligence, I don't think doing so is trivial (or it would have been done). Let's posit that your image of post-singularity non-friendly AIs is accurate. They are an immense and irresistible force, like a black hole. I can imagine something like that, but I have a great deal of difficulty imagining something like that that is also constrained by initial design decisions made by humans before the singularity, which is what the friendly AI people hope to achieve. To be fair I haven't read all their stuff yet and I'm sure they address this point.
marcosdumay 13y ago
I see only 2 possible problems with that line of reasoning, that would justify thinking directly about computer inteligence, bypassing institutional inteligence: 1 - The researchers can't program those institutions; 2 - The researchers intend to use AI as a tool for that goal.
gisely 13y ago
I really like your articulation of this. It has struck me for a while now that the people working on friendly AI could have more impact if they invested their energy and intellectual prowess on the real threats to humanity we currently face rather than an amorphous future hypothetical we are ill prepared to understand. I also like the alternative problem that you posit: what can we do about existing semi-autonomous systems whose functioning is harmful to humanity? However, I don't feel very hopeful about any sort of solutions we might have to these problems. Semi-autonomous systems at societal level have existed for probably nearly as long as as humanity itself. Are there some good examples of such systems that were initially harmful but have been carefully cultivated by humanity to make their effects largely beneficial?
gwern 13y ago
> However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. At what point would you allow research to go forth? And you know, there's a lot of work that can be done which is not 'directly coding an AI'. > But I take believe they are seriously interested in world improvement, so my suggestion to them in their work on "benevolent goal architectures" is to study how existing goal architectures work, that is to look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing. But why would one expect any of that to transfer to AI? What do the internal dynamics of a Board of Directors have to do with, say, the Code Red worm? What lessons can we extract from 501(c)3 nonprofits which will tell us anything about deep-learning-based architectures? Do CEO salaries really inform our understanding of Moore's Law? Or can study of Congressional lobbying seriously help us better understand progress on brain scanning and connectomes? Do Marxian dialectics truly help us improve forecasts for when feasible investments in neuromorphic chips will match human brains? The closer I look at corporations and modern economies, the more worthless they seem for understanding the possibilities of AI, much less engineering safe or powerful ones. Modern economies are based on large assemblages of human brains, acculturated in very specific ways (remember Adam Smith's other major work: _The Theory of Moral Sentiments_), which are limited in many ways, and which are fragile and nonrobust: consider psychopathy, or consider economic's focus on 'institutions'. Why do some economies explode like South Korea, and others go nowhere at all? Even with millennia of human history and almost identical genomes and brains, outcomes are staggeringly different. (You complain about corporations; well, how 'friendly' is North Korea?) And this is supposed to be so useful for understanding the issues that we should be focused on your favored political goals instead of directly tackling the issues?
- mtraven 13y ago
  I think you are homing in on a key philosophical difference between me and LessWrongsters. Don't really have time to get into it now, except to say that it is kind of arrogant to think you can design or think about superintelligences without reference to the best existing intelligent systems we have. Especially if you want to keep them goal-compatible. The pathologies of such systems are especially instructive.
cornholio 13y ago
You have not demonstrated that corporations have artificial intelligence. At most we can say they are an artificial life form that thrives and takes over when the environment is right for them. So by changing the contents of the Petri dish (laws, regulations, democratic power, competing grassroots organism etc.) we can still control the outcome. Sure, the corporations will fight back through lobbying etc. but at the end of the day we are still talking about the human intelligence of the corporate masters versus the collective intelligence of the citizenry. So the strategy of the corporation is not a form of artificial intelligence. Most importantly, the corporate masters fully realize that they live in a symbiosis with the rest of the society so they will not pursue paths that threaten the human civilization, for example they will not engage in nuclear war. Unfriendly AI on the other hand, is an artificial life form capable of superhuman intelligence that has no necessary symbiosis with humans. It will view humans in the same way we look at apes: an inferior life form from our past evolution that can be preserved in reservations, studied and certainly destroyed if it grows to the point of competing for our resources. Bonus, they will likely be imune to high doses of radiations and have a fault tolerant distributed architecture specifically designed to survive nuclear war.
Houshalter 13y ago
>However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. Really we have no idea. True nothing around today is remotely close to human intelligence, but the question is; will AI be a slow gradual process, or will someone make a breakthrough and suddenly make a lot of progress very quickly? In any case, it won't happen at all if people aren't working on it.
pilooch 13y ago
question is: is evolution 'target' for us to build machines or to leverage machines to increase its pace ? if pure AI were the target evolution may have 'targeted' it more directly. Singularity 'theory' forgets about evolution.

by mtraven 13y ago

topynate 13y ago

> Allocation of capital is a useful function; the problem is that the allocators are helping themselves to a far larger cut of the proceedings than the value they produce Yeah, I find this (that financial actors are acting in a hostile manner) more tenable than the impression I took from your blog post, which was that the financial system itself was acting with intentionality. > look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action I hope you can bring that perspective to the table; I've noticed that a lot of the public work is about preserving goals while modifying structure, while actually mediating between conflicting goals/values is a bit handwavey. And yeah, I hope we're far away from human level AI, but we're talking about our uncertainty on conceptual distance to an undiscovered theoretical framework, here. It's not quite as bad as trying to guess when the Riemann Hypothesis will be resolved, but it has that quality.

derefr 13y ago

> However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. Why do you think this? The idea behind the Singularity is that even an AI that starts far below human-level, but is capable of recursive self-modification, can bootstrap itself to superintelligence. We don't need to understand the human brain to create such an AI; we need only to: A. create a the simplest of "digital animals", and B. give it the capability to push itself into self-directed Lamarckian evolution to satisfy its preferences; C. and have this all happen on electronic time-scales (e.g. successive "generations" being created in milliseconds or less), and we'll likely see a Singularity. > look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing. "Existing economic and social systems" allow humans to cooperate with humans. The whole point of FAI research is to figure out how to make an AI that is "human"-enough that it would have cause to participate in such systems with us. A paperclip-maximizer has nothing to gain from participating in a stock market, or an election. People do not want to be turned into paperclips, and it does not want paperclip-resources turned to human ends. Its goals are not disparate, but rather incompatible with our own. A non-Friendly AI should not be conceptualized by your intuition as a certain kind of human that we can possibly learn to "live-and-let-live." It can, at best, be conceptualized as something like a Baby-Eating Alien[1]. Usually, though, it would seem to us something more akin to a supermassive black hole: an unlimited force that consumes many things we want and outputs nothing useful to us, that we must keep away from, and especially must avoid creating in the laboratory. [1] http://lesswrong.com/lw/y4/three_worlds_collide_08/ http://lesswrong.com/lw/y4/three_worlds_collide_08/

mtraven 13y ago
The fact that it hasn't happened yet is good evidence that it isn't as easy as all that. Also, evolution optimizes for survival and reproduction, not intelligence. While it may be possible to design an artificial evolutionary system that optimizes for superintelligence, I don't think doing so is trivial (or it would have been done). Let's posit that your image of post-singularity non-friendly AIs is accurate. They are an immense and irresistible force, like a black hole. I can imagine something like that, but I have a great deal of difficulty imagining something like that that is also constrained by initial design decisions made by humans before the singularity, which is what the friendly AI people hope to achieve. To be fair I haven't read all their stuff yet and I'm sure they address this point.

marcosdumay 13y ago

I see only 2 possible problems with that line of reasoning, that would justify thinking directly about computer inteligence, bypassing institutional inteligence: 1 - The researchers can't program those institutions; 2 - The researchers intend to use AI as a tool for that goal.

gisely 13y ago

I really like your articulation of this. It has struck me for a while now that the people working on friendly AI could have more impact if they invested their energy and intellectual prowess on the real threats to humanity we currently face rather than an amorphous future hypothetical we are ill prepared to understand. I also like the alternative problem that you posit: what can we do about existing semi-autonomous systems whose functioning is harmful to humanity? However, I don't feel very hopeful about any sort of solutions we might have to these problems. Semi-autonomous systems at societal level have existed for probably nearly as long as as humanity itself. Are there some good examples of such systems that were initially harmful but have been carefully cultivated by humanity to make their effects largely beneficial?

gwern 13y ago

> However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. At what point would you allow research to go forth? And you know, there's a lot of work that can be done which is not 'directly coding an AI'. > But I take believe they are seriously interested in world improvement, so my suggestion to them in their work on "benevolent goal architectures" is to study how existing goal architectures work, that is to look at existing economic and social systems where disparate individual goals are coordinated into cooperative or conflicting action, and how the results are related to human goals and human flourishing. But why would one expect any of that to transfer to AI? What do the internal dynamics of a Board of Directors have to do with, say, the Code Red worm? What lessons can we extract from 501(c)3 nonprofits which will tell us anything about deep-learning-based architectures? Do CEO salaries really inform our understanding of Moore's Law? Or can study of Congressional lobbying seriously help us better understand progress on brain scanning and connectomes? Do Marxian dialectics truly help us improve forecasts for when feasible investments in neuromorphic chips will match human brains? The closer I look at corporations and modern economies, the more worthless they seem for understanding the possibilities of AI, much less engineering safe or powerful ones. Modern economies are based on large assemblages of human brains, acculturated in very specific ways (remember Adam Smith's other major work: _The Theory of Moral Sentiments_), which are limited in many ways, and which are fragile and nonrobust: consider psychopathy, or consider economic's focus on 'institutions'. Why do some economies explode like South Korea, and others go nowhere at all? Even with millennia of human history and almost identical genomes and brains, outcomes are staggeringly different. (You complain about corporations; well, how 'friendly' is North Korea?) And this is supposed to be so useful for understanding the issues that we should be focused on your favored political goals instead of directly tackling the issues?

mtraven 13y ago
I think you are homing in on a key philosophical difference between me and LessWrongsters. Don't really have time to get into it now, except to say that it is kind of arrogant to think you can design or think about superintelligences without reference to the best existing intelligent systems we have. Especially if you want to keep them goal-compatible. The pathologies of such systems are especially instructive.

cornholio 13y ago

You have not demonstrated that corporations have artificial intelligence. At most we can say they are an artificial life form that thrives and takes over when the environment is right for them. So by changing the contents of the Petri dish (laws, regulations, democratic power, competing grassroots organism etc.) we can still control the outcome. Sure, the corporations will fight back through lobbying etc. but at the end of the day we are still talking about the human intelligence of the corporate masters versus the collective intelligence of the citizenry. So the strategy of the corporation is not a form of artificial intelligence. Most importantly, the corporate masters fully realize that they live in a symbiosis with the rest of the society so they will not pursue paths that threaten the human civilization, for example they will not engage in nuclear war. Unfriendly AI on the other hand, is an artificial life form capable of superhuman intelligence that has no necessary symbiosis with humans. It will view humans in the same way we look at apes: an inferior life form from our past evolution that can be preserved in reservations, studied and certainly destroyed if it grows to the point of competing for our resources. Bonus, they will likely be imune to high doses of radiations and have a fault tolerant distributed architecture specifically designed to survive nuclear war.

Houshalter 13y ago

>However, we are so far away from producing anything remotely close to that level of AI, that I fear their work is ungrounded. It strikes me as fun quasi-SF rather than serious engineering. Really we have no idea. True nothing around today is remotely close to human intelligence, but the question is; will AI be a slow gradual process, or will someone make a breakthrough and suddenly make a lot of progress very quickly? In any case, it won't happen at all if people aren't working on it.

pilooch 13y ago

question is: is evolution 'target' for us to build machines or to leverage machines to increase its pace ? if pure AI were the target evolution may have 'targeted' it more directly. Singularity 'theory' forgets about evolution.