5 ms·

> And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the con

by maaku 11y ago

> And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.

varjag 11y ago
It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.
- maaku 11y ago
  We are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)? Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same. Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option." A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals. Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals. Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it. [0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?
  - varjag 11y ago
    Sorry for not being clear enough, you misunderstood me completely. I was arguing about emerging behaviour which might be unavoidably complex and not at all tractable even in a very regular system with simple, relatively well understood components. Take cellular automata, or your very own example, the human brain. As to the free will, it makes total sense as a concept when you consider it versus other intelligent actors, and indeed this is the only useful operational perspective unless you are a theist. E.g. my actions do not depend on your agenda very much, other than we have some synchronized exchange of comments here. Denying it exists is a popular but fruitless philosophic cop-out, akin to using Zeno's paradoxes to challenge general relativity. I however used 'agency' to emphasize functional aspects of the behaviour. The emergent behaviour that might or might be not aligned with behaviour you try constrain it. > A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. Well no, it can't. None of the programs currently doing anything like that are simple, straightforward or put it bluntly, very good at that. All such systems today are able to operate in very narrow interval of a few controlled variables. And it's not for lack of trying. Your quick paced but narrow minded paper clip factory would have as much chance of wiping off life on Earth as a chess computer on bringing down the electric grid.
    - maaku 11y ago
      That's a bit like criticizing the potential for computers in 1959 because that newly deployed integrated circuit only does very simple logic gates. Cognitive architectures will improve with time.

by maaku 11y ago

varjag 11y ago

It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.

maaku 11y ago
We are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)? Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same. Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option." A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals. Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals. Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it. [0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?
- varjag 11y ago
  Sorry for not being clear enough, you misunderstood me completely. I was arguing about emerging behaviour which might be unavoidably complex and not at all tractable even in a very regular system with simple, relatively well understood components. Take cellular automata, or your very own example, the human brain. As to the free will, it makes total sense as a concept when you consider it versus other intelligent actors, and indeed this is the only useful operational perspective unless you are a theist. E.g. my actions do not depend on your agenda very much, other than we have some synchronized exchange of comments here. Denying it exists is a popular but fruitless philosophic cop-out, akin to using Zeno's paradoxes to challenge general relativity. I however used 'agency' to emphasize functional aspects of the behaviour. The emergent behaviour that might or might be not aligned with behaviour you try constrain it. > A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. Well no, it can't. None of the programs currently doing anything like that are simple, straightforward or put it bluntly, very good at that. All such systems today are able to operate in very narrow interval of a few controlled variables. And it's not for lack of trying. Your quick paced but narrow minded paper clip factory would have as much chance of wiping off life on Earth as a chess computer on bringing down the electric grid.
  - maaku 11y ago
    That's a bit like criticizing the potential for computers in 1959 because that newly deployed integrated circuit only does very simple logic gates. Cognitive architectures will improve with time.