4 ms·

I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense i

by maaku 11y ago

I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.

varjag 11y ago
And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
- maaku 11y ago
  > And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
  - varjag 11y ago
    It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.
    - maaku 11y ago
      We are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)? Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same. Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option." A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals. Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals. Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it. [0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?

by maaku 11y ago

varjag 11y ago

And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.

maaku 11y ago
> And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
- varjag 11y ago
  It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.
  - maaku 11y ago
    We are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)? Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same. Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option." A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals. Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals. Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it. [0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?