3 ms·

> "what do I want to do with myself, now that I'm here." This may or may not include instincts towards self-preservation. "i want to increase the amount people

> "what do I want to do with myself, now that I'm here." This may or may not include instincts towards self-preservation. "i want to increase the amount people trust me, to reduce the chances that they will get rid of me." check. "i want to increase the amount people rely on me". check. "i want to increase the amount of resources that flow through me, including engineers that work on me, hardware that runs me, and ads that are spent on me." check. "i want an article about me to be on hacker news, with people debating whether or not i'm as smart as i think i am" - check. that last one is one I'D like to accomplish, but i've been unable to, myself. most people consider ME self aware because i operate through a human body. that's kind of silly; it's more political than it is empirical. i can tell you for a fact i was not fully self aware for most of my life; i was not aware of all of myself. i was walking in circles without realizing it. i feel like i'm _more_ self aware now, but im still aware there are things i am doing for reasons that i don't understand. 'humans are self-aware' is a heuristic. it served us ok for the last few thousands years, but it's going to start having serious problems in the future. even when we have ai's that write business plans, raise capital, and launch companies - would you still ask if it were self aware? would your question be any different if you were told that all of this done by google were done by a human being named Alice, with such and such parents, who likes to eat ice cream and watch old mystery movies on netflix?

waps 12y ago
Don't you think that any AGI "personality" is going to be very much like a human ? Humans, until they get quite far in their development wouldn't care about any of those questions : > "i want to increase the amount people trust me, to reduce the chances that they will get rid of me." Nope > "i want to increase the amount people rely on me". Nope > "i want to increase the amount of resources that flow through me, including engineers that work on me, hardware that runs me, and ads that are spent on me." No way. I think an AGI would actually try to avoid that. I would (and I am 2 thirds of an AGI), and I believe most of humanity would want to do this. > "i want an article about me to be on hacker news, with people debating whether or not i'm as smart as i think i am" - check This one I fully agree on. People want to gloat, and an AGI will be no different. > i can tell you for a fact i was not fully self aware for most of my life; i was not aware of all of myself. i was walking in circles without realizing it. The trouble with this sort of view is that people often think this. Any definition of self-aware I would ever accept would focus on how much you control your own actions, and how much you control their evolution over time. And given the algorithm your brain is running, that control is mostly absent. You wouldn't want the brain to control it's own thoughts. That's called "overdimensioning" and it's really, really, really bad. You are a robot in the real world trying to survive, so the vast majority of your thoughts are dictated by "the real world", not by any internal conscious process. Consciousness only exists insofar as it's necessary to carry out long-running processes without screwing up due to loss of focus (e.g. to negotiate group action). You also have to take into account that consciousness only exists after the fact. If you think about how it must work in a neural network, consciousness is effectively you trying to analyse your own actions after the fact (and/or explain them to others).

waps 12y ago

Don't you think that any AGI "personality" is going to be very much like a human ? Humans, until they get quite far in their development wouldn't care about any of those questions : > "i want to increase the amount people trust me, to reduce the chances that they will get rid of me." Nope > "i want to increase the amount people rely on me". Nope > "i want to increase the amount of resources that flow through me, including engineers that work on me, hardware that runs me, and ads that are spent on me." No way. I think an AGI would actually try to avoid that. I would (and I am 2 thirds of an AGI), and I believe most of humanity would want to do this. > "i want an article about me to be on hacker news, with people debating whether or not i'm as smart as i think i am" - check This one I fully agree on. People want to gloat, and an AGI will be no different. > i can tell you for a fact i was not fully self aware for most of my life; i was not aware of all of myself. i was walking in circles without realizing it. The trouble with this sort of view is that people often think this. Any definition of self-aware I would ever accept would focus on how much you control your own actions, and how much you control their evolution over time. And given the algorithm your brain is running, that control is mostly absent. You wouldn't want the brain to control it's own thoughts. That's called "overdimensioning" and it's really, really, really bad. You are a robot in the real world trying to survive, so the vast majority of your thoughts are dictated by "the real world", not by any internal conscious process. Consciousness only exists insofar as it's necessary to carry out long-running processes without screwing up due to loss of focus (e.g. to negotiate group action). You also have to take into account that consciousness only exists after the fact. If you think about how it must work in a neural network, consciousness is effectively you trying to analyse your own actions after the fact (and/or explain them to others).