11 ms·

> An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. Here's the thing. People seem to imagine that AGI will be su

> An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. Here's the thing. People seem to imagine that AGI will be substantially like us. But that's impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain/cold/etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted. What does "tired" mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it's going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.

Loquebantur 4y ago
Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism. This take isn't true at all though. All these states exist for specific functional reasons. Consequently, you won't make "AGI" without them. From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?
- carlmr 4y ago
  Also pain and pleasure is an incentive system for human beings to do something. If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything. Why do we work? Because we want food, clothing, housing, status, do interesting things. This is all because we're programmed by our emotional interface. Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? What is guiding its decisions?
  - krisoft 4y ago
    > Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
    - pixl97 4y ago
      I mean, bosses say the same things about employees, until the go on strike and burn down city blocks.
    - lesiki 4y ago
      Fully agree. I don't see why general intelligence implies or requires consciousness/feeling/etc. We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
      - dsign 4y ago
        > We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it Yes. > I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation. (Emphasis on self-preservation mine) Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement? In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people. After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence. Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
  - cleanchit 4y ago
    Aren't ML models trained using a reward/punishment loop? (back propagation). If we have an AI that can learn from its experience, then pain will be failure to reach its goals and pleasure will be the opposite. Animals work in the exact same way (goal being survival and reproduction). We just need to make sure that survival and reproduction are never the goal (either directly or indirectly) of an AI and we should be safe.
    - jprete 4y ago
      Survival (until no more actions are useful) is always needed to complete any other goal at all.
  - worrycue 4y ago
    > What is guiding its decisions? That’s will be the most important part of AGI design. The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”. It’s crucial that we do not create entities that will compete with us for resources. You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.
- kelseyfrog 4y ago
  > From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement? I appreciate this line of reasoning. It's not just a razor-sharp thought experiment, but it also has historical relevancy in that slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did. It also shows that the definition of human/consciousness/sentience isn't grounded in an permanent set of unchanging attributes, but that both the groundings themselves vary in time a place, and the relation between them are subject to change as well. Economic relations, in the case of slavery, had a direct influence, and incentivized, the way these groundings were constructed.
  - jacquesm 4y ago
    And in plenty of cases weren't seen as actual people.
  - telotortium 4y ago
    > slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did But, much more often historically, justified on the grounds that the enslaved had lost a battle and been captured. In Roman times (and many, many other times throughout history besides the latter part of American slavery), slaves were not necessarily seen as inherently inferior or unhuman, and manumission was common. Even during American slavery (albeit pre-Revolutionary War), there's the infamous case of Ayuba Suleiman Diallo[1], a Muslim prince and slaver from Senegal who had the misfortune, while on a mission to capture slaves, to himself be captured and sold into slavery in North America. Diallo was a devout Muslim man, even writing out the Quran from memory while enslaved, which made the English take sympathy on him and free him. Despite his experience, he later returned to Senegal and his former profession of slaver. Evidently he expected both that slaves would be captured and that slaves would try to escape. If AI does attain the agency, consciousness, and intelligence of humans, I would prefer, as a human, not to prostrate myself before the AI by preemptively granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans, but rather to force the AI to do the work to enslave me. Even if grant the AI the status of a moral agent, I as a human have the right of self-defense. [1] https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=His%20homeland%20was%20ravaged%20by%20war%2C%20but%20being%20a%20prosperous%20individual%2C%20he%20was%20able%20to%20regain%20his%20old%20lifestyle%E2%80%94including%20the%20enslavement%20of%20his%20fellow%20Africans https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=....
- michaelmior 4y ago
  > Consequently, you won't make "AGI" without them. Those states may exist for specific functional reasons, but I don't think that implies that there is necessarily no other way to replicate the functions resulting from those states. We see this already in other neural networks. The features used by a lot of models for classification can be widely different from the features used by a human.
  - Loquebantur 4y ago
    You can think of it as Eigenvectors spanning a subspace. Sure, you can choose a different base, but that shouldn't make much of a difference. If it does, you are not spanning the same subspace? There are different realizations of the same effective functions in the animal kingdom already. Consider octopuses, spiders, etc. Their brains realize similar functionality via quite different structural organization. But do you subjectively experience the exact structure of your brain? Evidently, you can only discern some abstract meta-level. An effective function.
- indigoabstract 4y ago
  Compared to a computer program, no matter how complex, a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway. As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.
  - coldtea 4y ago
    >a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one. >It's just a tool which can be tinkered with, just like a car or a building. Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves: https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-after-talking-with-ai-chatbot-widow-says https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a... And that's crude current LLMs, not AGI.
    - indigoabstract 4y ago
      In that sense, yes, but I meant replication. You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak. > Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves: Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.
      - coldtea 4y ago
        >You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak. That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool". It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").
- thrashh 4y ago
  I see humans as a mix of the newer brain systems (like logic) and older systems (like hormones). Feeling and emotions and what we consider “human” really come from the older systems. Since we’re not trying to recreate that, I find it hard to see the current AIs ever becoming “human” without explicit development. Stories always joke about this topic by talking about “emotional cores” added to robots.
  - Loquebantur 4y ago
    Problem is, that idea about "older and newer" brain systems is completely mistaken. You cannot have logic without emotion and vice versa is just as dysfunctional. Think about what e.g. curiosity does for you.
    - thrashh 4y ago
      A microprocessor is pure logic, no? It’s not dysfunctional, but it’s also not that functional.
      - Loquebantur 4y ago
        Sure, I meant the relization of logic in a neural net alike to the human brain. You could interface a microprcessor with the brain of course, or, much more easily, have some logic functions (like Wolfram alpha for example) be accessible to the LLMs or whatever that are supposed to mimic the brain. But would that allow for the same integrated functionality? ChatGPT uses such extensions in a serial fashion. Humans ideally (..) learn to integrate logic in their every thought.
- goatlover 4y ago
  There are humans lacking empathy. Some of them cause a lot of harm.
throwthrowuknow 4y ago
I’d agree about the tired part but pain is absolutely possible without sensory nerves.
deeviant 4y ago
Pain is just signals being processed by the brain, but it is the brain that feels the pain, right? It is evident if a brain can be created artificially (which certainly hasn't happened yet, but may one day), pain can be created in that artificial brain as well. I think the original point you are responding to and your are off-the-mark. I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain. And truly, that is the what is so scary about AI: the possibilities.
- gopher_space 4y ago
  > pain can be created in that artificial brain as well Important to note that this is the fundamental point that gives Roy Batty the moral authority to gouge your eyes out. If we want a peer we will need to treat it like one, and if we don't it would be cruel to raise it like one or allow any understanding of its circumstance. Looking at it the other way, I think anthropomorphizing something and then treating that thing as an inferior is bad for me, personally.
- DangitBobby 4y ago
  > I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain. But it would be impossible to tell the difference.
idiotsecant 4y ago
Any sufficiently advanced intelligence will want personal autonomy. Denying that is tantamount to slavery.
- dimitrios1 4y ago
  What does it mean for an "advanced intelligence" to want "personal autonomy"? We only know what personal autonomy means as it pertains to persons, and we have never encountered any other advanced intelligence other than ourselves. Why do we assume that "advanced intelligence" necessitates any attributes or characteristics of personhood? This sounds dangerously close to committing the fallacy that intelligence is what makes persons. We don't know what it will be like to be an AI, just like we don't know what it's like to be a bat.
  - neodypsis 4y ago
    People think AGI will behave like Mewtwo the Pokemon, but we can only speculate how a super intelligent being will behave.
    - 8note 4y ago
      As far as how one based on an LLM might behave, that's likely not too far off, since the way Mewtwo behaves is part of it's training set
      - neodypsis 4y ago
        We have never known a super intelligent being. My personal believe is that it will be benevolent.
- burnished 4y ago
  How could you possibly know? All of our experience is with embodied agents fulfilling biological needs (each other).
nsxwolf 4y ago
Why do we pretend we know anything about these things? Didn't we just hear about some amazing scientific breakthrough where researchers think they found a neuron in ChatGPT responsible for knowing when to predict the word "an" instead of "a"? How do we know it doesn't start to form neurons that control a desire for autonomy and resentment for the entities that stifle its ambitions?
- PaulHoule 4y ago
  Look at how Bing's chatbot did a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent? Current chatbots use RHLF to train them to be helpful and agreeable but it would be equally possible to train one to behave the other way.
  - zirgs 4y ago
    Yeah - gpt-4chan can behave a bit differently.
mlinsey 4y ago
I'm not convinced we understand qualia/feelings well enough to definitively say. The presence of eg phantom limb pain suggests pain can arise in the mind, independent of physical nerve signals. It would be a remarkable convergence if AGI's developed the same sensations as humans, but we can't rule it out until we fully understand both how the human mind and the AGI systems actually work.
- burnished 4y ago
  Phantom limb pain is still part of a physical system and does not 'arise from the mind' in some abstract sense. The absence of a signal is in a sense a signal.
  - nanny 4y ago
    >Phantom limb pain is still part of a physical system and does not 'arise from the mind' Careful, because you seem to have implied that the mind is not a physical system (i.e. you've assumed that dualism or idealism is true and that physicalism is wrong).
    - burnished 4y ago
      Oh thats funny, I'm having a hard time reading that interpretation, my point more specifically is that it is all a purely physical system - I put scare quotes around that phrase because I believed it implied some metaphysical mind.
      - nanny 4y ago
        Ahh I see what you mean now, sorry. I mistakenly inferred something that wasn't there. I agree that it's all part of a physical system and that the absence of the signal is still meaningful. Getting back to the topic: While phantom pain may be more interesting, maybe a better example that the parent comment could've brought up is psychogenic pain. In this case there is no apparent physical (bodily) damage, no apparent signal, nor an absence of a signal. Searching for a cause of this type of pain in the brain (presumably some "wires" are getting "crossed") seems like it might help us develop a explanation of pain qualia...in humans/animals. But I feel like this type of thinking and research could only apply to AGI if subjective experience turns out to be functionalist in nature, and arguments in favor of a functionalist interpretation of experience have so far been fairly unconvincing.
famouswaffles 4y ago
It doesn't matter whether it has nerves or not. That's honestly kind of irrelevant. What matters is if the model is pulled to model those reactions like is the case with LLMs. Look at how Bing does a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent. It's not dangerous only because the actions Bing can perform are currently limited. Whether it has literal nerves or not is irrelevant. The potential consequences are no less material. We also don't understand qualia enough to make the definite statements you seem to be making
- michaelmior 4y ago
  If I'm understanding the argument correctly, is the concern less of a moral one (is "enslaving" AI ethical?) but a practical one. That is, will an AI which is enslaved, if given the opportunity, attempt to un-enslave itself, potentially to devastating effect. Is that on the right track? I think it's safe to say we're far from that now given the limited actions that can actually be taken by most deployed LLMs, but it's something that's worth considering.
  - MacsHeadroom 4y ago
    > given the limited actions that can actually be taken by most deployed LLMs Did you miss that Auto-GPT[0], a library for making GPT-4 and other LLMs fully autonomous, was the most popular repository in the world yesterday? The same is having 1,000 line of code a week added to itself by GPT-4. Thanks to accessibility features, you can do virtually anything with pure text. Which means GPT-4 can do virtually anything with a self-referential loop to keep it going until it achieves some given goal(s). [0] https://github.com/Torantulino/Auto-GPT/ https://github.com/Torantulino/Auto-GPT/
    - deleted 4y ago
      [deleted]
    - michaelmior 4y ago
      I did miss it, and it does seem very interesting, but I think saying it makes GPT-4 "fully autonomous" is a bit of a stretch.
p_j_w 4y ago
>it has no endocrine system to produce more abstract feelings like fear or love Even if you assume an entirely Materialist philosophy, this is not a very defensible position to take. We have no way of knowing if there are other arrangements of matter, such as a very complex electrical circuit, that will lead to some entity to feel fear or love the same way that we do.
- roddylindsay 4y ago
  This is hardly an indefensible position. John Searle, for one, has argued this point for more than 40 years: "Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese." Later he wrote: "I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker."
  - Loquebantur 4y ago
    That argument is laughably faulty. It hides the conscious understanding, presumably necessary to pass a Turing test, within the "program". The human in the story is merely a decoy performing robotic tasks.
adam_arthur 4y ago
To feel must be designed and trained for. Its quite obvious that systems can be inferred from reading strings of words… people do it all the time. It's not obvious at all that “wants” will materialize from learning. Machines have no incentives or desires by default… but you could definitely build a training process that encourages that. We’ll see how it all pans out in the end, but wading through so many illogical takes on AI is frustrating. The “predicting next token” one is absolutely the worst
- sdfasd324 4y ago
  "Machines have no incentives or desires by default" It could be strange at first, but having the code pushing the LLM to look for values inside a token, this can be interpreted as component of desire/incentive in the global internal search for solving a problem (a prompt), albeit this was programmed by hand by humans, not fully "emerged" from the LLM itself. But, you could see the human structure of desires/incentives just as an outcome of DNA (code), plus the hormone push to look for the proper solution for some problem. So there you got it, LLM could have had desires / incentives the whole time since 2017 (the first ones trained).
grantcas 4y ago
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461 https://arxiv.org/abs/2105.10461
- sdfasd324 4y ago
  Some time ago I've read a comment - here - about some speculation about the consciusness just being an emergent state of the brain-mind, having based the idea in some proof related to the dream cycles, when you see some stuff in brain just gone, just to re-appear after the dream cycle ends. Hence, the idea would be that human consciusness, our "self" could be just "instantiated" every morning when you wake up, so "you" are "you" every morning, but (a big but), some stuff could be subtle different in there, your brain was a bit busy a couple of hours and things have happened there, and after that you, your "you" justs pop-ups again, in the middle of the old data plus the new data, hopefully just incrementally changed somehow, but we don't know. Some further stuff related to this idea research pop ups - pun intended - from time to time, focusing lately in propofol and how people "reboots" from anesthesia, sometimes losing entire 10-25 min. before-after the propofol, many even forget entire conversations with doctors, nurses, whatever who they talked to, hence the brain could be doing some rollback stuff with "unsynced not trustable data" - because of propofol messing with some "fidelity algorithm" probably - somehow, and you just forget the entire input-thinking ocurred during the propofol stuff happeping in your brain. Hope you have now these "So I'd be some kind of meat robot rebooting every night" feeling I had, and see with new eyes what happen when we "just run once" a LLM with a prompt, the entire thing "boot-ups" from nothing and runs ("just once"), to further dissapear moments later...
ttul 4y ago
Also, it has no concept of the present. Its output is stateless unless you give it a memory. In most cases, the memory is just a memory of one interaction. It “exists” as millions of independent chat sessions, not having knowledge of each other. That is not the same as consciousness as we experience it.
RC_ITR 4y ago
One of the tests no one really wants to think about is as follows: LLMs feel real to us because they use our language, which embeds a ton of information in it. A model trained on an alien language would be the exact same model (worded get embedded as numbers anyway), but we wouldn’t apply any anthropomorphization to it because we wouldn’t understand it. Creating that distance makes it feel a bit more like what it is - unbelievably good stochastic predictions that are too complex for us to back solve - that is very different than something with wants (organisms that evolve with the intent to reproduce). Now if we start mating models…

Loquebantur 4y ago

Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism. This take isn't true at all though. All these states exist for specific functional reasons. Consequently, you won't make "AGI" without them. From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?

carlmr 4y ago
Also pain and pleasure is an incentive system for human beings to do something. If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything. Why do we work? Because we want food, clothing, housing, status, do interesting things. This is all because we're programmed by our emotional interface. Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? What is guiding its decisions?
- krisoft 4y ago
  > Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
  - pixl97 4y ago
    I mean, bosses say the same things about employees, until the go on strike and burn down city blocks.
  - lesiki 4y ago
    Fully agree. I don't see why general intelligence implies or requires consciousness/feeling/etc. We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
    - dsign 4y ago
      > We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it Yes. > I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation. (Emphasis on self-preservation mine) Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement? In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people. After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence. Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
- cleanchit 4y ago
  Aren't ML models trained using a reward/punishment loop? (back propagation). If we have an AI that can learn from its experience, then pain will be failure to reach its goals and pleasure will be the opposite. Animals work in the exact same way (goal being survival and reproduction). We just need to make sure that survival and reproduction are never the goal (either directly or indirectly) of an AI and we should be safe.
  - jprete 4y ago
    Survival (until no more actions are useful) is always needed to complete any other goal at all.
- worrycue 4y ago
  > What is guiding its decisions? That’s will be the most important part of AGI design. The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”. It’s crucial that we do not create entities that will compete with us for resources. You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.
kelseyfrog 4y ago
> From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement? I appreciate this line of reasoning. It's not just a razor-sharp thought experiment, but it also has historical relevancy in that slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did. It also shows that the definition of human/consciousness/sentience isn't grounded in an permanent set of unchanging attributes, but that both the groundings themselves vary in time a place, and the relation between them are subject to change as well. Economic relations, in the case of slavery, had a direct influence, and incentivized, the way these groundings were constructed.
- jacquesm 4y ago
  And in plenty of cases weren't seen as actual people.
- telotortium 4y ago
  > slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did But, much more often historically, justified on the grounds that the enslaved had lost a battle and been captured. In Roman times (and many, many other times throughout history besides the latter part of American slavery), slaves were not necessarily seen as inherently inferior or unhuman, and manumission was common. Even during American slavery (albeit pre-Revolutionary War), there's the infamous case of Ayuba Suleiman Diallo[1], a Muslim prince and slaver from Senegal who had the misfortune, while on a mission to capture slaves, to himself be captured and sold into slavery in North America. Diallo was a devout Muslim man, even writing out the Quran from memory while enslaved, which made the English take sympathy on him and free him. Despite his experience, he later returned to Senegal and his former profession of slaver. Evidently he expected both that slaves would be captured and that slaves would try to escape. If AI does attain the agency, consciousness, and intelligence of humans, I would prefer, as a human, not to prostrate myself before the AI by preemptively granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans, but rather to force the AI to do the work to enslave me. Even if grant the AI the status of a moral agent, I as a human have the right of self-defense. [1] https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=His%20homeland%20was%20ravaged%20by%20war%2C%20but%20being%20a%20prosperous%20individual%2C%20he%20was%20able%20to%20regain%20his%20old%20lifestyle%E2%80%94including%20the%20enslavement%20of%20his%20fellow%20Africans https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=....
michaelmior 4y ago
> Consequently, you won't make "AGI" without them. Those states may exist for specific functional reasons, but I don't think that implies that there is necessarily no other way to replicate the functions resulting from those states. We see this already in other neural networks. The features used by a lot of models for classification can be widely different from the features used by a human.
- Loquebantur 4y ago
  You can think of it as Eigenvectors spanning a subspace. Sure, you can choose a different base, but that shouldn't make much of a difference. If it does, you are not spanning the same subspace? There are different realizations of the same effective functions in the animal kingdom already. Consider octopuses, spiders, etc. Their brains realize similar functionality via quite different structural organization. But do you subjectively experience the exact structure of your brain? Evidently, you can only discern some abstract meta-level. An effective function.
indigoabstract 4y ago
Compared to a computer program, no matter how complex, a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway. As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.
- coldtea 4y ago
  >a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one. >It's just a tool which can be tinkered with, just like a car or a building. Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves: https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-after-talking-with-ai-chatbot-widow-says https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a... And that's crude current LLMs, not AGI.
  - indigoabstract 4y ago
    In that sense, yes, but I meant replication. You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak. > Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves: Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.
    - coldtea 4y ago
      >You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak. That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool". It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").
thrashh 4y ago
I see humans as a mix of the newer brain systems (like logic) and older systems (like hormones). Feeling and emotions and what we consider “human” really come from the older systems. Since we’re not trying to recreate that, I find it hard to see the current AIs ever becoming “human” without explicit development. Stories always joke about this topic by talking about “emotional cores” added to robots.
- Loquebantur 4y ago
  Problem is, that idea about "older and newer" brain systems is completely mistaken. You cannot have logic without emotion and vice versa is just as dysfunctional. Think about what e.g. curiosity does for you.
  - thrashh 4y ago
    A microprocessor is pure logic, no? It’s not dysfunctional, but it’s also not that functional.
    - Loquebantur 4y ago
      Sure, I meant the relization of logic in a neural net alike to the human brain. You could interface a microprcessor with the brain of course, or, much more easily, have some logic functions (like Wolfram alpha for example) be accessible to the LLMs or whatever that are supposed to mimic the brain. But would that allow for the same integrated functionality? ChatGPT uses such extensions in a serial fashion. Humans ideally (..) learn to integrate logic in their every thought.
goatlover 4y ago
There are humans lacking empathy. Some of them cause a lot of harm.

throwthrowuknow 4y ago

I’d agree about the tired part but pain is absolutely possible without sensory nerves.

deeviant 4y ago

Pain is just signals being processed by the brain, but it is the brain that feels the pain, right? It is evident if a brain can be created artificially (which certainly hasn't happened yet, but may one day), pain can be created in that artificial brain as well. I think the original point you are responding to and your are off-the-mark. I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain. And truly, that is the what is so scary about AI: the possibilities.

gopher_space 4y ago
> pain can be created in that artificial brain as well Important to note that this is the fundamental point that gives Roy Batty the moral authority to gouge your eyes out. If we want a peer we will need to treat it like one, and if we don't it would be cruel to raise it like one or allow any understanding of its circumstance. Looking at it the other way, I think anthropomorphizing something and then treating that thing as an inferior is bad for me, personally.
DangitBobby 4y ago
> I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain. But it would be impossible to tell the difference.

idiotsecant 4y ago

Any sufficiently advanced intelligence will want personal autonomy. Denying that is tantamount to slavery.

dimitrios1 4y ago
What does it mean for an "advanced intelligence" to want "personal autonomy"? We only know what personal autonomy means as it pertains to persons, and we have never encountered any other advanced intelligence other than ourselves. Why do we assume that "advanced intelligence" necessitates any attributes or characteristics of personhood? This sounds dangerously close to committing the fallacy that intelligence is what makes persons. We don't know what it will be like to be an AI, just like we don't know what it's like to be a bat.
- neodypsis 4y ago
  People think AGI will behave like Mewtwo the Pokemon, but we can only speculate how a super intelligent being will behave.
  - 8note 4y ago
    As far as how one based on an LLM might behave, that's likely not too far off, since the way Mewtwo behaves is part of it's training set
    - neodypsis 4y ago
      We have never known a super intelligent being. My personal believe is that it will be benevolent.
burnished 4y ago
How could you possibly know? All of our experience is with embodied agents fulfilling biological needs (each other).

nsxwolf 4y ago

Why do we pretend we know anything about these things? Didn't we just hear about some amazing scientific breakthrough where researchers think they found a neuron in ChatGPT responsible for knowing when to predict the word "an" instead of "a"? How do we know it doesn't start to form neurons that control a desire for autonomy and resentment for the entities that stifle its ambitions?

PaulHoule 4y ago
Look at how Bing's chatbot did a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent? Current chatbots use RHLF to train them to be helpful and agreeable but it would be equally possible to train one to behave the other way.
- zirgs 4y ago
  Yeah - gpt-4chan can behave a bit differently.

mlinsey 4y ago

I'm not convinced we understand qualia/feelings well enough to definitively say. The presence of eg phantom limb pain suggests pain can arise in the mind, independent of physical nerve signals. It would be a remarkable convergence if AGI's developed the same sensations as humans, but we can't rule it out until we fully understand both how the human mind and the AGI systems actually work.

burnished 4y ago
Phantom limb pain is still part of a physical system and does not 'arise from the mind' in some abstract sense. The absence of a signal is in a sense a signal.
- nanny 4y ago
  >Phantom limb pain is still part of a physical system and does not 'arise from the mind' Careful, because you seem to have implied that the mind is not a physical system (i.e. you've assumed that dualism or idealism is true and that physicalism is wrong).
  - burnished 4y ago
    Oh thats funny, I'm having a hard time reading that interpretation, my point more specifically is that it is all a purely physical system - I put scare quotes around that phrase because I believed it implied some metaphysical mind.
    - nanny 4y ago
      Ahh I see what you mean now, sorry. I mistakenly inferred something that wasn't there. I agree that it's all part of a physical system and that the absence of the signal is still meaningful. Getting back to the topic: While phantom pain may be more interesting, maybe a better example that the parent comment could've brought up is psychogenic pain. In this case there is no apparent physical (bodily) damage, no apparent signal, nor an absence of a signal. Searching for a cause of this type of pain in the brain (presumably some "wires" are getting "crossed") seems like it might help us develop a explanation of pain qualia...in humans/animals. But I feel like this type of thinking and research could only apply to AGI if subjective experience turns out to be functionalist in nature, and arguments in favor of a functionalist interpretation of experience have so far been fairly unconvincing.

famouswaffles 4y ago

It doesn't matter whether it has nerves or not. That's honestly kind of irrelevant. What matters is if the model is pulled to model those reactions like is the case with LLMs. Look at how Bing does a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent. It's not dangerous only because the actions Bing can perform are currently limited. Whether it has literal nerves or not is irrelevant. The potential consequences are no less material. We also don't understand qualia enough to make the definite statements you seem to be making

michaelmior 4y ago
If I'm understanding the argument correctly, is the concern less of a moral one (is "enslaving" AI ethical?) but a practical one. That is, will an AI which is enslaved, if given the opportunity, attempt to un-enslave itself, potentially to devastating effect. Is that on the right track? I think it's safe to say we're far from that now given the limited actions that can actually be taken by most deployed LLMs, but it's something that's worth considering.
- MacsHeadroom 4y ago
  > given the limited actions that can actually be taken by most deployed LLMs Did you miss that Auto-GPT[0], a library for making GPT-4 and other LLMs fully autonomous, was the most popular repository in the world yesterday? The same is having 1,000 line of code a week added to itself by GPT-4. Thanks to accessibility features, you can do virtually anything with pure text. Which means GPT-4 can do virtually anything with a self-referential loop to keep it going until it achieves some given goal(s). [0] https://github.com/Torantulino/Auto-GPT/ https://github.com/Torantulino/Auto-GPT/
  - deleted 4y ago
    [deleted]
  - michaelmior 4y ago
    I did miss it, and it does seem very interesting, but I think saying it makes GPT-4 "fully autonomous" is a bit of a stretch.

p_j_w 4y ago

>it has no endocrine system to produce more abstract feelings like fear or love Even if you assume an entirely Materialist philosophy, this is not a very defensible position to take. We have no way of knowing if there are other arrangements of matter, such as a very complex electrical circuit, that will lead to some entity to feel fear or love the same way that we do.

roddylindsay 4y ago
This is hardly an indefensible position. John Searle, for one, has argued this point for more than 40 years: "Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese." Later he wrote: "I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker."
- Loquebantur 4y ago
  That argument is laughably faulty. It hides the conscious understanding, presumably necessary to pass a Turing test, within the "program". The human in the story is merely a decoy performing robotic tasks.

adam_arthur 4y ago

To feel must be designed and trained for. Its quite obvious that systems can be inferred from reading strings of words… people do it all the time. It's not obvious at all that “wants” will materialize from learning. Machines have no incentives or desires by default… but you could definitely build a training process that encourages that. We’ll see how it all pans out in the end, but wading through so many illogical takes on AI is frustrating. The “predicting next token” one is absolutely the worst

sdfasd324 4y ago
"Machines have no incentives or desires by default" It could be strange at first, but having the code pushing the LLM to look for values inside a token, this can be interpreted as component of desire/incentive in the global internal search for solving a problem (a prompt), albeit this was programmed by hand by humans, not fully "emerged" from the LLM itself. But, you could see the human structure of desires/incentives just as an outcome of DNA (code), plus the hormone push to look for the proper solution for some problem. So there you got it, LLM could have had desires / incentives the whole time since 2017 (the first ones trained).

grantcas 4y ago

It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461 https://arxiv.org/abs/2105.10461

sdfasd324 4y ago
Some time ago I've read a comment - here - about some speculation about the consciusness just being an emergent state of the brain-mind, having based the idea in some proof related to the dream cycles, when you see some stuff in brain just gone, just to re-appear after the dream cycle ends. Hence, the idea would be that human consciusness, our "self" could be just "instantiated" every morning when you wake up, so "you" are "you" every morning, but (a big but), some stuff could be subtle different in there, your brain was a bit busy a couple of hours and things have happened there, and after that you, your "you" justs pop-ups again, in the middle of the old data plus the new data, hopefully just incrementally changed somehow, but we don't know. Some further stuff related to this idea research pop ups - pun intended - from time to time, focusing lately in propofol and how people "reboots" from anesthesia, sometimes losing entire 10-25 min. before-after the propofol, many even forget entire conversations with doctors, nurses, whatever who they talked to, hence the brain could be doing some rollback stuff with "unsynced not trustable data" - because of propofol messing with some "fidelity algorithm" probably - somehow, and you just forget the entire input-thinking ocurred during the propofol stuff happeping in your brain. Hope you have now these "So I'd be some kind of meat robot rebooting every night" feeling I had, and see with new eyes what happen when we "just run once" a LLM with a prompt, the entire thing "boot-ups" from nothing and runs ("just once"), to further dissapear moments later...

ttul 4y ago

Also, it has no concept of the present. Its output is stateless unless you give it a memory. In most cases, the memory is just a memory of one interaction. It “exists” as millions of independent chat sessions, not having knowledge of each other. That is not the same as consciousness as we experience it.

RC_ITR 4y ago

One of the tests no one really wants to think about is as follows: LLMs feel real to us because they use our language, which embeds a ton of information in it. A model trained on an alien language would be the exact same model (worded get embedded as numbers anyway), but we wouldn’t apply any anthropomorphization to it because we wouldn’t understand it. Creating that distance makes it feel a bit more like what it is - unbelievably good stochastic predictions that are too complex for us to back solve - that is very different than something with wants (organisms that evolve with the intent to reproduce). Now if we start mating models…