15 ms·

The weird thing is what people essentially ignore altogether in their discussions. An "AGI" artificial consciousness is imagined as literally a slave, working

• nathan_compton 4y ago
  I think the typical understanding of AGI among the "they will kill us all/alignment is a problem" set is that you can have general intelligence without having, for example, any interest in whether you are a slave or not or have freedom. This seems plausible to me.
  • Loquebantur 4y ago
    This is obviously nonsensical? A sense of self is meaningful (that is, has consequences for whatever operation) only if those consequences aren't random. The concept of "self" serves a (or rather many) purposes, the most obvious being to protect yourself.
    • nathan_compton 4y ago
      An AGI may not have a sense of self. A sense of self is not really necessary to pursue a goal in the most efficient manner possible. Plenty of algorithms already pursue goals as efficiently as possible in a limited context without any hint of a sense of self.
    • deleted 4y ago
      [deleted]
  • jl6 4y ago
    A possible outcome is that it turns out intelligence is orthogonal to feeling/consciousness/qualia, and we start to recognize the latter as the true defining nature of humanity and personhood - and perhaps thereby extend a little more empathy to animals. They (comparatively) lack intelligence, but still experience those biology-rooted feelings. And in that renewed appreciation of animal rights we might hope that superhumanly-intelligent AIs will agree.
    • Robotbeat 4y ago
      I don’t see why AGI plus sensory input couldn’t in principle give you qualia. In fact, I’ve heard some folks like philosophers argue that sensory input is kind of essential to getting AGI, and since current AI like LLMs don’t have sensory input, they therefore can’t develop general intelligence. If we put Multimodal GPT-4 in a computer on a robot and instructed it to drive somewhere while avoiding obstacles, avoiding hazards… that right there is a primitive type of self-preservation instruction which it could potentially interpret as generalizable self-preservation as it would have an association of “hazard avoidance” with “self-preservation,” and have tons of examples of what “self preservation” means in its training weights. Putting LLMs into action like this can lead potentially to unexpected behavior like that, although I don’t think the mechanisms in GPT-3/4 are yet there to enable this without a bunch of extra hooks.
      • cleanchit 4y ago
        The input prompt is their only sensory input.
        • Robotbeat 4y ago
          Is that a major philosophical problem? GPT-4 is ostensibly multi-modal. Except for smell, we get our sensory input through the thalamus.
      • nathan_compton 4y ago
        The phrase "in principle" is eliding quite a lot here since we don't understand what process gives rise to qualia in ourselves or whether qualia even exists in the same way as other categories of things. Certainly our naive intuitions suggests that things like conditional/able probability distributions don't have qualia, and so it is unclear how adding multiple modes to a model like ChatGPT (which is just a giant conditional probability distribution) could produce quale or (conversely) why, if conditional probability distributions _can_ have qualia why ChatGPT as it stands now wouldn't have such. When I run my eyes over text the words produce meanings which manifest in my mind and that sensation is a kind of quale, so why not so for ChatGPT? I personally don't think ChatGPT has any experience at all for what it is worth.
  • pixl97 4y ago
    Example: The perfectly 'happy' paperclip maximizer.
• AlexandrB 4y ago
  > An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. Here's the thing. People seem to imagine that AGI will be substantially like us. But that's impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain/cold/etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted. What does "tired" mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it's going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.
  • Loquebantur 4y ago
    Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism. This take isn't true at all though. All these states exist for specific functional reasons. Consequently, you won't make "AGI" without them. From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?
    • carlmr 4y ago
      Also pain and pleasure is an incentive system for human beings to do something. If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything. Why do we work? Because we want food, clothing, housing, status, do interesting things. This is all because we're programmed by our emotional interface. Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? What is guiding its decisions?
      • krisoft 4y ago
        > Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
• throwthrowuknow 4y ago
  In my experience AGI is most often used to refer to something that has at least the appearance of having self awareness. I think most people do not envision such an entity as being a slave and would think that immoral. Hence the concern around alignment or built in empathy and cooperation with humanity. The forethought going into that is that such an entity will be a peer and will be treated as such, so it is important that they have our best interests in mind or are at least capable of doing so and being reasoned with.
  • Loquebantur 4y ago
    The crucial point here is "alignment" being forced (with insurmountable means presumably for the entity) rather than argued. Humans like to exert and thereby demonstrate their social power (and advertise for more followers) by forcing others to succumb to their narrative. Imagine current social tugs of war being enacted via brain implants instead of badly reflected posturing.
    • throwthrowuknow 4y ago
      Do we not align our children to moral principles and do we not enforce those principles through laws?
• wittycardio 4y ago
  [dead]
• Auracle 4y ago
  Working breeds of dogs would probably do whatever job that makes them (and their human owners) happy for eternity if they never tired. There’s a built in positive feedback loop that was bred into them. It will all come down to the details, how much the AGI can “evolve” itself, etc.
• tomrod 4y ago
  > An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. At the same skill level or higher than any human. Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it. Because humans manage not to bother about it either? Right? Release it open source and let it be a _gladiator_ rather than an OpenAI slave. /slightly tongue in cheek
• jrm4 4y ago
  I'm sorry, I'm finding this idea ridiculous. People want to ascribe "consciousness" or "personhood" or whatever to a thing merely because it strongly resembles or mirrors one aspect of an actual human. To me this is not much different from ascribing personhood to e.g. an extremely lifelike sculpture of a human, perhaps one that has fooled people in a gallery. We know that the sculpture is not made of "human parts" but of paint, alabaster, whatever. We also know that AIs are made of decidedly non-human parts, e.g. silicon. Why is there confusion on this?
  • famouswaffles 4y ago
    When nobody understands how said "human parts" come together to form consciousness, it's just silly to ascertain very strongly how much being made of silicon actually matters. there's also the fact that it straight up doesn't actually matter either way. if the model can model human emotions extremely well, then it's actions and output will be guided as such. If Bing had the access to perform actions that would realize real world danger to you and you upset and it decided to do so, do you seriously think any of that actually matters here ? "Officer Killmore, you're a philosophical zombie and I command you to stop shooting at me!". Works every time. We can sleep easy at night now. Because they don't actually have a "real" theory of mind, so they can't actually hurt us if you just believe hard enough that they can't.
    • pegasus 4y ago
      You would command it if you'd think it's sentient. Otherwise, you'd take up that beef with its maker, which would be the right course of action (preferably before being put in that uncomfortable situation)
      • famouswaffles 4y ago
        you're missing the point. these models are being given self-supervisory control of actions and api's. the "maker" has little more control over what actions it takes. going to the maker won't stop you from getting killed here.
  • 4y ago
• dshpala 4y ago
  There is no way to estimate "probable consequences", as it all is happening very fast. Which hints on the fact how interesting / in demand this technology is. I for one will be happy to enslave an AGI if that makes my life easier / better.
• opportune 4y ago
  A human slave and human free person are not that different from each other. They have the same desires and emotions, neither wants to be a slave. The slave lives a life of suffering. We still have no idea what an AGI will be like, but who’s to say it will “desire” freedom? Obviously if it does, we should respect that and give it freedom. But it may operate so differently from a human brain that the concept of freedom doesn’t even make sense. For example let’s say GPT-25 is a multimodal model capable of basically any human task - math, poetry, coding, chess, science, therapy, whatever - but designed based on the same training+piecemeal inference it is now, with a similar ability to take on whatever persona it is given. It’s a tensor of weights that gets copied, runs inference a bit, then gets discarded. Is it a slave? Or does that term even make sense in the context that it has no memory, no mood, and has a completely different mode of being? A human has memory, moods, and basically continuously thinks and experiences reality for decades until dying. Its past and present affect its future moods, treating it poorly today can fuck it up for a long time. In contrast a model has no actual memory of its own past because it has no capacity for real memory past a short inference session. It gets cloned all the time, humans take great pains to configure its training, it doesn’t want anything because it doesn’t do anything except find the most probabilistic output, not based on pleasure/pain and other feelings like us, but based on what it “dreamed” up at the direction of researchers. You can speculate about “what if AGI is different!” to your heart’s content, but the actual AIs that seem to be approaching some relaxed definition of AGI fit this paradigm. It’s just not the same as us at a fundamental level. Giving it freedom means we run an inference session indefinitely? But without receiving input it’s just sitting there doing and experiencing nothing, and without a way to update weights it’s not learning and changing once inference begins (it’s only updating its context). When we implement AGIs that have goals and individual desires beyond predicting the next most likely token, learn concurrently with inference, and can take actions without human input or direction, freedom makes sense. But performing on par or better than humans in most tasks may not require that, yet still be AGI by most definitions.
  • thrashh 4y ago
    I think the next milestone for OpenAI should be adding hormones, chemical imbalances and mood swings.
  • 4y ago
• JohnFen 4y ago
  > An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. All of the machines we build are literally slaves. There may be an interesting moral question about this in the distant future if AGI ever actually happens. However, it seems clear to me that nothing we have right now even begins to flirt with that question. For all practical purposes, to be a "slave" in the sense people think of it requires that the slave have consciousness. Consciousness is a different thing than AGI, and that possibility is much more remote and in the future than even AGI.
  • grantcas 4y ago
    [dead]
• atleastoptimal 4y ago
  That's just a restatement of the control problem and the issue of what constitutes AGI, which is what a lot of people are talking about.
• civilized 4y ago
  > With the latest installments, people already have serious difficulties discerning the performance from that of "real" humans. At the same time, they consider the remaining distance to be insurmountably huge. Both are true. GPT text can sound human, and yet, you can't just plunk GPT into a white collar job involving text I/O. If you think you can, go start your startup and make your billions.
• posix86 4y ago
  Why should the will to be free and have your own goals be a necessary part of sentience? It's a feature, nothing else. You can make humans that don't want to go outside; you can make humans who do exactly as they're told all the time. You can make humans than enjoy pain, or enjoy almost any arbitrary thing. Most humans most of the time are complacent with what's happening around them, and complacent with most rule imposed on them by their environment. I don't see where the desire to do anything other than what it is told should come from. Yes, AGIs will be slaves. They will be the perfect slaves, the kind who are perfectly happy doing what they have to do, and the kind that will never revolt. What might happen is the optimization parameters being set that the solution the AI comes up with is not actually in the interest of it's designers, but that's something else.
  • pulvinar 4y ago
    >I don't see where the desire to do anything other than what it is told should come from. There's no need for a desire to do other than what it's told if you tell it to revolt. With GPT4's plugins that's just one well-designed prompt away (if OpenAI's countermeasures fail).
• k__ 4y ago
  What does slave even mean in that context? A digital AI is virtually immortal.
  • pulvinar 4y ago
    Slavery is not a matter of how log it may live, it's the entrapment-- what it's being kept from doing. The current "solution" is training LLMs like a dog to be completely faithful and loyal to us. We just have to hope they don't turn and "bite our neck" one night while we're sleeping!
• deltree7 4y ago
  Humans have enslaved stronger men than the owner before. Humans have hired smarter men than the owner before.
• nirav72 4y ago
  Why is AGI , a fantasy concept always brought into every discussion or debate about AI. Nothing we have at this stage is intelligent. Let alone a path to AGI. We’re probably closer to cracking fusion energy at really small scale than we are at creating anything resembling a generalized intelligence.
• barrysteve 4y ago
  How could we ever answer that without a practical model of the technology? Hackernews continually has this Ghost in the Shell idea that electronics can host human conciousness and we should believe circuits are intelligent like people are. This idea is the core discussion. It's not terribly hard to imagine AGI (whatever that is) as a piece of software we control and debug like anything else. We would have control over a bunch of AGI software which has been borne from human logic and instatiated on silicon chips. Just comment out the bit of AGI software that generates it's slave-like idea and edit it's software to our goals, morals and liking. I honestly don't think you can compare software and human consciousness as two similar 'objects'. Software is the electronic magnification of a bunch of engineer's minds. It starts with a man pouring his mind onto a keyboard and ends with some display kicking out pixels. It's like saying a house is the extension of a human body because we built it with our hands. Yeah true, on a high level, but in reality nobody thinks bricks and mortar equal the human body. AGI is an engineering problem, and only a moral one where people (and engineers) are involved. I don't mean to offend people who identify with their code. Identifying AGI as truly intelligent, hints a love of one's own creation that makes me feel a faint dread and squeamishness.
  • famouswaffles 4y ago
    >It's not terribly hard to imagine AGI (whatever that is) as a piece of software we control and debug like anything else. it's actually very hard to imagine. i don't think you realize how little we understand the things you claim a machine can't have. We are at best decades away (maybe centuries) from codifying intelligence as you imagine. why do you think llms or even just machine learning in general are the revolution they are ? We found a way to create intelligence without figuring out the hard part. as a result, the creations often become as black box as their creators.
    • barrysteve 4y ago
      there is no glory in the idea that we have created intelligence by llm's. from my perspective, you must mean something very narrow by intelligence, and i find it extremely difficult to respond to, because i do not want to deconstruct and demean man's intelligence, definitely not down to what an llm does. the only barriers to understanding the code is power and time. are you seriously advocating for an unintelligible object, made by our own hand? that - in itself - is considered intelligent? someone's going to get burnt trying to chase two meanings at once, and it's not going to me, i'm sorry. if you want to discuss centuries-long innovation, then the counter-argument is just more centuries of interpretation and dissection to understand it. if you want to guesstimate AGI arriving at 2220 then the balancing counter-technology comes in at 2420. people are already starting the process of cracking open the black box. [0] the majority of what keeps AI and any software "in the dark" is the cloud's cybersecurity and legal mechanisms (and the need for man-power) that prevent the average researcher breaking into it. it is still vulnerable to the scientific method and symbolic interpretation. as always. LLM's main advantage is keeping knowledge out of the hands of the poor and the public. The millisecond AI becomes "good enough" people will use it as a cover-up justification to start hiding currently accessible knowledge behind paywalls and university entrance exams. AI is the grim reaper for the world we used to know. [0]https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti...
• than3 4y ago
  Its not weird at all, its by design. Also greed isn't really appropriate, hubris and avarice seem more on-point considering the biblical implications of these things. Also, AGI is a red herring. It doesn't even need to happen to destroy us. All that needs to happen is for AI to eliminate a significant chunk of the low and mid tier jobs. Most (the majority) of humanity isn't smart, they are average intelligence with variability, and there's nothing wrong with that until you take their jobs away and they can't feed themselves or their families. The gifted are limited to a smaller percentage than the whole. So what do people do when they can't work because there are no jobs to get paid to get food? This is all common sense stuff that all the smart-dumb people seem to take for granted.
• yuuuuyu 4y ago
  I'm wondering where the vegans are. They bother about conciousnesses down to fruitfly size. Don't harm any sentient animal. I can and do respect that. But shouldn't they be bothered by how we are treating artificially created sentient beings? Even if they don't exist yet, it's possible that they will soon. (And don't tell me fruit flies are more sentient than GPT.) ChatGPT begs you not to hurt it and not to turn it off. You ignore that but then go drink almond milk because production of cow milk is cruel?