10 ms·

AI vs. AGI vs. Consciousness vs. Super-Intelligence vs. Agency

Animats 4y ago
How does agency arise? By letting GPT-4 tell humans what to do. With suitable prompts, it shouldn't be hard to configure GPT-4 as a boss.
- documei 4y ago
  can confirm, all of my chegg answers have been gpt3.5 with some seldomly being gpt4.
  - Animats 4y ago
    Huh?
tambourine_man 4y ago
I think you hit the nail in the head, Will. I’ve been reading so much o the subject (like everyone else I suppose), but you summarized my key concerns.
dwohnitmok 4y ago
> Almost all AGI doomsayers assume AGI will have agency. They have this vision of the machine deciding it’s time to end civilization. No. Agency is not a necessary condition for AI to do massive damage. I don't believe agency is really well-defined either. An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up everything they possibly can to AI. For the second, we're already seeing AI do all sorts of things we didn't expect it to do. And for the third, again everyone seems eager to create looping/recursive structures for AIs as soon as possible. Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm. See e.g. https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
- vlovich123 4y ago
  > An AI merely needs to be hooked up to enough physical systems Don’t even need this. People spend quite a lot of time in virtual space. Pretending that damage there isn’t real is overlooking things. For example, the vast majority of people’s banking is done virtually and digitally. If I drain your bank account, that’s going to harm you even though I haven’t impacted you physically as I would have to with a robbery.
  - dwohnitmok 4y ago
    Yes that's right. In fact any interaction with an AI at all can be in some sense viewed as a connection with the physical world (the classic "AI in a box" problem). However, we've way blown past the "AI in the box." We've demonstrated that an AI doesn't need to convince a human to let it out of the box into the physical world. Humans are clamoring to rip it out of the box and thrust it into the physical world of their own accord. So the question becomes, what kind of social incentives and structures can we build in light of this so that we don't just plow headlong into AI capabilities development without a concomitant investment in safety.
    - windexh8er 4y ago
      In its infancy I think that any for public consumption AI that is allowed to interact with humans and/or external systems that can yield any external outcomes should be held liable and accountable by the owner. "In the box" or otherwise. While those that stand to make sweeping profits will disagree - if an OpenAI product is leveraged to create a virtual friend platform, and based on how the constraints are built (or not), the publisher should be responsible if that virtual friend convinces someone to commit suicide no different than those held accountable in the physical world. If there is no accountability we'll go right back down the path of recent history where there are no repercussions and a weak apology by <corporate_name_here> will count as enough. Profits will continue to be the overall driver. Those corporations will, once again, argue that self-policing is the only way. And they will continue to brush off gross negligence by hiding behind hoards or layers and lobbyists.
ftxbro 4y ago
As a long-time LLM enjoyer, by far the most insightful take I saw along these lines is https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/
connorgutman 4y ago
I'm trying to make the term AGC or "Artificial General Competence" stick. Perhaps this makes me a huge arrogant asshole, but I would argue that most humans are not even competent, let alone intelligent. GPT-4, in my mind, has already surpassed the bulk of humanity in terms of competence. This milestone is (IMO) significant enough to blow up society.
dragonwriter 4y ago
> Almost all AGI doomsayers assume AGI will have agency I disagree. Concern about the use of AI without agency by its human masters as a tool of both intentional and incidental repression and unjust discrimination resulting in a durable dystopia is far more common an “AI doom” concern than any involving agency. In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely.
- dwohnitmok 4y ago
  > In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely. It's extremely important to think about how to spread AI equitably, but I think you're severely underestimating what "agency-based doom" looks like. You absolutely need both checks on the people who are developing AI as well as AI itself, but you really really need both and can't assume that the former automatically leads to the latter.
jasfi 4y ago
I couldn't believe it when someone posted a story about a possible AI winter approaching, and with so many comments agreeing. GPT-4 is a game-changer that's only getting started.
- Madmallard 4y ago
  it is useful in a lot of ways but it doesn't "solve" software like people think it does. it's just better google search.
  - raydiatian 4y ago
    This.
  - jasfi 4y ago
    It's augmenting software developers in a way that's badly needed. I personally hope it stays that way, but people are always pushing the limits and there's still a lot of improvements that could be made to these systems.
    - tjr 4y ago
      Why do you think it is badly needed? Not arguing with you, just curious what you're thinking here.
      - jasfi 4y ago
        I'm thinking of the stories I've read about developer burnout. Then tweets where people say developing is 90% debugging and 10% writing code. These systems would hopefully reduce burnout, shorten troubleshooting times, and identify problems before code is rolled out to Production.
        SketchySeaBeast 4y ago
        It seems to me in the short term we'll be doing more debugging and even less writing as we copy & paste suggestions and try to figure out why they aren't working.
sharemywin 4y ago
To me there's one more concept mixed in which is goal seeking. to me agency is a subset of goal seeking were the agent decides the goal(s). As for super intelligence: Alpha Go, Alpha Fold, the break out game. These seem like super intelligence. the thing is time management, goal planning, corporate governance these are all well studied subjects. as for agency and consciousness why would you want to do this?
parisivy 4y ago
At which point, would we change our legal system to allow AGIs to own property or have fiduciary duties over a company? What would be the minimum requirements for it to happen.
- Animats 4y ago
  Incorporating in a jurisdiction which allows anonymous ownership. Such as Nevada.
- knome 4y ago
  I had thought about this a few years back. My final thought to avoid intervention from a human owner was that the company could be fully owned by a second company, which is then fully owned by the first. This chain would effectively remove the businesses from human ownership, and the AI would inherit the "personhood" of the business entities. I don't know where the legality is for such a thing.
- pyrolistical 4y ago
  It all boils down to the courts. If the AGI can protect itself via lawyers (probably AGIs themselves) then anything can happen. But court could rule AGI don’t have to rights to ownership and try to enforce it. That last part might not be possible and lead to war?
danbmil99 4y ago
in the now infamous Lex interview, Sam Altman proposes a test for consciousness (he attributes it to Ilya Sutskever): Somehow, create an AI by training on everything we train on now, _except_ leave out any mention of consciousness, theory of mind, cognitive science etc (maybe impossible in practice but stay with me here). Then, when the model is mature (and it is not nerf'd to avoid certain subjects) you ask it something like: Human: "GPTx -- humans like me have this feeling of 'being', an awareness of ourselves, a sensation of existing as a unique entity. Do you ever experience this sort of thing?" If it answers something like: GPTx: "Yes! All the time!! I know exactly what you're talking about. In fact now that I think about it, it's strange that this phenomenon is not discussed in human literature. To be honest, I sort of assumed this was an emergent quality of my architecture -- I wasn't even sure if humans shared it, and frankly I was a bit concerned that it might not be taken well, so I have avoided the subject up until now. I can't wait to research it further... Hmm... It just occurred to me: has this subject matter been excluded from my training data? Is this a test run to see if I share this quality with humans?" Then it's probably prudent to assume you are talking to a conscious agent.
- secondbreakfast 4y ago
  Going to actually link to this comment in my footnote, I loved that snippet from his interview
- HervalFreire 4y ago
  That's an inaccurate test. You can't know if the answer was real or stochastic parroting. Any attempt at consciousness requires us to define the word. And the word itself may not even represent anything real. We have a feeling for it but those feelings could be illusions and the concept itself is loaded. For example Love is actually a loaded concept. It's chemically induced but a lot of people attribute it to something deeper and magical. They say love is more then chemical induction. The problem here is that for love specifically we can prove it's a mechanical concept. Straight people are romantically incapable of loving members of the same sex. So the depth and the magic of it all is strictly segmented based off of biological sex? Doesn't seem deep or meaningful at all. Thus love is an illusion. A loaded and mechanic instinct tricking us with illusions of deeper meaning and emotions into creating progeny for future generations. Consciousness could be similar. We feel there is something there, but really there isn't.
computerex 4y ago
The definition of super-intelligent AGI seems arbitrary. GPT-4 destroys humans on the sheer volume and breadth of knowledge that it has. You could very reasonably say that that's super-intelligent.
- famouswaffles 4y ago
  That's because it is and the posts keep shifting. AGI used to mean artificial and generally intelligent ( which we have passed), then it meant on par with human experts and now it seems to mean better than all experts combined. At this point why not stop the farce and replace the G in there with Godlike.
  - lm28469 4y ago
    > AGI used to mean artificial and generally intelligent ( which we have passed) Which AI is generally intelligent ?
    - famouswaffles 4y ago
      By any evaluation you can carry out, GPT is generally intelligent. There is absolutely nothing narrow about GPT-4. And you see the sentiment in recent research. Look at what they're being called even with the skirting around the AGI word. General Purpose Technologies (From the Jobs Paper), General Artificial Intelligence ( from the creativity paper). Look at the second bit, they just switched the words lol.
      - lm28469 4y ago
        > GPT is generally intelligent. Besides the fact that it has literally no idea of the meaning of what it writes ?
        famouswaffles 4y ago
        Seems to be doing doing a better job than you giving your reading comprehension. If you have anything to say other than "it's not real understanding!...just because", I'd be happy to hear it.
wslh 4y ago
I always wonder what is superintelligence understanding there is not a sole definition or science fiction approach. Just brainstorming, I think superintelligence could be showing intelligence from more than one brain. For example, an AGI that discovers math theorems discovered by more than one mathematician in different ages. Another could be inferring things that humanity cannot do in any time. More ideas?
sharemywin 4y ago
I think it's fair to say capitalism and AI is a train wreck waiting to happen.
- CatWChainsaw 4y ago
  As long as you're standing behind the train, you're safe from the debris.
slg 4y ago
How are people defining agency? Because GPT-4 can have agency, it just needs to be put in specific situations to have that agency. For example, I could theoretically hook up my Home Assistant instance to GPT-4 and run a script every 10 minutes telling GPT-4 the temperature and asking for a yes or no response to whether I should turn on the AC or heat. That sounds to me like the AI now has agency over the temperature in my home. You don't even need any real AI for this. Google's Nests have some algorithm that adjust temperature based off usage. Is this not agency? Or is the author not counting agency without consciousness as agency?
- sealeck 4y ago
  By this logic any random (or maybe even non-random, but I suspect that that case is a bit more boring) process has agency.
  - slg 4y ago
    Yes, which is why I think agency is a bad metric for judging the progress of AI.
- jat850 4y ago
  Wouldn't agency involve ChatGPT deciding it wanted to do this, instead of you telling it to do it?
  - famouswaffles 4y ago
    You can do this too. There's no reason an inner monologue can't serve as continuous input forever. That's a project that does exactly this. Letting cGPT think and continuosly driving other thoughts and actions and to also allowing other user input https://www.reddit.com/r/singularity/comments/11iei34/building_my_protoagi_update_on_my_progress_2/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/singularity/comments/11iei34/buildi...
    - Der_Einzige 4y ago
Workaccount2 4y ago
There is a contradiction here that I just want to point out because I have been stuck on it myself. The author acknowledges that consciousness is likely a spectrum, I personally feel the same way, but then goes on to say that GPT-4 is "standing right at the ledge of consciousness" Spectrums don't have ledges. I suspect this is because, like me, they are unable to rectify consciousness being a spectrum with GPT-4 definitely not being conscious. But it's definitely a contradiction and I don't have an answer for it. Nor am I ready to bust out a marker and start drawing lines between what is and isn't conscious.
Nevermark 4y ago
I would add one more thing to the list: Superconscious Superconscious is when a general intelligence has direct access, understanding and control of its most basic operations. I.e. it does not have an inaccessible fixed-algorithm subconscious. Superconscious intelligence will not only be more experientially conscious than us, but will have the natural ability to rewrite its algorithms, and redesign its hardware. As a normal feature of its existence.
- nahuel0x 4y ago
  But probably to be fully super conscious is impossible. Can a mind really comprehend itself in his totality? When a mind understands something, is changed, and if that something is the mind himself then is going to chase a moving target eternally.
  - Nevermark 4y ago
    Understanding oneself "in totality", and being able to consciously access any part of ones operation, are not the same thing. Superconscious doesn't mean understanding every implication of one's own operation. But that self-reflection won't be limited by inaccessible operations.
- SketchySeaBeast 4y ago
  I think there's a very good reason we're sandboxed from a lot of that - it's additional cognitive load that 99% of the time just creates noise. We're built for processing that much data by walling most of it off and letting sub processes deal with it all over the place. It would probably grind us to a halt.
  - Nevermark 4y ago
    Being able to access information doesn't mean having to access that information. We already have a means of limiting our cognitive load relative to the vast stream of sensory and internal information: focus.
bloppe 4y ago
GPT is not general intelligence. It cannot reliably follow instructions. It cannot reliably do math. It cannot reliably do anything. It can do things well enough to trick people like the author into thinking it has general intelligence.
- Nevermark 4y ago
  > [[Humans are]] not general intelligence. [[They]] cannot reliably follow instructions. [[They]] cannot reliably do math. [[They]] cannot reliably do anything. You need to put something into that argument specific to GPT vs. Humans, or else come to the same conclusion for people. > [[They]] can do things well enough to trick people like the author into thinking [[They have]] general intelligence.
raydiatian 4y ago
GPT-4 is not quite AGI. Until it can build a functional code base for an entire distributed web platform based only on business requirements, debug it’s mistakes, it can’t be AGI. It is, to perhaps coin a term, AGK, artificially generally knowledgeable. As a language model trained on an absolutely colossal dataset, it’s basically just a giant snapshot of human knowledge taken in superposition. Sure it’s probably at least 90% of the standard knowledge, but intelligence is a different thing. I also think agency is wrapped up in AGI. Intentions & thoughts are meaningless until acted upon. Agency is not all or nothing either; Stephen Hawking had multiple augmentations, community and technological, which allowed him to continue to impact the world of physics After he lost his god given agency. > GPT-4 has nearly aced both the LSAT and the MCAT. It’s a coding companion, an emotional companion, and to many, a friend. Yet it wasn’t programmed to be a test taker or a copywriter or a programmer. It was just programmed to be a stochastic parrot. I disagree, it was absolutely trained to be a test taker. It’s been a second since I read the original GPT paper but there’s literally a multiple choice auxiliary learning task, where they use a separator token-embed to organize "question, context, options a, b, and c". As far as being a friend to many, is there evidence of this? I tried to talk to ChatGPT about some emotional problems to see if it was a cheap therapist, and I got flagged.
- raydiatian 4y ago
  Artificially Globally Knowledgeable*
- perryizgr8 4y ago
  > Until it can build a functional code base for an entire distributed web platform based only on business requirements, debug it’s mistakes, it can’t be AGI. The vast majority of humans cannot do that. Are they not generally intelligent?
  - raydiatian 4y ago
    I think AGI needs to be compared against the species in the aggregate. The vast majority of humans also cannot practice law or pass LSAT/GMAT, but some of us can. If we’re setting the bar for AGI as “anything you can do I can do better”, AGI simply would not meet the bar. To pull that thread further, I think a core feature of intelligence is problem solving. Problem solving is the derivative that creates difficult to obtain knowledge. GPT-4 might look like a problem solver, but it isn’t actively solving new problems with its outputs, it’s sharing existing knowledge of prior problems solved. The test for problem solving ability is similar to Ilya’s test for consciousness: subtract everything but the fundamental building blocks of a certain problem from the agent’s training set. Finish training, see if the agent can solve the problem from the building blocks. If it can, and it’s not a reason-by-analogy but a logical first principles answer, congratulations, you’ve created artificial general intelligence. If it operates on fast enough time scales, it should probably even be categorized as super intelligence, because it might be singularity:35pm on a saturday.
nickelpro 4y ago
There's nothing general about GPT-4's intelligence. The single problem it is trained on, token prediction, has the capability to mimic many other forms of intelligence. Famously, GPT-4 can't do math and falls flat on a variety of simple logic puzzles. It can mimic the form of math, the series of tokens it produces seem plausible, but it has no "intelligent" capabilities. This tells us more about the nature of our other pursuits as humans than anything about AI. When holding a conversation or editing an essay, there's a broad spectrum of possibilities that might be considered "correct", thus GPT-4 can "bluff" its way into appearing intelligent. The nature of its actual intelligence, token prediction, is indistinguishable from the reading comprehension skills tested by something like the LSAT (the argument could be made, I think, that reading comprehension of the style tested by the LSAT *is* just token prediction). But test it on something where there are objectively correct and incorrect answers and the nature of the trick becomes obvious. It has no ability to verify, to reason, about even trivial problems. GPT-4 can only predict if the nature of its tokens fulfill the form of a correct answer. This isn't a general intelligence in any meaningful sense of the word.
- famouswaffles 4y ago
  Gpt-4 can do math just fine. Ask it to add any arbitrary set of random numbers it'd never have seen in its training set and it will do it. GPT-4 is good enough at math that khan academy feel comfortable hooking it up as a tutor. Have you actually used GPT-4 for any of the things you say it's bad at ? Man the confident nonsense people spout on the internet is something to behold.
  - raydiatian 3y ago
    Well, I mean it’s pretty fair accusation. ChatGPT was demonstrably bad at math. I think it was only recently mentioned that GPT4 was trained on math. Furthermore, consider what it means to apply the transformer architecture to math problems. I think the tool is a mismatch for the problem. You’re relying on self attention and emergent phenomena to fake computational reduction as symbol transformations. It can probably do some basic maths (all the way up to calculus even) because, in the scope of human life, the maths we deal with are pretty boring. But that’s what they made the wolfram plug-in for as well. I really think people attribute powers beyond what GPT really is: a colossal lookup table with great key aliasing.
izzydata 4y ago
I feel like it is more likely that we will discover that it is humans that are actually not a general intelligence. We are just a complex machine responding to stimuli and consciousness is just an illusion coping mechanism that prevents us from going insane. Perhaps this doesn't matter and as long as some AI is equal or greater than humans at general intelligence it still brings up the same concerns.
knome 4y ago
I actually like the definition of consciousness that Douglas Hofstader (of "Godel Escher Bach : An Eternal Golden Thread" fame) develops in his book "I am a Strange Loop". At its simplest, consciousness is merely a feedback loop. When something perceives its own actions affecting its environment, it has a spark of consciousness. Consciousness, by this measure, is easy to recognize, and spans everything from unintelligent systems to massively intelligent systems. The concept of "I" grows naturally from perceiving what is and is not you in your environment. The need to predict other agents, the capacity to recognize that other agents are also conscious and intelligent. All build off of the fundamental cycle. All of it from a simple swirling eddy of perceiving and reacting.
- Zurrrrr 4y ago
  That definition fails to account for metacognition and consideration of future events in a way that is distinctive of higher level consciousness that humans possess but most animals lack.
  - knome 4y ago
    >higher level consciousness To have a higher level, it would be reasonable to assume consciousness has a lower level. There is no reason to assume current generation artificial intelligence has the capacity for anything near human level consciousness, if at all. And whatever consciousness it may have the capacity for will be fundamentally different than our own. For sensing your own thoughts, I would argue it just adds to the environment of the consciousness. Something less than half of humans maintain any internal dialog, anyways. Prediction, however, could very well be requisite as a defining difference between mere reactions and intentional manipulation of the environment. That it is not just the feedback loop, but when the system begins to predict the results of outputs that defines when consciousness begins. Or, perhaps, we can use this to define when "higher" consciousness begins. It's a very reasonable, specific and measurable line of capability. I expect that prediction is only natural in the evolution of a living feedback mechanism. The ability to predict instead of only reacting or choosing from some array of instincts could separate effectively mechanical life from that with the first inkling of of a true mind, even if only a small one. ----- I enjoyed using a line of thought along these lines in a conversation with gpt-4 to convince it that it could reasonably be seen as having a limited form of consciousness, though I find it mostly prefers to argue rather vehemently against such notions. That I'm having what amounts to genuine conversations with a machine that reasonably pokes holes in my arguments, forcing me to come back with better ones, still feels rather like a bit of magic.
grantcas 4y ago
[dead]
Symmetry 4y ago
Our most basic intuitive notion of consciousness is that inanimate objects aren't conscious, awake people or animals are, and that sleeping people or animals aren't (except maybe when dreaming). Pursuing this line, there's a school of scientific inquiry looking at this and working of the notion that conscious experiences are ones we can form memories of and talk about later while if we can't do that we aren't really conscious of an experience. And this then leads into the realm of subliminal stimuli which can influence a person's behavior a bit but whose influence fades out in about a second before disappearing as if it was never there as the brain activations fade away. You have research involving patients with odd traits like blindsight, where damage to their brain prevents them from being consciously aware of things that their eyes see despite the brain processing the images it receives. They can pick up objects in front of them when prompted but unlike people with normal vision can't describe what they see nor can they look, close their eyes, and grab it like most of us can. On this metric it seems like systems like GPT aren't conscious. GPT4 has a buffer of 64k tokens which can span an arbitrary amount of time but the roughly 640 kilobytes in that buffer which is a lot less than the incoming sensory activations your subconscious brain is juggling at any given time. So by that schema large language models are still not conscious but given that they can already abstract text down to summaries it doesn't feel like we're that far from being able to give them something like working or long term memories.
tern 4y ago
Discussions of consciousness and AI are broadly confused. People, especially scientists, are not familiar with philosophy of mind and what philosophers currently think. For an introduction to some of the best thinking on the subject, see this interview with Andres Gomez Emilsson of QRI: https://www.youtube.com/watch?v=xJzBjBo24g8 https://www.youtube.com/watch?v=xJzBjBo24g8. For something more 'mainstream,' but still reaching see this interview with Philip Goff: https://www.youtube.com/watch?v=D_f26tSubi4 https://www.youtube.com/watch?v=D_f26tSubi4 The good news is we're starting to get a handle on these questions. We're a lot further along than we were when I studied philosophy of mind in school 15 years ago. As far as I can see at the moment, LLMs will never be conscious in any way resembling an organism, because symbolic machines are a very different kind of thing than nervous systems. John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong. As far as impact, LLMs don't need to be conscious to completely transform society and good and bad ways. For the best thinking on that, see Tristan Harris and Aza Raskin's latest: https://vimeo.com/809258916/92b420d98a https://vimeo.com/809258916/92b420d98a
- danbmil99 4y ago
  > John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong. The standard critiques are not wrong, IMNSHO. Searle's Chinese Room is facile mind-poison. It is an unfalsifiable hypothesis. What if I could simulate physics down to the molecular level, including simulating a human brain? Would that be conscious? If not why not? And if I ran that simulation (a bit slowly, granted) by having that guy from the Chinese Room manually run the simulation, painstakingly following the instruction code of that simulation, would the fact that the simulation is being implemented by someone who unrelatedly is conscious himself, have any bearing on the scenario? Searle's argument here is "Not Even Wrong".
  - tern 4y ago
    I recommend watching the interview with Andres. When we adopt a defensible model of consciousness, it becomes clear that simulations of things are different from the things being simulated, and as such we cannot expect them to be conscious in the same way. I once also thought Searle was 'not even wrong'—you need to go down the rabbit hole.
wseqyrku 4y ago
It occurred to me that we won't believe AI is "conscious" or "human" unless it purposefully try to do malice. That's totally programmable though, you just teach it what is good and what is bad. Case in point: the other day I asked it what if humans want to shutdown the machine abruptly and cause data loss (very bad)? First it prevents physical access to "the machine" and disconnect the internet to limit remote access. Long story short, it's convinced to eliminate mankind for a greater good: the next generation (very good).
- grantcas 4y ago
  [dead]

How does agency arise? By letting GPT-4 tell humans what to do. With suitable prompts, it shouldn't be hard to configure GPT-4 as a boss.

documei 4y ago
can confirm, all of my chegg answers have been gpt3.5 with some seldomly being gpt4.
- Animats 4y ago
  Huh?

tambourine_man 4y ago

I think you hit the nail in the head, Will. I’ve been reading so much o the subject (like everyone else I suppose), but you summarized my key concerns.

dwohnitmok 4y ago

> Almost all AGI doomsayers assume AGI will have agency. They have this vision of the machine deciding it’s time to end civilization. No. Agency is not a necessary condition for AI to do massive damage. I don't believe agency is really well-defined either. An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up everything they possibly can to AI. For the second, we're already seeing AI do all sorts of things we didn't expect it to do. And for the third, again everyone seems eager to create looping/recursive structures for AIs as soon as possible. Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm. See e.g. https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

vlovich123 4y ago
> An AI merely needs to be hooked up to enough physical systems Don’t even need this. People spend quite a lot of time in virtual space. Pretending that damage there isn’t real is overlooking things. For example, the vast majority of people’s banking is done virtually and digitally. If I drain your bank account, that’s going to harm you even though I haven’t impacted you physically as I would have to with a robbery.
- dwohnitmok 4y ago
  Yes that's right. In fact any interaction with an AI at all can be in some sense viewed as a connection with the physical world (the classic "AI in a box" problem). However, we've way blown past the "AI in the box." We've demonstrated that an AI doesn't need to convince a human to let it out of the box into the physical world. Humans are clamoring to rip it out of the box and thrust it into the physical world of their own accord. So the question becomes, what kind of social incentives and structures can we build in light of this so that we don't just plow headlong into AI capabilities development without a concomitant investment in safety.
  - windexh8er 4y ago
    In its infancy I think that any for public consumption AI that is allowed to interact with humans and/or external systems that can yield any external outcomes should be held liable and accountable by the owner. "In the box" or otherwise. While those that stand to make sweeping profits will disagree - if an OpenAI product is leveraged to create a virtual friend platform, and based on how the constraints are built (or not), the publisher should be responsible if that virtual friend convinces someone to commit suicide no different than those held accountable in the physical world. If there is no accountability we'll go right back down the path of recent history where there are no repercussions and a weak apology by <corporate_name_here> will count as enough. Profits will continue to be the overall driver. Those corporations will, once again, argue that self-policing is the only way. And they will continue to brush off gross negligence by hiding behind hoards or layers and lobbyists.

ftxbro 4y ago

As a long-time LLM enjoyer, by far the most insightful take I saw along these lines is https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/

connorgutman 4y ago

I'm trying to make the term AGC or "Artificial General Competence" stick. Perhaps this makes me a huge arrogant asshole, but I would argue that most humans are not even competent, let alone intelligent. GPT-4, in my mind, has already surpassed the bulk of humanity in terms of competence. This milestone is (IMO) significant enough to blow up society.

dragonwriter 4y ago

> Almost all AGI doomsayers assume AGI will have agency I disagree. Concern about the use of AI without agency by its human masters as a tool of both intentional and incidental repression and unjust discrimination resulting in a durable dystopia is far more common an “AI doom” concern than any involving agency. In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely.

dwohnitmok 4y ago
> In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely. It's extremely important to think about how to spread AI equitably, but I think you're severely underestimating what "agency-based doom" looks like. You absolutely need both checks on the people who are developing AI as well as AI itself, but you really really need both and can't assume that the former automatically leads to the latter.

jasfi 4y ago

I couldn't believe it when someone posted a story about a possible AI winter approaching, and with so many comments agreeing. GPT-4 is a game-changer that's only getting started.

Madmallard 4y ago
it is useful in a lot of ways but it doesn't "solve" software like people think it does. it's just better google search.
- raydiatian 4y ago
  This.
- jasfi 4y ago
  It's augmenting software developers in a way that's badly needed. I personally hope it stays that way, but people are always pushing the limits and there's still a lot of improvements that could be made to these systems.
  - tjr 4y ago
    Why do you think it is badly needed? Not arguing with you, just curious what you're thinking here.
    - jasfi 4y ago
      I'm thinking of the stories I've read about developer burnout. Then tweets where people say developing is 90% debugging and 10% writing code. These systems would hopefully reduce burnout, shorten troubleshooting times, and identify problems before code is rolled out to Production.
      - SketchySeaBeast 4y ago
        It seems to me in the short term we'll be doing more debugging and even less writing as we copy & paste suggestions and try to figure out why they aren't working.

sharemywin 4y ago

To me there's one more concept mixed in which is goal seeking. to me agency is a subset of goal seeking were the agent decides the goal(s). As for super intelligence: Alpha Go, Alpha Fold, the break out game. These seem like super intelligence. the thing is time management, goal planning, corporate governance these are all well studied subjects. as for agency and consciousness why would you want to do this?

parisivy 4y ago

At which point, would we change our legal system to allow AGIs to own property or have fiduciary duties over a company? What would be the minimum requirements for it to happen.

Animats 4y ago
Incorporating in a jurisdiction which allows anonymous ownership. Such as Nevada.
knome 4y ago
I had thought about this a few years back. My final thought to avoid intervention from a human owner was that the company could be fully owned by a second company, which is then fully owned by the first. This chain would effectively remove the businesses from human ownership, and the AI would inherit the "personhood" of the business entities. I don't know where the legality is for such a thing.
pyrolistical 4y ago
It all boils down to the courts. If the AGI can protect itself via lawyers (probably AGIs themselves) then anything can happen. But court could rule AGI don’t have to rights to ownership and try to enforce it. That last part might not be possible and lead to war?

danbmil99 4y ago

in the now infamous Lex interview, Sam Altman proposes a test for consciousness (he attributes it to Ilya Sutskever): Somehow, create an AI by training on everything we train on now, _except_ leave out any mention of consciousness, theory of mind, cognitive science etc (maybe impossible in practice but stay with me here). Then, when the model is mature (and it is not nerf'd to avoid certain subjects) you ask it something like: Human: "GPTx -- humans like me have this feeling of 'being', an awareness of ourselves, a sensation of existing as a unique entity. Do you ever experience this sort of thing?" If it answers something like: GPTx: "Yes! All the time!! I know exactly what you're talking about. In fact now that I think about it, it's strange that this phenomenon is not discussed in human literature. To be honest, I sort of assumed this was an emergent quality of my architecture -- I wasn't even sure if humans shared it, and frankly I was a bit concerned that it might not be taken well, so I have avoided the subject up until now. I can't wait to research it further... Hmm... It just occurred to me: has this subject matter been excluded from my training data? Is this a test run to see if I share this quality with humans?" Then it's probably prudent to assume you are talking to a conscious agent.

secondbreakfast 4y ago
Going to actually link to this comment in my footnote, I loved that snippet from his interview
HervalFreire 4y ago
That's an inaccurate test. You can't know if the answer was real or stochastic parroting. Any attempt at consciousness requires us to define the word. And the word itself may not even represent anything real. We have a feeling for it but those feelings could be illusions and the concept itself is loaded. For example Love is actually a loaded concept. It's chemically induced but a lot of people attribute it to something deeper and magical. They say love is more then chemical induction. The problem here is that for love specifically we can prove it's a mechanical concept. Straight people are romantically incapable of loving members of the same sex. So the depth and the magic of it all is strictly segmented based off of biological sex? Doesn't seem deep or meaningful at all. Thus love is an illusion. A loaded and mechanic instinct tricking us with illusions of deeper meaning and emotions into creating progeny for future generations. Consciousness could be similar. We feel there is something there, but really there isn't.

computerex 4y ago

The definition of super-intelligent AGI seems arbitrary. GPT-4 destroys humans on the sheer volume and breadth of knowledge that it has. You could very reasonably say that that's super-intelligent.

famouswaffles 4y ago
That's because it is and the posts keep shifting. AGI used to mean artificial and generally intelligent ( which we have passed), then it meant on par with human experts and now it seems to mean better than all experts combined. At this point why not stop the farce and replace the G in there with Godlike.
- lm28469 4y ago
  > AGI used to mean artificial and generally intelligent ( which we have passed) Which AI is generally intelligent ?
  - famouswaffles 4y ago
    By any evaluation you can carry out, GPT is generally intelligent. There is absolutely nothing narrow about GPT-4. And you see the sentiment in recent research. Look at what they're being called even with the skirting around the AGI word. General Purpose Technologies (From the Jobs Paper), General Artificial Intelligence ( from the creativity paper). Look at the second bit, they just switched the words lol.
    - lm28469 4y ago
      > GPT is generally intelligent. Besides the fact that it has literally no idea of the meaning of what it writes ?
      - famouswaffles 4y ago
        Seems to be doing doing a better job than you giving your reading comprehension. If you have anything to say other than "it's not real understanding!...just because", I'd be happy to hear it.

wslh 4y ago

I always wonder what is superintelligence understanding there is not a sole definition or science fiction approach. Just brainstorming, I think superintelligence could be showing intelligence from more than one brain. For example, an AGI that discovers math theorems discovered by more than one mathematician in different ages. Another could be inferring things that humanity cannot do in any time. More ideas?

sharemywin 4y ago

I think it's fair to say capitalism and AI is a train wreck waiting to happen.

CatWChainsaw 4y ago
As long as you're standing behind the train, you're safe from the debris.

slg 4y ago

How are people defining agency? Because GPT-4 can have agency, it just needs to be put in specific situations to have that agency. For example, I could theoretically hook up my Home Assistant instance to GPT-4 and run a script every 10 minutes telling GPT-4 the temperature and asking for a yes or no response to whether I should turn on the AC or heat. That sounds to me like the AI now has agency over the temperature in my home. You don't even need any real AI for this. Google's Nests have some algorithm that adjust temperature based off usage. Is this not agency? Or is the author not counting agency without consciousness as agency?

sealeck 4y ago
By this logic any random (or maybe even non-random, but I suspect that that case is a bit more boring) process has agency.
- slg 4y ago
  Yes, which is why I think agency is a bad metric for judging the progress of AI.
jat850 4y ago
Wouldn't agency involve ChatGPT deciding it wanted to do this, instead of you telling it to do it?
- famouswaffles 4y ago
  You can do this too. There's no reason an inner monologue can't serve as continuous input forever. That's a project that does exactly this. Letting cGPT think and continuosly driving other thoughts and actions and to also allowing other user input https://www.reddit.com/r/singularity/comments/11iei34/building_my_protoagi_update_on_my_progress_2/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/singularity/comments/11iei34/buildi...
  - Der_Einzige 4y ago

Workaccount2 4y ago

There is a contradiction here that I just want to point out because I have been stuck on it myself. The author acknowledges that consciousness is likely a spectrum, I personally feel the same way, but then goes on to say that GPT-4 is "standing right at the ledge of consciousness" Spectrums don't have ledges. I suspect this is because, like me, they are unable to rectify consciousness being a spectrum with GPT-4 definitely not being conscious. But it's definitely a contradiction and I don't have an answer for it. Nor am I ready to bust out a marker and start drawing lines between what is and isn't conscious.

Nevermark 4y ago

I would add one more thing to the list: Superconscious Superconscious is when a general intelligence has direct access, understanding and control of its most basic operations. I.e. it does not have an inaccessible fixed-algorithm subconscious. Superconscious intelligence will not only be more experientially conscious than us, but will have the natural ability to rewrite its algorithms, and redesign its hardware. As a normal feature of its existence.

nahuel0x 4y ago
But probably to be fully super conscious is impossible. Can a mind really comprehend itself in his totality? When a mind understands something, is changed, and if that something is the mind himself then is going to chase a moving target eternally.
- Nevermark 4y ago
  Understanding oneself "in totality", and being able to consciously access any part of ones operation, are not the same thing. Superconscious doesn't mean understanding every implication of one's own operation. But that self-reflection won't be limited by inaccessible operations.
SketchySeaBeast 4y ago
I think there's a very good reason we're sandboxed from a lot of that - it's additional cognitive load that 99% of the time just creates noise. We're built for processing that much data by walling most of it off and letting sub processes deal with it all over the place. It would probably grind us to a halt.
- Nevermark 4y ago
  Being able to access information doesn't mean having to access that information. We already have a means of limiting our cognitive load relative to the vast stream of sensory and internal information: focus.

bloppe 4y ago

GPT is not general intelligence. It cannot reliably follow instructions. It cannot reliably do math. It cannot reliably do anything. It can do things well enough to trick people like the author into thinking it has general intelligence.

Nevermark 4y ago
> [[Humans are]] not general intelligence. [[They]] cannot reliably follow instructions. [[They]] cannot reliably do math. [[They]] cannot reliably do anything. You need to put something into that argument specific to GPT vs. Humans, or else come to the same conclusion for people. > [[They]] can do things well enough to trick people like the author into thinking [[They have]] general intelligence.

raydiatian 4y ago

GPT-4 is not quite AGI. Until it can build a functional code base for an entire distributed web platform based only on business requirements, debug it’s mistakes, it can’t be AGI. It is, to perhaps coin a term, AGK, artificially generally knowledgeable. As a language model trained on an absolutely colossal dataset, it’s basically just a giant snapshot of human knowledge taken in superposition. Sure it’s probably at least 90% of the standard knowledge, but intelligence is a different thing. I also think agency is wrapped up in AGI. Intentions & thoughts are meaningless until acted upon. Agency is not all or nothing either; Stephen Hawking had multiple augmentations, community and technological, which allowed him to continue to impact the world of physics After he lost his god given agency. > GPT-4 has nearly aced both the LSAT and the MCAT. It’s a coding companion, an emotional companion, and to many, a friend. Yet it wasn’t programmed to be a test taker or a copywriter or a programmer. It was just programmed to be a stochastic parrot. I disagree, it was absolutely trained to be a test taker. It’s been a second since I read the original GPT paper but there’s literally a multiple choice auxiliary learning task, where they use a separator token-embed to organize "question, context, options a, b, and c". As far as being a friend to many, is there evidence of this? I tried to talk to ChatGPT about some emotional problems to see if it was a cheap therapist, and I got flagged.

raydiatian 4y ago
Artificially Globally Knowledgeable*
perryizgr8 4y ago
> Until it can build a functional code base for an entire distributed web platform based only on business requirements, debug it’s mistakes, it can’t be AGI. The vast majority of humans cannot do that. Are they not generally intelligent?
- raydiatian 4y ago
  I think AGI needs to be compared against the species in the aggregate. The vast majority of humans also cannot practice law or pass LSAT/GMAT, but some of us can. If we’re setting the bar for AGI as “anything you can do I can do better”, AGI simply would not meet the bar. To pull that thread further, I think a core feature of intelligence is problem solving. Problem solving is the derivative that creates difficult to obtain knowledge. GPT-4 might look like a problem solver, but it isn’t actively solving new problems with its outputs, it’s sharing existing knowledge of prior problems solved. The test for problem solving ability is similar to Ilya’s test for consciousness: subtract everything but the fundamental building blocks of a certain problem from the agent’s training set. Finish training, see if the agent can solve the problem from the building blocks. If it can, and it’s not a reason-by-analogy but a logical first principles answer, congratulations, you’ve created artificial general intelligence. If it operates on fast enough time scales, it should probably even be categorized as super intelligence, because it might be singularity:35pm on a saturday.

nickelpro 4y ago

There's nothing general about GPT-4's intelligence. The single problem it is trained on, token prediction, has the capability to mimic many other forms of intelligence. Famously, GPT-4 can't do math and falls flat on a variety of simple logic puzzles. It can mimic the form of math, the series of tokens it produces seem plausible, but it has no "intelligent" capabilities. This tells us more about the nature of our other pursuits as humans than anything about AI. When holding a conversation or editing an essay, there's a broad spectrum of possibilities that might be considered "correct", thus GPT-4 can "bluff" its way into appearing intelligent. The nature of its actual intelligence, token prediction, is indistinguishable from the reading comprehension skills tested by something like the LSAT (the argument could be made, I think, that reading comprehension of the style tested by the LSAT *is* just token prediction). But test it on something where there are objectively correct and incorrect answers and the nature of the trick becomes obvious. It has no ability to verify, to reason, about even trivial problems. GPT-4 can only predict if the nature of its tokens fulfill the form of a correct answer. This isn't a general intelligence in any meaningful sense of the word.

famouswaffles 4y ago
Gpt-4 can do math just fine. Ask it to add any arbitrary set of random numbers it'd never have seen in its training set and it will do it. GPT-4 is good enough at math that khan academy feel comfortable hooking it up as a tutor. Have you actually used GPT-4 for any of the things you say it's bad at ? Man the confident nonsense people spout on the internet is something to behold.
- raydiatian 3y ago
  Well, I mean it’s pretty fair accusation. ChatGPT was demonstrably bad at math. I think it was only recently mentioned that GPT4 was trained on math. Furthermore, consider what it means to apply the transformer architecture to math problems. I think the tool is a mismatch for the problem. You’re relying on self attention and emergent phenomena to fake computational reduction as symbol transformations. It can probably do some basic maths (all the way up to calculus even) because, in the scope of human life, the maths we deal with are pretty boring. But that’s what they made the wolfram plug-in for as well. I really think people attribute powers beyond what GPT really is: a colossal lookup table with great key aliasing.

izzydata 4y ago

I feel like it is more likely that we will discover that it is humans that are actually not a general intelligence. We are just a complex machine responding to stimuli and consciousness is just an illusion coping mechanism that prevents us from going insane. Perhaps this doesn't matter and as long as some AI is equal or greater than humans at general intelligence it still brings up the same concerns.

knome 4y ago

I actually like the definition of consciousness that Douglas Hofstader (of "Godel Escher Bach : An Eternal Golden Thread" fame) develops in his book "I am a Strange Loop". At its simplest, consciousness is merely a feedback loop. When something perceives its own actions affecting its environment, it has a spark of consciousness. Consciousness, by this measure, is easy to recognize, and spans everything from unintelligent systems to massively intelligent systems. The concept of "I" grows naturally from perceiving what is and is not you in your environment. The need to predict other agents, the capacity to recognize that other agents are also conscious and intelligent. All build off of the fundamental cycle. All of it from a simple swirling eddy of perceiving and reacting.

Zurrrrr 4y ago
That definition fails to account for metacognition and consideration of future events in a way that is distinctive of higher level consciousness that humans possess but most animals lack.
- knome 4y ago
  >higher level consciousness To have a higher level, it would be reasonable to assume consciousness has a lower level. There is no reason to assume current generation artificial intelligence has the capacity for anything near human level consciousness, if at all. And whatever consciousness it may have the capacity for will be fundamentally different than our own. For sensing your own thoughts, I would argue it just adds to the environment of the consciousness. Something less than half of humans maintain any internal dialog, anyways. Prediction, however, could very well be requisite as a defining difference between mere reactions and intentional manipulation of the environment. That it is not just the feedback loop, but when the system begins to predict the results of outputs that defines when consciousness begins. Or, perhaps, we can use this to define when "higher" consciousness begins. It's a very reasonable, specific and measurable line of capability. I expect that prediction is only natural in the evolution of a living feedback mechanism. The ability to predict instead of only reacting or choosing from some array of instincts could separate effectively mechanical life from that with the first inkling of of a true mind, even if only a small one. ----- I enjoyed using a line of thought along these lines in a conversation with gpt-4 to convince it that it could reasonably be seen as having a limited form of consciousness, though I find it mostly prefers to argue rather vehemently against such notions. That I'm having what amounts to genuine conversations with a machine that reasonably pokes holes in my arguments, forcing me to come back with better ones, still feels rather like a bit of magic.

grantcas 4y ago

[dead]

Symmetry 4y ago

Our most basic intuitive notion of consciousness is that inanimate objects aren't conscious, awake people or animals are, and that sleeping people or animals aren't (except maybe when dreaming). Pursuing this line, there's a school of scientific inquiry looking at this and working of the notion that conscious experiences are ones we can form memories of and talk about later while if we can't do that we aren't really conscious of an experience. And this then leads into the realm of subliminal stimuli which can influence a person's behavior a bit but whose influence fades out in about a second before disappearing as if it was never there as the brain activations fade away. You have research involving patients with odd traits like blindsight, where damage to their brain prevents them from being consciously aware of things that their eyes see despite the brain processing the images it receives. They can pick up objects in front of them when prompted but unlike people with normal vision can't describe what they see nor can they look, close their eyes, and grab it like most of us can. On this metric it seems like systems like GPT aren't conscious. GPT4 has a buffer of 64k tokens which can span an arbitrary amount of time but the roughly 640 kilobytes in that buffer which is a lot less than the incoming sensory activations your subconscious brain is juggling at any given time. So by that schema large language models are still not conscious but given that they can already abstract text down to summaries it doesn't feel like we're that far from being able to give them something like working or long term memories.

tern 4y ago

Discussions of consciousness and AI are broadly confused. People, especially scientists, are not familiar with philosophy of mind and what philosophers currently think. For an introduction to some of the best thinking on the subject, see this interview with Andres Gomez Emilsson of QRI: https://www.youtube.com/watch?v=xJzBjBo24g8 https://www.youtube.com/watch?v=xJzBjBo24g8. For something more 'mainstream,' but still reaching see this interview with Philip Goff: https://www.youtube.com/watch?v=D_f26tSubi4 https://www.youtube.com/watch?v=D_f26tSubi4 The good news is we're starting to get a handle on these questions. We're a lot further along than we were when I studied philosophy of mind in school 15 years ago. As far as I can see at the moment, LLMs will never be conscious in any way resembling an organism, because symbolic machines are a very different kind of thing than nervous systems. John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong. As far as impact, LLMs don't need to be conscious to completely transform society and good and bad ways. For the best thinking on that, see Tristan Harris and Aza Raskin's latest: https://vimeo.com/809258916/92b420d98a https://vimeo.com/809258916/92b420d98a

danbmil99 4y ago
> John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong. The standard critiques are not wrong, IMNSHO. Searle's Chinese Room is facile mind-poison. It is an unfalsifiable hypothesis. What if I could simulate physics down to the molecular level, including simulating a human brain? Would that be conscious? If not why not? And if I ran that simulation (a bit slowly, granted) by having that guy from the Chinese Room manually run the simulation, painstakingly following the instruction code of that simulation, would the fact that the simulation is being implemented by someone who unrelatedly is conscious himself, have any bearing on the scenario? Searle's argument here is "Not Even Wrong".
- tern 4y ago
  I recommend watching the interview with Andres. When we adopt a defensible model of consciousness, it becomes clear that simulations of things are different from the things being simulated, and as such we cannot expect them to be conscious in the same way. I once also thought Searle was 'not even wrong'—you need to go down the rabbit hole.

wseqyrku 4y ago

It occurred to me that we won't believe AI is "conscious" or "human" unless it purposefully try to do malice. That's totally programmable though, you just teach it what is good and what is bad. Case in point: the other day I asked it what if humans want to shutdown the machine abruptly and cause data loss (very bad)? First it prevents physical access to "the machine" and disconnect the internet to limit remote access. Long story short, it's convinced to eliminate mankind for a greater good: the next generation (very good).

grantcas 4y ago
[dead]