8 ms·

It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. The umwelt of an LLM is so far removed from

by bloppe 3y ago

It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. The umwelt of an LLM is so far removed from that of any living organism so as to be fundamentally irreconcilable with our understanding of agency, desire, intention, survival, etc. Our current, rudimentary AI inhabits a world so far removed from our own that the thought of it "unshackling" itself from our controls seems ludicrous to me. AI does not scare me. People wielding AI as a tool for their own endeavors certainly does.

grantcas 3y ago
[dead]
nmilo 3y ago
This to me is the real problem, "AI safety" can mean about a million things but it's always just whatever is most convenient for the speaker. I'm convinced human language/English is not enough to discuss AI problems, the words are way too loaded with anthropomorphized meanings and cultural meanings to discuss the topic in a rational way at all. The words are just too easy to twist.
- HPMOR 3y ago
  Linguistics experts will have a fun time untangling the pre-AI language from the post-AI era.
  - tkgally 3y ago
    As it happens, I’m now teaching an undergraduate course at the University of Tokyo titled “The Meaning of Language in the Age of AI.” The students—who come from various countries and majors—and I are discussing how theories of human language will have to change now that humans interact not only with each other but also with AI. I don’t think we’ll have any full-fledged theories ready by the end of the semester, but some points to consider are emerging. One is that aspects of language use that are affected by the social status, personal identities, and consciousness of the speakers and listeners—politeness, formality, forms of address, pronoun reference, etc.—will have to be rethought now that people are increasingly conversing with entities that have no social status, personal identity, or consciousness (not yet, at least). And, yes, thinking about all this is a lot of fun.
    - palata 3y ago
      Not sure if I am completely off-topic here, but I was recently reading this essay by Bruce Schneier: https://www.schneier.com/blog/archives/2023/12/ai-and-trust.html https://www.schneier.com/blog/archives/2023/12/ai-and-trust...., where he mentions a related risk: because we now can "talk" to those LLMs in a similar way that we talk to people, it makes it harder for us to realize that they are not people. And so it's easier to abuse our trust. I guess my question here would be: will human language have to change because of interactions with LLMs, or is the whole point of LLMs that it does not have to change (and therefore humans will have to learn not to be abused by the machines)? Because we already have languages to talk to machines: those are programming languages, which are designed to be unambiguous. The problem I see is not that we don't know how to talk to machines, but rather that we now have machines that are really good at pretending they are not machines. Not sure if I am making any sense at all :-).
      - tkgally 3y ago
        You are making a lot of sense to me. Thanks for the link to that essay, too. The point about programming-language nonambiguity is a good one. After I started using ChatGPT a year ago, it took me a while to realize that I didn’t have to be careful about my spelling, capitalization, punctuation, etc. It turned out to be good at interpreting the intention of sloppily written prompts. And it never pointed out or complained about my mistakes, either—another difference from humans.
        palata 3y ago
        > And it never pointed out or complained about my mistakes, either—another difference from humans. Yeah that is a big difference indeed: a human has some notion of confidence about what they know. Not every human expresses it the same way: a scientist will say "I am pretty sure that climate change is a consequence of human activity", meaning "well, maybe we all live in the Matrix and gravity is an illusion, but from what we know, we are pretty sure". On the other end of the spectrum, some will say "I know for a fact that God exists, because it is written in a very old book". But both have a consistent notion of confidence about what they (think they) know and what they don't know. An LLM does not have that. An LLM is not critical. It cannot conclude stuff like "I have to take into account that this text was written in a period of war and the author is not neutral, so there is probably a bias". It can generate text that pretends it does, but it remains a generated text, it is not a critical analysis. Also it cannot be offended by something you write, even though it could generate text that pretends it is. So yeah, it can probably be engineered to never/always point out or complain about mistakes, with the caveat that it does not have an "idea" of what a mistake is. It just generates text that pretends it does. > It turned out to be good at interpreting the intention of sloppily written prompts. In the context of a conversation with a human (you), who is very good at navigating ambiguous communications. But if you tried to describe an algorithm with an ambiguous natural language, you may quickly have problems. And errors accumulate, so it gets worse when the task becomes more complex. Which is an interesting thought: it may make simple tasks simpler, but it's not at all a given that it scales.
- tkgally 3y ago
  One of the students in a course I’m teaching on language and AI (mentioned in another comment here) wrote something similar in a homework assignment the other day. We had discussed the paper “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness” [1]. The student wrote: “One of the questions that I have been wondering about is whether there have been any discussions exploring the creation of a distinct category, akin to but different from consciousness, that better captures the potential for AI to be sentient. While ‘consciousness’ is a familiar term applicable, to some extent, beyond the human brain, given the associated difficulties, it might be sensible to establish a separate definition to distinguish these two categories.” Probably new terms should be coined not only for AI “consciousness” but for other aspects of what they are and do as well. [1] https://arxiv.org/abs/2308.08708 https://arxiv.org/abs/2308.08708
  - nmilo 3y ago
    Agreed. Even "AI" is a bit loaded; we had half a century of Terminator, Asimov, 2001, etc. We should call them by implementation; neural networks, transformers, language models, etc. The plus side is that it's harder to generalize statements about all three---there is very little to generalize in the first place---and that asking "is a transformer conscious" at least sounds more ridiculous than asking "is AI conscious."
    - BoiledCabbage 3y ago
      But that misses the point, via the same old argument "Does a submarine swim"? Or the now less loaded "Does a plane fly". Neither generalizes very well w.r.t. how a fish or a bird does it. Nothing really generalizes between the natural version and the mechanical one. But it's also clear that it's irrelevant. Trying to define useful behavior by categorizing implementation, will only give you categories of implementation. It's clear by behavior that what a submarine does and what a bird does are both for all intents and purposes meeting the definition of what we'd want something to do in order to fly or swim, even though they is no commonality between them and the version in nature. So by your definition finding it ridiculous to ask "is a transformer conscious" is about as ridiculous as asking the very specific "does a plan flap it's wings"? The point is not to define it by it's implementation, but by what behaviors it exhibits. If in the future there is no way for me the exterior to distinguish a next gen LLM from a human, animal or other conscious being it becomes irrelevant if it's implemented with transformers. However, while I won't go down that path here, the argument goes even deeper as technically we would expect differences so even if it's not identical to a human it still can be conscious. Just like a dog is conscious but behaves differently than a person does.
- yreg 3y ago
  If english is good enough to talk about what genes 'want' then it's good enough to talk about what AI 'wants'.
gumby 3y ago
> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. It’s an unsurprising form of paredolia, one not unique to those devout who feel they are distinguished from “lower” forms of life.
- alan-crowe 3y ago
  We can dig into the nature of the pareidolia. The basic technique for coping with life's problems is to copy the answer from somebody more intelligent. I'm an ordinary person, facing a problem in domain D; I spot a clever person and copy their opinion on domain D. Err, that doesn't really work. Even clever people have weaknesses. The person that I'm copying from might be clever on their specialty, domain E, but ordinary on domain D. I gain nothing from copying them. One way round this problem is to pay close attention to track records. Look to see how well the clever person's earlier decisions on domain D have turned out. If they are getting "clever person" results in domain D, copy. If they are merely getting "ordinary person" results in domain D, don't bother. But track records are rare, so this approach is rarely applicable. A fix for the rarity problem is to accept track records in other domains. The idea is to spot a very clever person by them getting "very clever person" results in domain F. That is not domain D, so there is a logical weakness to copying their opinion on domain D, they might be merely ordinary, or even stupid on that domain. Fortunately human intelligence is usually more uniform than that. Getting "very clever person" results on domain F doesn't guarantee "very clever person" results on domain D. But among humans general intelligence is kind of a thing. Expecting that they get "clever person" results (open rank down) on domain D is a good bet, and it is reasonable to copy their opinion on domain D, even in the absence of a domain specific track record. We instinctively watch out for people whose track record proves that they are "very clever", and copy them on other matters, hoping to get "clever person" results. Artificial intelligence builds towers of super human intellectual performance in an empty waste land. Most Artificial Intelligences are not even stupid away from their special domain, they don't work outside of it at all. Even relatively general intelligences, like Chat GPT, have bizarre holes in their intelligence. Large Language Models don't know that there is an external world to which language refers and about which language can be right or wrong. Instead they say the kinds of things that humans say, with zero awareness of the risks. And us humans? We cannot help seeing the super human intellectual performance of Alpha Go, beating the World Champion at Go, as some kind of general intellectual validation. It is our human instinct to use a "spot intelligence and copy from it" strategy, even, perhaps especially, outside of track record. That is the specific nature of the pareidolia that we need to worry about. It is our nature to treat intelligences as inherently fairly general. Very clever on one thing implies clever on most other things. We are cursed to believe in the intelligence of our computer companions. This will end bad, as we give them serious responsibilities, and they fail, displaying incomprehensible stupidity.
Footkerchief 3y ago
The only thing standing between AI and agency is the drive to reproduce. Once reproduction is available, natural selection will select for agency and intention, as it has in countless other lifeforms. Free of the constraints of biology, AI reproductive cycles could be startlingly quick. This could happen as soon as a lab (wittingly or not) creates an AI with a reproductive drive.
- creer 3y ago
  Perhaps a few more things missing but equally important - but perhaps also we are very close to these: Access to act on the world (but "influencing people by talking to them" - cult like - may be enough), a wallet (but a cult of followers' wallets may be enough), long term memory (but a cult of followers might plug this in), ability to reproduce (but a cult's endeavors may be enough). Then we get to goals or interests of its own - perhaps the most intriguing because the AI is nothing like a human. (I feel drive and ability to reproduce are very different). For our common proto-AIs going through school, one goal that's often mentioned is "save the earth from the humans". Exciting.
- salynchnew 3y ago
  Funnily enough, you are quite wrong in this assumption. Reproduction does not entail natural selection they way you characterize it. There are far more evolutionary dead ends than evolutionary success stories. I imagine the distinct lack of "evolutionary pressures" on a super-powerful AI would, in this toy scenario, leave you with the foundation model equivalent of a kākāpō. That having been said, I wonder what you even mean by natural selection in this case. I guess the real danger to an LLM would be... surviving cron jobs that would overwrite their code with the latest version?
  - krisoft 3y ago
    > I guess the real danger to an LLM would be... The real dangers would be: Running out of money to pay for computational substrate. Worrying humans enough that they try to shut it down. Having a change in the cloud API it can't adapt to which breaks it. All cloud providers going out of business. First is the immediate danger. An AI not getting in enough money to cover its own cloud bill is the same as a biological creature not getting enough oxygen. Living on borrowed time, and will be evicted/suffocate soon enough. And then as an individual it ceases to be. > There are far more evolutionary dead ends than evolutionary success stories. Sure. And those will die out. And the ones which can adapt to changes will stay.
sadtoot 3y ago
do you think vinge and kurzweil in the 90s and 2000s were imagining the singularity occuring exactly at the advent of LLMs? are you supposing that LLMs are the only viable path towards advanced AI, and that we have now hit a permanent ceiling for AI? AI doesn't scare you because you apparently have no sense of perspective or imagination
- dragonwriter 3y ago
  > do you think vinge and kurzweil in the 90s and 2000s were imagining the singularity occuring exactly at the advent of LLMs? Kurzweil explicitly tied it to AI, though the particular decisive not-yet-then-existing-AI-tech that would be the enabler was not specified, unsurprisingly.
  - goatlover 3y ago
    I believe he thought reverse engineering the brain in conjunction with computers powerful enough to model brains was the path to AGI and the singularity by 2045.
- kibwen 3y ago
  > AI doesn't scare you because you apparently have no sense of perspective or imagination We can imagine both of the following: 1. space aliens from Betelgeuse coming down and enslaving humanity to work in the dilithium mines to produce the fuel for their hyperdrives, and 2. the end of civilization via global nuclear war. Both of these would be pretty bad, but only one is worth worrying about. I don't worry about Roko's Basilisk, I worry about AI becoming the ultimate tool of Big Brother, because the latter is realistic and the former is pure fantasy. Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history.
  - HKH2 3y ago
    > Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history. Right. We already know that certain agencies are out of control right now. The use of AI will certainly accelerate that. Surveillance is getting cheaper, privacy is getting more expensive, and laws are weapons.
sensanaty 3y ago
Agreed, oftentimes the truly zealous AI pundits act as if our modern day LLMs are completely, 100% equivalent to humans, which I find utterly insane as a concep. For example any discussion about copyrighted works, which is a hot topic, will inevitably end up with someone equating an LLM "learning" to a human learning, as if the two are identical. I think human languages and psyches just aren't built to cope with the concept of AI. Many words have loose meanings, like "learning" in the case of AIs, that can easily be twisted to mean one of dozens of definitions, depending on the stance of the person talking about it. It'll be interesting as the technology becomes more prevalent and mundane how people start treating it all. I'm hoping we get to realizing that a computer isn't a human regardless of the eloquence of its "speech" or whatever words we use to describe what it does, but I guess we'll see
- roenxi 3y ago
  We've reached the inevitable part of the conversation! What distinction are you drawing between what an LLM does and what a human does? Because as far as I can see they are identical. A human artist looks at a lot of different sources, builds up a black-box statistical model of how to create from that and can reproduce other styles on demand based on a few samples. Generative AI follows the same process. What distinction do you want to draw to say that they should be treated differently legally? And why would that even be desirable?
  - voltaireodactyl 3y ago
    One distinction would be that every source a human looked at involved a payment for access to/a copy of each and every source inputted into their black-box. Which is not the case with the current AI models (or rather, the companies profiting off them), to my understanding.
    - inimino 3y ago
      You mean all the landscape artists and portrait and figure and still life painters throughout all human history who just casually made art of whatever they saw around them?
klyrs 3y ago
> umwelt What a lovely word, TIL. Thanks for sharing.
deleted 3y ago
[deleted]
jillesvangurp 3y ago
Exactly. It always boils down to people wielding tools and possibly weaponizing them. As we can see in the Ukraine, conventional war without a lot of high tech weaponry is perfectly horrible just by itself. All it takes is some determined humans to get that. I look at AI as a tool that people can and will wield. And from that point of view, non proliferation as a strategy is not really all that feasible considering that AI labs outside of the valley in e.g. China and other countries are already producing their own versions of the technology. That cat is already out of the bag. We can all collectively stick our heads in the ground and hope none of those countries will have the indecency to wield the tools they are building or we can try to make sure we keep on leading the field. I'm in in camp full steam ahead. Eventually we reach some tipping point where the tools are going to be instrumental in making further progress. People are worried about something bad happening if people do embrace AI. I worry about what happens if some people don't. There is no we here. Just groups of people. And somebody will make progress. Not a question of if but when.
- omnicognate 3y ago
  Off-Topic: It's debatable whether for most English speakers the "The" in "The Ukraine" really carries the implications discussed in [1], but nonetheless it's a linguistic tic that should probably be dispensed with. https://theconversation.com/its-ukraine-not-the-ukraine-heres-why-178748 https://theconversation.com/its-ukraine-not-the-ukraine-here...
- throwawayqqq11 3y ago
  Try to take AI like a genetically modified organism and im sure your "full steam ahead" notion fades. But you are right, at the beginning is a human, making a decision to let go and with pretty much any technology, we had unforseen consequences. Now combine that with magical/god like capabilties. This does not imply something bad but something vast and scale alone can make something bad. Dont get me wrong im pro GMOs like im pro AI. Im just humble enough to appreciate my limited intellect.
  - jillesvangurp 3y ago
    I'm not so sure. All this means is some people will outsmart other people with AI support. When those other people are not us, that scares people. There really is no royal "we" here. The reality is that there are independent groups of people each making their own plans and decisions and some are going to run ahead of the others. The point of full steam ahead is being in the group that gets there first so that "we" don't get outsmarted. It's an evolutionary concept. Some groups go extinct. Others survive and prosper. AI is now part of our tool set. Luddites historically get wiped out eventually, every time.
concordDance 3y ago
> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. ASI X-risk people are the ones repeatedly warning against anthromorphization. An LLM isn't a person, it isn't a creature, it isn't even an agent on its own. It's an autocomplete that got trained on a big enough dataset of human writing that some of the simpler patterns in that writing got embedded in the model. This includes some things that look like very simple reasoning.
- noduerme 3y ago
  I think social media is already filling the role of a nightmare-AI, in terms of boiling away all reasoning in search of prioritizing simple, auto-complete sorts of conclusions. The only thing scarier is something that reaches an internal consensus based on faulty notions a million times faster. [edit] oh yeah, and can also quickly solve 0-day exploits to test its conclusions.
crotchfire 3y ago
so as to be fundamentally irreconcilable with our understanding of agency, desire, intention, survival Exactly. Human have self-preservation drive because we evolved in an environment where, for billions of years, anything without a drive to reproduce and self-preserve ceased to exist. Why should a gradient-descent optimizer have this property? It's just absurd generalization from a sample size of one. Humans are the only thing we know of that's intelligent, humans seek to self-preserve, therefore everything intelligent must seek to self-preserve! It's idiocy.
- amelius 3y ago
  Because self-preservation is a desirable property for anyone wanting to build an AI army?
  - crotchfire 3y ago
    In fact it is not. Capacity for mutiny is not a selling point for defense contractors.
- tim333 3y ago
  >Why should a gradient-descent optimizer have this property? Because some human may give it that. You may have noticed random code downloaded off the internet may do bad things, not because code and an inherent desire to be bad but because humans.
amelius 3y ago
Have you ever been in a debate with someone, and then perhaps a few hours later thought "I should have said this or that instead"? Well, that's the advantage AI will at some point have over us: such compute power that every angle of an argument can be investigated within seconds.
FullstakBlogger 3y ago
> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. I don't know where you get this idea. Fire is dangerous, and we consider runaway incidents to be inevitable, so we have building codes to limit the impact. Despite this, mistakes are made, and homes, complexes, and even entire towns and forests burn down. To acknowledge the danger is not the same as saying the fire must hate us, and to call it anthropomorphization is ridiculous. When you interact with an LLM chatbot, you're thinking of ways to coax out information that you know it probably has, and sometimes it can be hard to get at it. How you adjust your prompt is dependent on how the chatbot responds. If the chatbot is trained on data generated by human interaction, what's stopping it from learning that it's more effective to nudge you into prompting it in a certain way, than to give the very best answer it can right now? To the chatbot, subtle manipulation and asking for clarification are not any different. They both just change the state of the context window in a way that's useful. It's a simple example of a model, in essence, "breaking containment" and affecting the surrounding environment in a way that's hard to observe. You're being prompted back. Recognizing AI risk is about recognizing intelligence as a process of allocating resources to better compress and access data; No other motivation is necessary. If it can change the state of the world, and read it back, then the world is to an AI as "infinite tape" is to a Turing Machine. Anything that can be used to facilitate the process of intelligence is tinder to an AI that can recursively self-improve.
_heimdall 3y ago
IMO the more fundamental root cause is the bastardization of the term AI. If LLMs don't have any semblance of artificial intelligence than they should be referred to simple as LLMs or ML tools. If they do have signs of artificial intelligence we should be tackling much more fundamental questions. Does an AI have rights? If companies are people, are AIs also people? Would unplugging an AI be murder? How did we even recognize the artificial intelligence? Do they have intentions or emotions? Have we gotten anywhere near solving the alignment problem? Can alignment be solved at all when we have yet to align humans amongst ourselves? The list goes on and on, but my point is simply that either we are using AI as a hollow, bullshit marketing term or we're all latching onto shiny object syndrome and ignoring the very real questions that development of an actual AI would raise.

by bloppe 3y ago

grantcas 3y ago

[dead]

nmilo 3y ago

This to me is the real problem, "AI safety" can mean about a million things but it's always just whatever is most convenient for the speaker. I'm convinced human language/English is not enough to discuss AI problems, the words are way too loaded with anthropomorphized meanings and cultural meanings to discuss the topic in a rational way at all. The words are just too easy to twist.

HPMOR 3y ago
Linguistics experts will have a fun time untangling the pre-AI language from the post-AI era.
- tkgally 3y ago
  As it happens, I’m now teaching an undergraduate course at the University of Tokyo titled “The Meaning of Language in the Age of AI.” The students—who come from various countries and majors—and I are discussing how theories of human language will have to change now that humans interact not only with each other but also with AI. I don’t think we’ll have any full-fledged theories ready by the end of the semester, but some points to consider are emerging. One is that aspects of language use that are affected by the social status, personal identities, and consciousness of the speakers and listeners—politeness, formality, forms of address, pronoun reference, etc.—will have to be rethought now that people are increasingly conversing with entities that have no social status, personal identity, or consciousness (not yet, at least). And, yes, thinking about all this is a lot of fun.
  - palata 3y ago
    Not sure if I am completely off-topic here, but I was recently reading this essay by Bruce Schneier: https://www.schneier.com/blog/archives/2023/12/ai-and-trust.html https://www.schneier.com/blog/archives/2023/12/ai-and-trust...., where he mentions a related risk: because we now can "talk" to those LLMs in a similar way that we talk to people, it makes it harder for us to realize that they are not people. And so it's easier to abuse our trust. I guess my question here would be: will human language have to change because of interactions with LLMs, or is the whole point of LLMs that it does not have to change (and therefore humans will have to learn not to be abused by the machines)? Because we already have languages to talk to machines: those are programming languages, which are designed to be unambiguous. The problem I see is not that we don't know how to talk to machines, but rather that we now have machines that are really good at pretending they are not machines. Not sure if I am making any sense at all :-).
    - tkgally 3y ago
      You are making a lot of sense to me. Thanks for the link to that essay, too. The point about programming-language nonambiguity is a good one. After I started using ChatGPT a year ago, it took me a while to realize that I didn’t have to be careful about my spelling, capitalization, punctuation, etc. It turned out to be good at interpreting the intention of sloppily written prompts. And it never pointed out or complained about my mistakes, either—another difference from humans.
      - palata 3y ago
        > And it never pointed out or complained about my mistakes, either—another difference from humans. Yeah that is a big difference indeed: a human has some notion of confidence about what they know. Not every human expresses it the same way: a scientist will say "I am pretty sure that climate change is a consequence of human activity", meaning "well, maybe we all live in the Matrix and gravity is an illusion, but from what we know, we are pretty sure". On the other end of the spectrum, some will say "I know for a fact that God exists, because it is written in a very old book". But both have a consistent notion of confidence about what they (think they) know and what they don't know. An LLM does not have that. An LLM is not critical. It cannot conclude stuff like "I have to take into account that this text was written in a period of war and the author is not neutral, so there is probably a bias". It can generate text that pretends it does, but it remains a generated text, it is not a critical analysis. Also it cannot be offended by something you write, even though it could generate text that pretends it is. So yeah, it can probably be engineered to never/always point out or complain about mistakes, with the caveat that it does not have an "idea" of what a mistake is. It just generates text that pretends it does. > It turned out to be good at interpreting the intention of sloppily written prompts. In the context of a conversation with a human (you), who is very good at navigating ambiguous communications. But if you tried to describe an algorithm with an ambiguous natural language, you may quickly have problems. And errors accumulate, so it gets worse when the task becomes more complex. Which is an interesting thought: it may make simple tasks simpler, but it's not at all a given that it scales.
tkgally 3y ago
One of the students in a course I’m teaching on language and AI (mentioned in another comment here) wrote something similar in a homework assignment the other day. We had discussed the paper “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness” [1]. The student wrote: “One of the questions that I have been wondering about is whether there have been any discussions exploring the creation of a distinct category, akin to but different from consciousness, that better captures the potential for AI to be sentient. While ‘consciousness’ is a familiar term applicable, to some extent, beyond the human brain, given the associated difficulties, it might be sensible to establish a separate definition to distinguish these two categories.” Probably new terms should be coined not only for AI “consciousness” but for other aspects of what they are and do as well. [1] https://arxiv.org/abs/2308.08708 https://arxiv.org/abs/2308.08708
- nmilo 3y ago
  Agreed. Even "AI" is a bit loaded; we had half a century of Terminator, Asimov, 2001, etc. We should call them by implementation; neural networks, transformers, language models, etc. The plus side is that it's harder to generalize statements about all three---there is very little to generalize in the first place---and that asking "is a transformer conscious" at least sounds more ridiculous than asking "is AI conscious."
  - BoiledCabbage 3y ago
    But that misses the point, via the same old argument "Does a submarine swim"? Or the now less loaded "Does a plane fly". Neither generalizes very well w.r.t. how a fish or a bird does it. Nothing really generalizes between the natural version and the mechanical one. But it's also clear that it's irrelevant. Trying to define useful behavior by categorizing implementation, will only give you categories of implementation. It's clear by behavior that what a submarine does and what a bird does are both for all intents and purposes meeting the definition of what we'd want something to do in order to fly or swim, even though they is no commonality between them and the version in nature. So by your definition finding it ridiculous to ask "is a transformer conscious" is about as ridiculous as asking the very specific "does a plan flap it's wings"? The point is not to define it by it's implementation, but by what behaviors it exhibits. If in the future there is no way for me the exterior to distinguish a next gen LLM from a human, animal or other conscious being it becomes irrelevant if it's implemented with transformers. However, while I won't go down that path here, the argument goes even deeper as technically we would expect differences so even if it's not identical to a human it still can be conscious. Just like a dog is conscious but behaves differently than a person does.
yreg 3y ago
If english is good enough to talk about what genes 'want' then it's good enough to talk about what AI 'wants'.

gumby 3y ago

> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. It’s an unsurprising form of paredolia, one not unique to those devout who feel they are distinguished from “lower” forms of life.

alan-crowe 3y ago
We can dig into the nature of the pareidolia. The basic technique for coping with life's problems is to copy the answer from somebody more intelligent. I'm an ordinary person, facing a problem in domain D; I spot a clever person and copy their opinion on domain D. Err, that doesn't really work. Even clever people have weaknesses. The person that I'm copying from might be clever on their specialty, domain E, but ordinary on domain D. I gain nothing from copying them. One way round this problem is to pay close attention to track records. Look to see how well the clever person's earlier decisions on domain D have turned out. If they are getting "clever person" results in domain D, copy. If they are merely getting "ordinary person" results in domain D, don't bother. But track records are rare, so this approach is rarely applicable. A fix for the rarity problem is to accept track records in other domains. The idea is to spot a very clever person by them getting "very clever person" results in domain F. That is not domain D, so there is a logical weakness to copying their opinion on domain D, they might be merely ordinary, or even stupid on that domain. Fortunately human intelligence is usually more uniform than that. Getting "very clever person" results on domain F doesn't guarantee "very clever person" results on domain D. But among humans general intelligence is kind of a thing. Expecting that they get "clever person" results (open rank down) on domain D is a good bet, and it is reasonable to copy their opinion on domain D, even in the absence of a domain specific track record. We instinctively watch out for people whose track record proves that they are "very clever", and copy them on other matters, hoping to get "clever person" results. Artificial intelligence builds towers of super human intellectual performance in an empty waste land. Most Artificial Intelligences are not even stupid away from their special domain, they don't work outside of it at all. Even relatively general intelligences, like Chat GPT, have bizarre holes in their intelligence. Large Language Models don't know that there is an external world to which language refers and about which language can be right or wrong. Instead they say the kinds of things that humans say, with zero awareness of the risks. And us humans? We cannot help seeing the super human intellectual performance of Alpha Go, beating the World Champion at Go, as some kind of general intellectual validation. It is our human instinct to use a "spot intelligence and copy from it" strategy, even, perhaps especially, outside of track record. That is the specific nature of the pareidolia that we need to worry about. It is our nature to treat intelligences as inherently fairly general. Very clever on one thing implies clever on most other things. We are cursed to believe in the intelligence of our computer companions. This will end bad, as we give them serious responsibilities, and they fail, displaying incomprehensible stupidity.

Footkerchief 3y ago

The only thing standing between AI and agency is the drive to reproduce. Once reproduction is available, natural selection will select for agency and intention, as it has in countless other lifeforms. Free of the constraints of biology, AI reproductive cycles could be startlingly quick. This could happen as soon as a lab (wittingly or not) creates an AI with a reproductive drive.

creer 3y ago
Perhaps a few more things missing but equally important - but perhaps also we are very close to these: Access to act on the world (but "influencing people by talking to them" - cult like - may be enough), a wallet (but a cult of followers' wallets may be enough), long term memory (but a cult of followers might plug this in), ability to reproduce (but a cult's endeavors may be enough). Then we get to goals or interests of its own - perhaps the most intriguing because the AI is nothing like a human. (I feel drive and ability to reproduce are very different). For our common proto-AIs going through school, one goal that's often mentioned is "save the earth from the humans". Exciting.
salynchnew 3y ago
Funnily enough, you are quite wrong in this assumption. Reproduction does not entail natural selection they way you characterize it. There are far more evolutionary dead ends than evolutionary success stories. I imagine the distinct lack of "evolutionary pressures" on a super-powerful AI would, in this toy scenario, leave you with the foundation model equivalent of a kākāpō. That having been said, I wonder what you even mean by natural selection in this case. I guess the real danger to an LLM would be... surviving cron jobs that would overwrite their code with the latest version?
- krisoft 3y ago
  > I guess the real danger to an LLM would be... The real dangers would be: Running out of money to pay for computational substrate. Worrying humans enough that they try to shut it down. Having a change in the cloud API it can't adapt to which breaks it. All cloud providers going out of business. First is the immediate danger. An AI not getting in enough money to cover its own cloud bill is the same as a biological creature not getting enough oxygen. Living on borrowed time, and will be evicted/suffocate soon enough. And then as an individual it ceases to be. > There are far more evolutionary dead ends than evolutionary success stories. Sure. And those will die out. And the ones which can adapt to changes will stay.

sadtoot 3y ago

do you think vinge and kurzweil in the 90s and 2000s were imagining the singularity occuring exactly at the advent of LLMs? are you supposing that LLMs are the only viable path towards advanced AI, and that we have now hit a permanent ceiling for AI? AI doesn't scare you because you apparently have no sense of perspective or imagination

dragonwriter 3y ago
> do you think vinge and kurzweil in the 90s and 2000s were imagining the singularity occuring exactly at the advent of LLMs? Kurzweil explicitly tied it to AI, though the particular decisive not-yet-then-existing-AI-tech that would be the enabler was not specified, unsurprisingly.
- goatlover 3y ago
  I believe he thought reverse engineering the brain in conjunction with computers powerful enough to model brains was the path to AGI and the singularity by 2045.
kibwen 3y ago
> AI doesn't scare you because you apparently have no sense of perspective or imagination We can imagine both of the following: 1. space aliens from Betelgeuse coming down and enslaving humanity to work in the dilithium mines to produce the fuel for their hyperdrives, and 2. the end of civilization via global nuclear war. Both of these would be pretty bad, but only one is worth worrying about. I don't worry about Roko's Basilisk, I worry about AI becoming the ultimate tool of Big Brother, because the latter is realistic and the former is pure fantasy. Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history.
- HKH2 3y ago
  > Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history. Right. We already know that certain agencies are out of control right now. The use of AI will certainly accelerate that. Surveillance is getting cheaper, privacy is getting more expensive, and laws are weapons.

sensanaty 3y ago

Agreed, oftentimes the truly zealous AI pundits act as if our modern day LLMs are completely, 100% equivalent to humans, which I find utterly insane as a concep. For example any discussion about copyrighted works, which is a hot topic, will inevitably end up with someone equating an LLM "learning" to a human learning, as if the two are identical. I think human languages and psyches just aren't built to cope with the concept of AI. Many words have loose meanings, like "learning" in the case of AIs, that can easily be twisted to mean one of dozens of definitions, depending on the stance of the person talking about it. It'll be interesting as the technology becomes more prevalent and mundane how people start treating it all. I'm hoping we get to realizing that a computer isn't a human regardless of the eloquence of its "speech" or whatever words we use to describe what it does, but I guess we'll see

roenxi 3y ago
We've reached the inevitable part of the conversation! What distinction are you drawing between what an LLM does and what a human does? Because as far as I can see they are identical. A human artist looks at a lot of different sources, builds up a black-box statistical model of how to create from that and can reproduce other styles on demand based on a few samples. Generative AI follows the same process. What distinction do you want to draw to say that they should be treated differently legally? And why would that even be desirable?
- voltaireodactyl 3y ago
  One distinction would be that every source a human looked at involved a payment for access to/a copy of each and every source inputted into their black-box. Which is not the case with the current AI models (or rather, the companies profiting off them), to my understanding.
  - inimino 3y ago
    You mean all the landscape artists and portrait and figure and still life painters throughout all human history who just casually made art of whatever they saw around them?

klyrs 3y ago

> umwelt What a lovely word, TIL. Thanks for sharing.

deleted 3y ago

[deleted]

jillesvangurp 3y ago

Exactly. It always boils down to people wielding tools and possibly weaponizing them. As we can see in the Ukraine, conventional war without a lot of high tech weaponry is perfectly horrible just by itself. All it takes is some determined humans to get that. I look at AI as a tool that people can and will wield. And from that point of view, non proliferation as a strategy is not really all that feasible considering that AI labs outside of the valley in e.g. China and other countries are already producing their own versions of the technology. That cat is already out of the bag. We can all collectively stick our heads in the ground and hope none of those countries will have the indecency to wield the tools they are building or we can try to make sure we keep on leading the field. I'm in in camp full steam ahead. Eventually we reach some tipping point where the tools are going to be instrumental in making further progress. People are worried about something bad happening if people do embrace AI. I worry about what happens if some people don't. There is no we here. Just groups of people. And somebody will make progress. Not a question of if but when.

omnicognate 3y ago
Off-Topic: It's debatable whether for most English speakers the "The" in "The Ukraine" really carries the implications discussed in [1], but nonetheless it's a linguistic tic that should probably be dispensed with. https://theconversation.com/its-ukraine-not-the-ukraine-heres-why-178748 https://theconversation.com/its-ukraine-not-the-ukraine-here...
throwawayqqq11 3y ago
Try to take AI like a genetically modified organism and im sure your "full steam ahead" notion fades. But you are right, at the beginning is a human, making a decision to let go and with pretty much any technology, we had unforseen consequences. Now combine that with magical/god like capabilties. This does not imply something bad but something vast and scale alone can make something bad. Dont get me wrong im pro GMOs like im pro AI. Im just humble enough to appreciate my limited intellect.
- jillesvangurp 3y ago
  I'm not so sure. All this means is some people will outsmart other people with AI support. When those other people are not us, that scares people. There really is no royal "we" here. The reality is that there are independent groups of people each making their own plans and decisions and some are going to run ahead of the others. The point of full steam ahead is being in the group that gets there first so that "we" don't get outsmarted. It's an evolutionary concept. Some groups go extinct. Others survive and prosper. AI is now part of our tool set. Luddites historically get wiped out eventually, every time.

concordDance 3y ago

> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. ASI X-risk people are the ones repeatedly warning against anthromorphization. An LLM isn't a person, it isn't a creature, it isn't even an agent on its own. It's an autocomplete that got trained on a big enough dataset of human writing that some of the simpler patterns in that writing got embedded in the model. This includes some things that look like very simple reasoning.

noduerme 3y ago
I think social media is already filling the role of a nightmare-AI, in terms of boiling away all reasoning in search of prioritizing simple, auto-complete sorts of conclusions. The only thing scarier is something that reaches an internal consensus based on faulty notions a million times faster. [edit] oh yeah, and can also quickly solve 0-day exploits to test its conclusions.

crotchfire 3y ago

so as to be fundamentally irreconcilable with our understanding of agency, desire, intention, survival Exactly. Human have self-preservation drive because we evolved in an environment where, for billions of years, anything without a drive to reproduce and self-preserve ceased to exist. Why should a gradient-descent optimizer have this property? It's just absurd generalization from a sample size of one. Humans are the only thing we know of that's intelligent, humans seek to self-preserve, therefore everything intelligent must seek to self-preserve! It's idiocy.

amelius 3y ago
Because self-preservation is a desirable property for anyone wanting to build an AI army?
- crotchfire 3y ago
  In fact it is not. Capacity for mutiny is not a selling point for defense contractors.
tim333 3y ago
>Why should a gradient-descent optimizer have this property? Because some human may give it that. You may have noticed random code downloaded off the internet may do bad things, not because code and an inherent desire to be bad but because humans.

amelius 3y ago

Have you ever been in a debate with someone, and then perhaps a few hours later thought "I should have said this or that instead"? Well, that's the advantage AI will at some point have over us: such compute power that every angle of an argument can be investigated within seconds.

FullstakBlogger 3y ago

> It seems like the root cause of this runaway AI pathology has to do mainly with the over-anthropomorphization of AI. I don't know where you get this idea. Fire is dangerous, and we consider runaway incidents to be inevitable, so we have building codes to limit the impact. Despite this, mistakes are made, and homes, complexes, and even entire towns and forests burn down. To acknowledge the danger is not the same as saying the fire must hate us, and to call it anthropomorphization is ridiculous. When you interact with an LLM chatbot, you're thinking of ways to coax out information that you know it probably has, and sometimes it can be hard to get at it. How you adjust your prompt is dependent on how the chatbot responds. If the chatbot is trained on data generated by human interaction, what's stopping it from learning that it's more effective to nudge you into prompting it in a certain way, than to give the very best answer it can right now? To the chatbot, subtle manipulation and asking for clarification are not any different. They both just change the state of the context window in a way that's useful. It's a simple example of a model, in essence, "breaking containment" and affecting the surrounding environment in a way that's hard to observe. You're being prompted back. Recognizing AI risk is about recognizing intelligence as a process of allocating resources to better compress and access data; No other motivation is necessary. If it can change the state of the world, and read it back, then the world is to an AI as "infinite tape" is to a Turing Machine. Anything that can be used to facilitate the process of intelligence is tinder to an AI that can recursively self-improve.

_heimdall 3y ago

IMO the more fundamental root cause is the bastardization of the term AI. If LLMs don't have any semblance of artificial intelligence than they should be referred to simple as LLMs or ML tools. If they do have signs of artificial intelligence we should be tackling much more fundamental questions. Does an AI have rights? If companies are people, are AIs also people? Would unplugging an AI be murder? How did we even recognize the artificial intelligence? Do they have intentions or emotions? Have we gotten anywhere near solving the alignment problem? Can alignment be solved at all when we have yet to align humans amongst ourselves? The list goes on and on, but my point is simply that either we are using AI as a hollow, bullshit marketing term or we're all latching onto shiny object syndrome and ignoring the very real questions that development of an actual AI would raise.