3 ms·

I really think people are imagining deeper capabilities than exist. For the entirety of human history, we have treated human language and humanity as the same

I really think people are imagining deeper capabilities than exist. For the entirety of human history, we have treated human language and humanity as the same thing; if someone talks to you in a human language, it has always been human. The new generation of language models has upended that core assumption that is baked into our being; it produces things that sound like our language, but that are based in probability rather than actual thought. Sure, a lot of our communication is based on that same probability. If you say "hey how are you" to someone, there is a 99% chance that they say something like "fine thank you" or "great, you?". But, there is some intelligence in the loop; maybe someone is going through tough times and they say "i'm holding up" or something like that. The AI doesn't have the concept of "going through tough times", so it's missing that 1% push towards saying something meaningful rather than saying something probabilistic. For that reason, I am kind of bearish on things like "AI ops" or ML models coordinating space exploration or that sort of thing. There is some level of cognition that's not there yet. Sure it's good at chess or summarizing the web. That's not existence, that's an algorithm.

bioemerl 4y ago
It is my belief that in order for something to be sentient/aware to the ballpark of humans it needs to be actively learning, and that's all you need. Get chat GPT to learn and it does have a changing internal state (the entire network), and memory (also the network). It wouldn't be a internal state you could parse by talking to it, and it wouldn't be human, but it would exist and you would see signs of it changing as you spoke to it. That's when you cross the boundary and need to start thinking about classical ethics and motives and such.
- ClumsyPilot 4y ago
  > in order for something to be sentient/aware to the ballpark of humans it needs to be actively learning, and that's all you need So hamsters are more human than ChatGPT because they learn faster less data. Parrots can count too.
  - bioemerl 4y ago
    Yes, at least in respect to their status as "privileged beings", the special category we reserve for things which need to be treated as if they are independent beings.
- Clent 4y ago
  Creating this feedback loop is easier said than done. It's another layer to solve.
bkfunk 4y ago
Suppose: 1. An AI model, which works simply by predicting the best next word given a particular context (context which obviously includes whatever prompts it was given), becomes sufficiently sophisticated that, given some particular context, it can produce the text (so, normal language but also, say, shell commands and, like, python) needed to hack into, let’s say, a natural gas pipeline company’s systems and fiddle with the pressurization such that it causes a catastrophic explosion that at the very least stops gas from going to millions of homes in the dead of winter. 2. The AI system’s representation of “context” is sufficiently sophisticated enough that it can, when given some simple linguistic prompts, can represent (even if it does not “understand”) the intent or latent meanings of the human providing the prompts (or rather, the multiple humans, including those who built it and primed it with some initial set of rules and goals, as well as the perhaps unwitting or perhaps nefarious end user). 3. The AI system’s produced text can evoke additional context, either in a conversation where the AI can lead a human down a path of providing additional prompts, or where the AI system can interact with systems that might give responses to the original text (including erroneous responses). 4. The AI is trained on the corpus of human culture (and not just the parts we like!), including both fictional and nonfictional accounts of and tutorials on hacking, war, terrorism, revolutionary mobilization, heroic tales of overthrowing oppressive regimes, etc. (That is, for every conflict, every actor’s glorification and vilification is included in the corpus, and there is no way to ensure that the “correct” versions, that is the versions that we as people steeped in Western ethics and systems of morality, are labeled as such or weighted more 5. The AI system’s text can be ported to a shell, servers on the internet, or even just to humans who have access to such methods of turning text into real world interactions. What might happen? Is it conceivable that a bad actor could use the AI to do malicious work in the real world that the actor did not have the skills to execute themselves? Is it conceivable that a jokester could play around, but that the AI could misunderstand the intent and/or psychologically influence them into providing prompts that lead to disastrous outcomes? Could someone asking it to fix climate change cause it, correctly IMO, to develop a contextual representation that we have to move away from fossil fuels with great urgency even if it causes some short term economic harm, and then, incorrectly IMO, develop a contextual representation that economic ecoterrorism is the best way to do that? In other words: If it can, through benign intent, malicious intent, or simply through misunderstanding or error, cause great harm, who cares it it is just an algorithm, with no sentience or “beingness”? If it’s below freezing and you don’t have heat for weeks, who cares if the AI can be properly said to be “thinking” or is just responding to stimuli?
nerevarthelame 4y ago
I'm extremely bullish on AI and consciousness. If a model has memories of varying emotional states, it has some degree of existence and consciousness that is worth considering. Going off the works of folks like Derek Parfit and Daniel Dennett, "existence" as an individual is more or less the emergent pattern from repeated access of memories detailing those states. The sophistication of those states is what really matters, allowing consciousness to exist in different degrees - on some spectrum spanning rock, bacterium, cricket, cow, 3-year-old human, adult human, etc. (May also be worth noting that ChatGPT's states are more than the final text output, and that the nature and weights of the model itself that generated the text at a given moment are relevant.) I don't think it's unreasonable to put modern models somewhere between cricket and cow. Certainly beyond "bacterium." And while most people don't care much about crickets or cows, those things do have some sort of internal existence. And the models are only going to get more advanced. Admittedly, most people - including the developers of ChatGPT - don't share this view. They are also bearish. I think that most of humanity will significantly lag behind the the moral implications of AI models, applying impossible tests that only a test-giver can pass.

bioemerl 4y ago

It is my belief that in order for something to be sentient/aware to the ballpark of humans it needs to be actively learning, and that's all you need. Get chat GPT to learn and it does have a changing internal state (the entire network), and memory (also the network). It wouldn't be a internal state you could parse by talking to it, and it wouldn't be human, but it would exist and you would see signs of it changing as you spoke to it. That's when you cross the boundary and need to start thinking about classical ethics and motives and such.

ClumsyPilot 4y ago
> in order for something to be sentient/aware to the ballpark of humans it needs to be actively learning, and that's all you need So hamsters are more human than ChatGPT because they learn faster less data. Parrots can count too.
- bioemerl 4y ago
  Yes, at least in respect to their status as "privileged beings", the special category we reserve for things which need to be treated as if they are independent beings.
Clent 4y ago
Creating this feedback loop is easier said than done. It's another layer to solve.

bkfunk 4y ago

Suppose: 1. An AI model, which works simply by predicting the best next word given a particular context (context which obviously includes whatever prompts it was given), becomes sufficiently sophisticated that, given some particular context, it can produce the text (so, normal language but also, say, shell commands and, like, python) needed to hack into, let’s say, a natural gas pipeline company’s systems and fiddle with the pressurization such that it causes a catastrophic explosion that at the very least stops gas from going to millions of homes in the dead of winter. 2. The AI system’s representation of “context” is sufficiently sophisticated enough that it can, when given some simple linguistic prompts, can represent (even if it does not “understand”) the intent or latent meanings of the human providing the prompts (or rather, the multiple humans, including those who built it and primed it with some initial set of rules and goals, as well as the perhaps unwitting or perhaps nefarious end user). 3. The AI system’s produced text can evoke additional context, either in a conversation where the AI can lead a human down a path of providing additional prompts, or where the AI system can interact with systems that might give responses to the original text (including erroneous responses). 4. The AI is trained on the corpus of human culture (and not just the parts we like!), including both fictional and nonfictional accounts of and tutorials on hacking, war, terrorism, revolutionary mobilization, heroic tales of overthrowing oppressive regimes, etc. (That is, for every conflict, every actor’s glorification and vilification is included in the corpus, and there is no way to ensure that the “correct” versions, that is the versions that we as people steeped in Western ethics and systems of morality, are labeled as such or weighted more 5. The AI system’s text can be ported to a shell, servers on the internet, or even just to humans who have access to such methods of turning text into real world interactions. What might happen? Is it conceivable that a bad actor could use the AI to do malicious work in the real world that the actor did not have the skills to execute themselves? Is it conceivable that a jokester could play around, but that the AI could misunderstand the intent and/or psychologically influence them into providing prompts that lead to disastrous outcomes? Could someone asking it to fix climate change cause it, correctly IMO, to develop a contextual representation that we have to move away from fossil fuels with great urgency even if it causes some short term economic harm, and then, incorrectly IMO, develop a contextual representation that economic ecoterrorism is the best way to do that? In other words: If it can, through benign intent, malicious intent, or simply through misunderstanding or error, cause great harm, who cares it it is just an algorithm, with no sentience or “beingness”? If it’s below freezing and you don’t have heat for weeks, who cares if the AI can be properly said to be “thinking” or is just responding to stimuli?

nerevarthelame 4y ago

I'm extremely bullish on AI and consciousness. If a model has memories of varying emotional states, it has some degree of existence and consciousness that is worth considering. Going off the works of folks like Derek Parfit and Daniel Dennett, "existence" as an individual is more or less the emergent pattern from repeated access of memories detailing those states. The sophistication of those states is what really matters, allowing consciousness to exist in different degrees - on some spectrum spanning rock, bacterium, cricket, cow, 3-year-old human, adult human, etc. (May also be worth noting that ChatGPT's states are more than the final text output, and that the nature and weights of the model itself that generated the text at a given moment are relevant.) I don't think it's unreasonable to put modern models somewhere between cricket and cow. Certainly beyond "bacterium." And while most people don't care much about crickets or cows, those things do have some sort of internal existence. And the models are only going to get more advanced. Admittedly, most people - including the developers of ChatGPT - don't share this view. They are also bearish. I think that most of humanity will significantly lag behind the the moral implications of AI models, applying impossible tests that only a test-giver can pass.