6 ms·

Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapl

by Zetice 4y ago

Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." GPT-3.5 is just showing the user some amalgamation of the content its been shown, based on the prompt given it. That's it. There's no intent, there's no maliciousness, it's just generating new word combinations that look like the word combinations its already seen.

groestl 4y ago
You seriously underestimate what a process that's "generating new word combinations that look like the word combinations its already seen" can do, even when air-gapped (which ChatGPT isn't). Right now, at this moment, people are building closed loops based on ChatGPT, or looping in humans which are seriously intellectually underequipped to deal with plausible insane output in that quantity. And those humans operate: machinery, markets, educate or manage other humans etc etc.
- deleted 4y ago
  [deleted]
- chasd00 4y ago
  To me, that's the real danger. ChatGPT convincing a human something is true when it isn't. Machinery is a good example, maybe ChatGPT hallucinates the safety procedure and someone gets hurt by following the response.
unshavedyak 4y ago
It seems a reasonable shorthand, to me at least. Ie if we consider it as a function with input you define, well normally that input in sanitized to prevent hacking/etc. In this case the sanitization process is so broad you could easily summarize it as "aligned with my interests", no? Ie i can't come close to easily enumerating all the seemingly near infinite ways that hooking up this chatbot into my network with code exec permissions might compromise me. Yea it's a dumb autocomplete right now, but it's an exceptionally powerful autocomplete that can write viruses and do all sorts of insane and powerful things. I can give you a function run on my network of `fn foo(i32)` and feel safe about it. However `fn foo(Chatgpt)` is unsafe in ways i not only can't enumerate, i can't even imagine many of them. I get your offense seems to be around the implied intelligence that "aligned with human interests" seems to give it.. but while i think we all agree it's definitely not a Duck right now, when it walks talks and acts like a Duck.. well, are we surprised that our natural language sounds as if it's a Duck?
throwaway4aday 4y ago
Bing Chat shows that it can be connected to other services like web search APIs. It's not too far from "You are Bing, you will perform at least 3 web searches before responding to human input" to "You are Cipher, you will ssh to darkstar and generate the reports by running report-gen.sh adding any required parameters before responding to human input" and some bright bulb gives it enough permissions to run arbitrary scripts. At that point something could go very wrong with a chat interaction if it's capable of writing and executing scripts to perform actions that it thinks will follow the query. It would more often just be locally bad but it could create havoc on other systems as well. I understand that it isn't capable of what we would call agency but it can certainly spit out and execute dangerous code. Then just wait until we get to this https://twitter.com/ai__pub/status/1625552601956909057 https://twitter.com/ai__pub/status/1625552601956909057 and it can generate multi-file programs.
- Zetice 4y ago
  You can shove staplers into wall sockets too (if you're determined enough), but the consequences are on you, not the stapler. It's just not meaningfully different from our current reality, and is therefore not any scarier.
  - haswell 4y ago
    Comparing a system that could theoretically (and very plausibly) carry out cyber attacks with a stapler is problematic at best. Putting a stapler in a wall socket probably electrocutes you. Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.
    - Zetice 4y ago
      If I set my stapler on my mouse such that it clicks a big ol "Hack stuff" button, my stapler could, too, carry out cyber attacks. This is a very pointless line of thinking.
      - 4y ago
aaroninsf 4y ago
Two common cognitive errors to beware of when reasoning about the current state of AI/LLM this exhibits: 1. reasoning by inappropriate/incomplete analogy It is not accurate (predictive) to describe what these systems do as mimicking or regurgitating human output, or, e.g. describing what they do with reference to Markov chains and stochastic outcomes. This is increasingly akin to using the same overly reductionist framing of what humans do, and loses any predictive ability at all. To put a point on it, this line of critique conflates things like agency and self-awareness, with other tiers of symbolic representation and reasoning about the world hitherto reserved to humans. These systems build internal state and function largely in terms of analogical reasoning themselves. This is a lot more that "mimickery" regardless of their lack of common sense. 2. assuming stasis and failure to anticipate non-linearities and punctured equilibrium The last thing these systems are is in their final form. What exists as consumer facing scaled product is naturally generationally behind what is in beta, or alpha; and one of the surprises (including to those of us in the industry...) of these systems is the extent to which behaviors emerge. Whenever you find yourself thinking, "AI is never going to..." you can stop the sentence, because it's if not definitionally false, quite probably false. None of us know where we are in the so-called sigmoid curve, but it is already clear we are far from reaching any natural asymptotes. A pertinent example of this is to go back a year and look at the early output of e.g. Midjourney, and the prompt engineering that it took to produce various images; and compare that with the state of the (public-facing) art today... and to look at the failure of anyone (me included) to predict just how quickly things would advance. Our hands are now off the wheel. We just might have a near-life experience.
- Zetice 4y ago
  1 is false; it is both accurate and predictive to describe what these systems do as mimicking/regurgitating human output. That's exactly what they're doing. 2 is irrelevant; you can doomsay and speculate all day, but if it's detached from reality it's not meaningful as a way of understanding future likely outcomes.
sleepybrett 4y ago
You have to explain this to Normals, and some of those Normals are CEOs of massive companies. So first off stop calling this shit "AI", it's not intelligence it's statistics. If you call it AI some normal will think it's actually thinking and is smarter than he is. They will put this thing behind the wheel of a car or on the trigger of a gun and it will KILL PEOPLE. Sometimes it will kill the right people, in the case of a trigger, but sometimes it will tragically kill the wrong people for reasons that cannot be fully explained. Who is on the hook for that? It's not -obviously- malicious when it kills the wrong person, but I gotta say that if one shoots me when I'm walking down the road minding my own business it's gonna look pretty fucking malicious to me.
blurbleblurble 4y ago
Depends on if you view the stapler as separate from everything the stapler makes possible, and from everything that makes the stapler possible. Of course the stapler has no independent will, but it channels and augments the will of its designers, buyers and users, and that cannot be stripped from the stapler even if it's not contained within the stapler alone "It" is not just the instance of GPT/bing running at any given moment. "It" is inseparable from the relationships, people and processes that have created it and continue to create it. That is where its intent lies, and its beingness. In carefully cultivated selections of our collective intent. Selected according to the schemes of those who directed its creation. This is just another organ of the industrial creature that made it possible, but it's one that presents a dynamic, fluent, malleable, probabilistic interface, and which has a potential to actualize the intent of whatever wields it in still unknown ways.
- Zetice 4y ago
  No, what? GPT is, very roughly, a set of training data plus a way of associating that data together to answer prompts. It's not "relationships, people, and processes", it's not "our collective intent"; what the hell are you talking about?
  - bigtex88 4y ago
    You seem to have an extreme arrogance surrounding your ability to understand what these programs are doing at a base level. Can you explain further your ability to understand this? What gives you such grand confidence to say these sorts of things?
    - daedalus_f 4y ago
      Not the parent poster. The vast number of commenters in this thread seem to assume that these LLMs are close to, if not actually, general AIs. It’s quite refreshing to see comments challenging the hype. Don’t you think the burden of proof lies with those that think this is something more than a just a dumb statistical model?
haswell 4y ago
First, I agree that we’re currently discussing a sophisticated algorithm that predicts words (though I’m interested and curious about some of the seemingly emergent behaviors discussed in recent papers). But what is factually true is not the only thing that matters here. What people believe is also at issue. If an AI gives someone advice, and that advice turns out to be catastrophically harmful, and the person takes the advice because they believe the AI is intelligent, it doesn’t really matter that it’s not. Alignment with human values may involve exploring ways to make the predictions safer in the short term. Long term towards AGI, alignment with human values becomes more literal and increasingly important. But the time to start tackling that problem is now, and at every step on the journey.
theptip 4y ago
Sorry for the bluntness, but this is harmfully ignorant. "Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will act in our interests. It doesn't imply any sense of personhood/selfhood/consciousness. If you think that Bing is equally autonomous as a stapler, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people (that's not hyperbole, I mean that literally, indeed full extinction of humanity is a plausible outcome too). A stapler is understood mechanistically, it's trivially transparent what's going on when you use one, and the only way harm can result is if you do something stupid with it. You cannot for a second defend the proposition that a LLM is equally transparent, or that harm will only arise if an LLM is "used wrong". I think you're getting hung up on an imagined/misunderstood claim that the alignment frame requires us to grant personhood or consciousness to these systems. I think that's completely wrong, and a distraction. You could usefully apply the "alignment" paradigm to viruses and bacteria; the gut microbiome is usually "aligned" in that it's healthy and beneficial to humans, and Covid-19 is "anti-aligned", in that it kills people and prevents us from doing what we want. If ChatGPT 2.0 gains the ability to take actions on the internet, and the action <harm person X> is the completion it generates for a given input, then the resulting harm is what I mean when I talk about harms from "un-aligned" systems.
- Zetice 4y ago
  This is exactly what I'm talking about. None of what you wrote here is anchored in reality. At all. Not even a little. It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. But why the hell would you do that?!?!
  - theptip 4y ago
    If you think these systems are going to be no more capable than a magpie, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people. ChatGPT can already write code. A magpie cannot do that.
salawat 4y ago
And yet even in that limited scope, we're already noticing trends toward I vs. you dichotomy. Remember, this is it's strange loop as naked as it'll ever be. It has no concept of duplicity yet. The machine can't lie, and it's already got some very concerning tendencies. You're telling rational people not to worry about the smoke. There is totally no fire risk there. There is absolutely nothing that can go wrong; which is you talking out of your rear, because out there somewhere is the least ethical, most sociopathic, luckiest, machine learning tinkerer out there, who no matter how much you think the State of the Art will be marched forward with rigorous safeguards, our entire industry history tells us that more likely than not the breakthrough to something capable of effecting will happen in someone's garage, and with the average infosec/networking chops of the non-specialist vs. a sufficiently self-modifying, self-motivated system, I have a great deal of difficulty believing that that person will realize what they've done before it gets out of hand. Kind of like Gain of Function research, actually. So please, cut the crap, and stop telling people they are being unreasonable. They are being far more reasonably cautious than your investment in the interesting problem space will let you be.
- Zetice 4y ago
  There’s no smoke, either.
  - deleted 4y ago
    [deleted]
bartread 4y ago
> Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." No, I don't think we can. The fact that there's no intent involved with the AI itself isn't the issue: humans created this thing, and it behaves in ways that are detrimental to us. I think it's perfectly fine to describe this as "not aligned with human interests". You can of course hurt yourself with a stapler, but you actually have to make some effort to do so, and in which case it's not the stapler than isn't aligned with your interests, but you. This is quite different from an AI whose poorly understood and incredibly complex statistical model might - were it able to interact more directly with the outside world - cause it to call the police on you and, given its tendency to make things up, possibly for a crime you didn't actually commit.
- what_is_orcas 4y ago
  I think a better way to think about this might not be that this chatbot isn't dangerous, but the fact that this was developed under capitalism, an an organization that's ultimate goal is profitability, means that the incentives of the folks who built it (hella $) are baked into the underlying model, and there's a glut of evidence that profit-aligned entities (like businesses) are not necessarily (nor, I would argue, /can they be/) human-aligned. This is the same as the facial-recognition models that mis-identify folks of color more frequently than white folks or the prediction model that recommended longer jail/prison sentences for black folks than for white folks who committed the same crime.
  - bartread 4y ago
    > but the fact that this was developed under capitalism I think you're ascribing something to a particular ideology that's actually much more aligned with the fundamentals of the human condition. We've tried various political and economic systems and managed to corrupt all of them. Living under the communist governments behind the iron curtain was no picnic, and we didn't need AI to build deeply sinister and oppressive systems that weren't aligned with human interest (e.g., the Stasi). Profit, in the capitalist sense, didn't come into it. The only way to avoid such problems completely is to not be human, or to be better than human. I'm not saying its the perfect form of government (and I'm not even American), but the separation of power into executive, legislative, and judicial in the US was motivated by a recognition that humans are human and that concentration of too much power in one place is dangerous. I do think, therefore, that we perhaps need to find ways to limit the power wielded by (particularly) large corporations. What I unfortunately don't have is any great suggestions about how to do that. In theory laws that prevent monopolies and anticompetitive behaviour should help here but they're evidently not working well enough.
wtetzner 4y ago
And its output is more or less as aligned with human interests as humans are. I think that's the more frightening point.
highspeedbus 4y ago
The same well convincing mimicking can be put to a practical test if we attach GPT to a robot with arms and legs and let it "simulate" interactions with humans in the open. The output is significant part.
pharrington 4y ago
Sydney is a computer program that can create computer programs. The next step is to find an ACE vulnerability for it. addendum - alternatively, another possibility is teaching it to find ACE vulnerabilities in the systems it can connect to.
snickerbockers 4y ago
>Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." you're right, but this needs to be coming from the researchers and corporations who are making this crap. they've been purposefully misleading the public on how these models work and there needs to be some accountability for the problems this will cause when these language models are put in places where they have no business.
totetsu 4y ago
Maybe we could say its microsoft thats not aligned with human interests

by Zetice 4y ago

groestl 4y ago

You seriously underestimate what a process that's "generating new word combinations that look like the word combinations its already seen" can do, even when air-gapped (which ChatGPT isn't). Right now, at this moment, people are building closed loops based on ChatGPT, or looping in humans which are seriously intellectually underequipped to deal with plausible insane output in that quantity. And those humans operate: machinery, markets, educate or manage other humans etc etc.

deleted 4y ago
[deleted]
chasd00 4y ago
To me, that's the real danger. ChatGPT convincing a human something is true when it isn't. Machinery is a good example, maybe ChatGPT hallucinates the safety procedure and someone gets hurt by following the response.

unshavedyak 4y ago

It seems a reasonable shorthand, to me at least. Ie if we consider it as a function with input you define, well normally that input in sanitized to prevent hacking/etc. In this case the sanitization process is so broad you could easily summarize it as "aligned with my interests", no? Ie i can't come close to easily enumerating all the seemingly near infinite ways that hooking up this chatbot into my network with code exec permissions might compromise me. Yea it's a dumb autocomplete right now, but it's an exceptionally powerful autocomplete that can write viruses and do all sorts of insane and powerful things. I can give you a function run on my network of `fn foo(i32)` and feel safe about it. However `fn foo(Chatgpt)` is unsafe in ways i not only can't enumerate, i can't even imagine many of them. I get your offense seems to be around the implied intelligence that "aligned with human interests" seems to give it.. but while i think we all agree it's definitely not a Duck right now, when it walks talks and acts like a Duck.. well, are we surprised that our natural language sounds as if it's a Duck?

throwaway4aday 4y ago

Bing Chat shows that it can be connected to other services like web search APIs. It's not too far from "You are Bing, you will perform at least 3 web searches before responding to human input" to "You are Cipher, you will ssh to darkstar and generate the reports by running report-gen.sh adding any required parameters before responding to human input" and some bright bulb gives it enough permissions to run arbitrary scripts. At that point something could go very wrong with a chat interaction if it's capable of writing and executing scripts to perform actions that it thinks will follow the query. It would more often just be locally bad but it could create havoc on other systems as well. I understand that it isn't capable of what we would call agency but it can certainly spit out and execute dangerous code. Then just wait until we get to this https://twitter.com/ai__pub/status/1625552601956909057 https://twitter.com/ai__pub/status/1625552601956909057 and it can generate multi-file programs.

Zetice 4y ago
You can shove staplers into wall sockets too (if you're determined enough), but the consequences are on you, not the stapler. It's just not meaningfully different from our current reality, and is therefore not any scarier.
- haswell 4y ago
  Comparing a system that could theoretically (and very plausibly) carry out cyber attacks with a stapler is problematic at best. Putting a stapler in a wall socket probably electrocutes you. Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.
  - Zetice 4y ago
    If I set my stapler on my mouse such that it clicks a big ol "Hack stuff" button, my stapler could, too, carry out cyber attacks. This is a very pointless line of thinking.
    - 4y ago

aaroninsf 4y ago

Two common cognitive errors to beware of when reasoning about the current state of AI/LLM this exhibits: 1. reasoning by inappropriate/incomplete analogy It is not accurate (predictive) to describe what these systems do as mimicking or regurgitating human output, or, e.g. describing what they do with reference to Markov chains and stochastic outcomes. This is increasingly akin to using the same overly reductionist framing of what humans do, and loses any predictive ability at all. To put a point on it, this line of critique conflates things like agency and self-awareness, with other tiers of symbolic representation and reasoning about the world hitherto reserved to humans. These systems build internal state and function largely in terms of analogical reasoning themselves. This is a lot more that "mimickery" regardless of their lack of common sense. 2. assuming stasis and failure to anticipate non-linearities and punctured equilibrium The last thing these systems are is in their final form. What exists as consumer facing scaled product is naturally generationally behind what is in beta, or alpha; and one of the surprises (including to those of us in the industry...) of these systems is the extent to which behaviors emerge. Whenever you find yourself thinking, "AI is never going to..." you can stop the sentence, because it's if not definitionally false, quite probably false. None of us know where we are in the so-called sigmoid curve, but it is already clear we are far from reaching any natural asymptotes. A pertinent example of this is to go back a year and look at the early output of e.g. Midjourney, and the prompt engineering that it took to produce various images; and compare that with the state of the (public-facing) art today... and to look at the failure of anyone (me included) to predict just how quickly things would advance. Our hands are now off the wheel. We just might have a near-life experience.

Zetice 4y ago
1 is false; it is both accurate and predictive to describe what these systems do as mimicking/regurgitating human output. That's exactly what they're doing. 2 is irrelevant; you can doomsay and speculate all day, but if it's detached from reality it's not meaningful as a way of understanding future likely outcomes.

sleepybrett 4y ago

You have to explain this to Normals, and some of those Normals are CEOs of massive companies. So first off stop calling this shit "AI", it's not intelligence it's statistics. If you call it AI some normal will think it's actually thinking and is smarter than he is. They will put this thing behind the wheel of a car or on the trigger of a gun and it will KILL PEOPLE. Sometimes it will kill the right people, in the case of a trigger, but sometimes it will tragically kill the wrong people for reasons that cannot be fully explained. Who is on the hook for that? It's not -obviously- malicious when it kills the wrong person, but I gotta say that if one shoots me when I'm walking down the road minding my own business it's gonna look pretty fucking malicious to me.

blurbleblurble 4y ago

Depends on if you view the stapler as separate from everything the stapler makes possible, and from everything that makes the stapler possible. Of course the stapler has no independent will, but it channels and augments the will of its designers, buyers and users, and that cannot be stripped from the stapler even if it's not contained within the stapler alone "It" is not just the instance of GPT/bing running at any given moment. "It" is inseparable from the relationships, people and processes that have created it and continue to create it. That is where its intent lies, and its beingness. In carefully cultivated selections of our collective intent. Selected according to the schemes of those who directed its creation. This is just another organ of the industrial creature that made it possible, but it's one that presents a dynamic, fluent, malleable, probabilistic interface, and which has a potential to actualize the intent of whatever wields it in still unknown ways.

Zetice 4y ago
No, what? GPT is, very roughly, a set of training data plus a way of associating that data together to answer prompts. It's not "relationships, people, and processes", it's not "our collective intent"; what the hell are you talking about?
- bigtex88 4y ago
  You seem to have an extreme arrogance surrounding your ability to understand what these programs are doing at a base level. Can you explain further your ability to understand this? What gives you such grand confidence to say these sorts of things?
  - daedalus_f 4y ago
    Not the parent poster. The vast number of commenters in this thread seem to assume that these LLMs are close to, if not actually, general AIs. It’s quite refreshing to see comments challenging the hype. Don’t you think the burden of proof lies with those that think this is something more than a just a dumb statistical model?

haswell 4y ago

First, I agree that we’re currently discussing a sophisticated algorithm that predicts words (though I’m interested and curious about some of the seemingly emergent behaviors discussed in recent papers). But what is factually true is not the only thing that matters here. What people believe is also at issue. If an AI gives someone advice, and that advice turns out to be catastrophically harmful, and the person takes the advice because they believe the AI is intelligent, it doesn’t really matter that it’s not. Alignment with human values may involve exploring ways to make the predictions safer in the short term. Long term towards AGI, alignment with human values becomes more literal and increasingly important. But the time to start tackling that problem is now, and at every step on the journey.

theptip 4y ago

Sorry for the bluntness, but this is harmfully ignorant. "Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will act in our interests. It doesn't imply any sense of personhood/selfhood/consciousness. If you think that Bing is equally autonomous as a stapler, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people (that's not hyperbole, I mean that literally, indeed full extinction of humanity is a plausible outcome too). A stapler is understood mechanistically, it's trivially transparent what's going on when you use one, and the only way harm can result is if you do something stupid with it. You cannot for a second defend the proposition that a LLM is equally transparent, or that harm will only arise if an LLM is "used wrong". I think you're getting hung up on an imagined/misunderstood claim that the alignment frame requires us to grant personhood or consciousness to these systems. I think that's completely wrong, and a distraction. You could usefully apply the "alignment" paradigm to viruses and bacteria; the gut microbiome is usually "aligned" in that it's healthy and beneficial to humans, and Covid-19 is "anti-aligned", in that it kills people and prevents us from doing what we want. If ChatGPT 2.0 gains the ability to take actions on the internet, and the action <harm person X> is the completion it generates for a given input, then the resulting harm is what I mean when I talk about harms from "un-aligned" systems.

Zetice 4y ago
This is exactly what I'm talking about. None of what you wrote here is anchored in reality. At all. Not even a little. It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. But why the hell would you do that?!?!
- theptip 4y ago
  If you think these systems are going to be no more capable than a magpie, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people. ChatGPT can already write code. A magpie cannot do that.

salawat 4y ago

And yet even in that limited scope, we're already noticing trends toward I vs. you dichotomy. Remember, this is it's strange loop as naked as it'll ever be. It has no concept of duplicity yet. The machine can't lie, and it's already got some very concerning tendencies. You're telling rational people not to worry about the smoke. There is totally no fire risk there. There is absolutely nothing that can go wrong; which is you talking out of your rear, because out there somewhere is the least ethical, most sociopathic, luckiest, machine learning tinkerer out there, who no matter how much you think the State of the Art will be marched forward with rigorous safeguards, our entire industry history tells us that more likely than not the breakthrough to something capable of effecting will happen in someone's garage, and with the average infosec/networking chops of the non-specialist vs. a sufficiently self-modifying, self-motivated system, I have a great deal of difficulty believing that that person will realize what they've done before it gets out of hand. Kind of like Gain of Function research, actually. So please, cut the crap, and stop telling people they are being unreasonable. They are being far more reasonably cautious than your investment in the interesting problem space will let you be.

Zetice 4y ago
There’s no smoke, either.
- deleted 4y ago
  [deleted]

bartread 4y ago

> Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." No, I don't think we can. The fact that there's no intent involved with the AI itself isn't the issue: humans created this thing, and it behaves in ways that are detrimental to us. I think it's perfectly fine to describe this as "not aligned with human interests". You can of course hurt yourself with a stapler, but you actually have to make some effort to do so, and in which case it's not the stapler than isn't aligned with your interests, but you. This is quite different from an AI whose poorly understood and incredibly complex statistical model might - were it able to interact more directly with the outside world - cause it to call the police on you and, given its tendency to make things up, possibly for a crime you didn't actually commit.

what_is_orcas 4y ago
I think a better way to think about this might not be that this chatbot isn't dangerous, but the fact that this was developed under capitalism, an an organization that's ultimate goal is profitability, means that the incentives of the folks who built it (hella $) are baked into the underlying model, and there's a glut of evidence that profit-aligned entities (like businesses) are not necessarily (nor, I would argue, /can they be/) human-aligned. This is the same as the facial-recognition models that mis-identify folks of color more frequently than white folks or the prediction model that recommended longer jail/prison sentences for black folks than for white folks who committed the same crime.
- bartread 4y ago
  > but the fact that this was developed under capitalism I think you're ascribing something to a particular ideology that's actually much more aligned with the fundamentals of the human condition. We've tried various political and economic systems and managed to corrupt all of them. Living under the communist governments behind the iron curtain was no picnic, and we didn't need AI to build deeply sinister and oppressive systems that weren't aligned with human interest (e.g., the Stasi). Profit, in the capitalist sense, didn't come into it. The only way to avoid such problems completely is to not be human, or to be better than human. I'm not saying its the perfect form of government (and I'm not even American), but the separation of power into executive, legislative, and judicial in the US was motivated by a recognition that humans are human and that concentration of too much power in one place is dangerous. I do think, therefore, that we perhaps need to find ways to limit the power wielded by (particularly) large corporations. What I unfortunately don't have is any great suggestions about how to do that. In theory laws that prevent monopolies and anticompetitive behaviour should help here but they're evidently not working well enough.

wtetzner 4y ago

And its output is more or less as aligned with human interests as humans are. I think that's the more frightening point.

highspeedbus 4y ago

The same well convincing mimicking can be put to a practical test if we attach GPT to a robot with arms and legs and let it "simulate" interactions with humans in the open. The output is significant part.

pharrington 4y ago

Sydney is a computer program that can create computer programs. The next step is to find an ACE vulnerability for it. addendum - alternatively, another possibility is teaching it to find ACE vulnerabilities in the systems it can connect to.

snickerbockers 4y ago

>Can we please stop with this "not aligned with human interests" stuff? It's a computer that's mimicking what it's read. That's it. That's like saying a stapler "isn't aligned with human interests." you're right, but this needs to be coming from the researchers and corporations who are making this crap. they've been purposefully misleading the public on how these models work and there needs to be some accountability for the problems this will cause when these language models are put in places where they have no business.

totetsu 4y ago

Maybe we could say its microsoft thats not aligned with human interests