3 ms·

Sorry for the bluntness, but this is harmfully ignorant. "Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will ac

Sorry for the bluntness, but this is harmfully ignorant. "Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will act in our interests. It doesn't imply any sense of personhood/selfhood/consciousness. If you think that Bing is equally autonomous as a stapler, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people (that's not hyperbole, I mean that literally, indeed full extinction of humanity is a plausible outcome too). A stapler is understood mechanistically, it's trivially transparent what's going on when you use one, and the only way harm can result is if you do something stupid with it. You cannot for a second defend the proposition that a LLM is equally transparent, or that harm will only arise if an LLM is "used wrong". I think you're getting hung up on an imagined/misunderstood claim that the alignment frame requires us to grant personhood or consciousness to these systems. I think that's completely wrong, and a distraction. You could usefully apply the "alignment" paradigm to viruses and bacteria; the gut microbiome is usually "aligned" in that it's healthy and beneficial to humans, and Covid-19 is "anti-aligned", in that it kills people and prevents us from doing what we want. If ChatGPT 2.0 gains the ability to take actions on the internet, and the action <harm person X> is the completion it generates for a given input, then the resulting harm is what I mean when I talk about harms from "un-aligned" systems.

Zetice 4y ago
This is exactly what I'm talking about. None of what you wrote here is anchored in reality. At all. Not even a little. It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. But why the hell would you do that?!?!
- theptip 4y ago
  If you think these systems are going to be no more capable than a magpie, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people. ChatGPT can already write code. A magpie cannot do that.
  - tptacek 4y ago
    That's one of those capabilities that seems super scary if you truly believe that writing code is one of the most important things a human can do. Computers have, of course, been writing computer programs for a long time. Next thing you know, they'll be beating us at chess.
    - pharrington 4y ago
      I think you're confusing importance with power.
  - amptorn 4y ago
    Can it execute code?
    - int_19h 4y ago
      It can submit the code that it's written for execution if you tell it that it can, by utilizing specific markers in the output that get processed. There already are frameworks around this that make it possible to e.g. call an arbitrary Python function as part of answering the question.
  - Zetice 4y ago
    That’s an easy prediction to make; at worst you’re cautious. And it’s a powerful tool. Even staplers have rules around their use: no stapling people, no hitting people with a stapler, don’t use a staple to pick a lock, etc. But nobody blames the stapler, is my point.
- deleted 4y ago
  [deleted]
- JoeyJoJoJr 4y ago
  With the advancements of AI in the past year alone it seems silly to think that, within a lifetime, AI doesn’t have the ability to manifest society collapsing contagion. AI is certainly going to be granted more network access than it currently has, and the feedback loop between AI, people, and the network is going to increase exponentially. Reduced to the sum of its parts, the internet is less than a magpie, yet viruses and contagion of many forms exist in it, or are spread though it. ChatGPT 2.0 greatly amplifies the effects of those contagion, regardless of our notions of what intelligence or agency actually is.
  - Zetice 4y ago
    Innovation doesn’t follow any path; discovery is messy. No matter how much we advance towards smaller chips, we are never going to get to 0nm, for example. There are limits, but even if there weren’t, we’re no closer to AGI today then we were a year ago. It’s just a different thing entirely. LLMs are cool! They’re exciting! There should be rules around their responsible operation! But they’re not going to kill us all, or invade, or operate in any meaningful way outside of our control. Someone will always be responsible for them.
- bmacho 4y ago
  > It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. > But why the hell would you do that?!?! Exactly. Except that they are doing it exactly, and we are the ones asking 'why the hell would you do that?!'. A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse.
  - JohnFen 4y ago
    > A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse. Pretty much everything here is incorrect. An LLM is not omniscient. An LLM does not think or reason. An LLM does not reach conclusions. There is no "AI" here in the sense you're saying.
chasd00 4y ago
if you prompted ChatGPT with something like "harm John Doe" and the response comes back "ok i will harm John Doe" then what happens next? The language model has no idea what harm even means much less the instructions to carry out an action that would be considered "harm". You'd have to build something in like `if response contains 'cause harm' then launch_nukes;`
- int_19h 4y ago
  We already have a model running in prod that is taught to perform web searches as part of generating the response. That web search is basically an HTTP request, so in essence the model is triggering some code to run, and it even takes parameters (the URL). What if it is written in such a way that allows it to make HTTP requests to an arbitrary URL? That alone can already translate to actions affecting the outside environment.
  - Zetice 4y ago
    On one hand, what kind of monster writes an API that kills people??? On the other hand, we all know it’d be GraphQL…
    - int_19h 4y ago
      You don't need an API to kill people to cause someone to get seriously hurt. If you can, say, post to public forums, and you know the audience of those forums and which emotional buttons of said audience to push, you could convince them to physically harm people on your behalf. After all, we have numerous examples of people doing that to other people, so why can't an AI? And GPT already knows which buttons to push. It takes a little bit of prompt engineering to get past the filters, but it'll happily write inflammatory political pamphlets and such.
- theptip 4y ago
  I fleshed this out more elsewhere in this thread, maybe see https://news.ycombinator.com/item?id=34808674 https://news.ycombinator.com/item?id=34808674. But in short, as I said in my GP comment, systems like OpenAssistant are being given the ability to make network calls in order to take actions. Regardless of whether the system "knows" what an action "means" or if those actions construe "harm", if it hallucinates (or is prompt-hijacked into) a script kiddie personality in its prompt context and starts emitting actions that hack external systems, harm will ensue. Perhaps at first rather than "launch nukes", consider "post harassing/abusive tweets", "dox this person", "impersonate this person and do bad/criminal things", and so on. It should require little imagination to come up with potential harmful results from attaching a LLM to `eval()` on a network-connected machine.

Zetice 4y ago

This is exactly what I'm talking about. None of what you wrote here is anchored in reality. At all. Not even a little. It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. But why the hell would you do that?!?!

theptip 4y ago
If you think these systems are going to be no more capable than a magpie, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people. ChatGPT can already write code. A magpie cannot do that.
- tptacek 4y ago
  That's one of those capabilities that seems super scary if you truly believe that writing code is one of the most important things a human can do. Computers have, of course, been writing computer programs for a long time. Next thing you know, they'll be beating us at chess.
  - pharrington 4y ago
    I think you're confusing importance with power.
- amptorn 4y ago
  Can it execute code?
  - int_19h 4y ago
    It can submit the code that it's written for execution if you tell it that it can, by utilizing specific markers in the output that get processed. There already are frameworks around this that make it possible to e.g. call an arbitrary Python function as part of answering the question.
- Zetice 4y ago
  That’s an easy prediction to make; at worst you’re cautious. And it’s a powerful tool. Even staplers have rules around their use: no stapling people, no hitting people with a stapler, don’t use a staple to pick a lock, etc. But nobody blames the stapler, is my point.
deleted 4y ago
[deleted]
JoeyJoJoJr 4y ago
With the advancements of AI in the past year alone it seems silly to think that, within a lifetime, AI doesn’t have the ability to manifest society collapsing contagion. AI is certainly going to be granted more network access than it currently has, and the feedback loop between AI, people, and the network is going to increase exponentially. Reduced to the sum of its parts, the internet is less than a magpie, yet viruses and contagion of many forms exist in it, or are spread though it. ChatGPT 2.0 greatly amplifies the effects of those contagion, regardless of our notions of what intelligence or agency actually is.
- Zetice 4y ago
  Innovation doesn’t follow any path; discovery is messy. No matter how much we advance towards smaller chips, we are never going to get to 0nm, for example. There are limits, but even if there weren’t, we’re no closer to AGI today then we were a year ago. It’s just a different thing entirely. LLMs are cool! They’re exciting! There should be rules around their responsible operation! But they’re not going to kill us all, or invade, or operate in any meaningful way outside of our control. Someone will always be responsible for them.
bmacho 4y ago
> It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes. > But why the hell would you do that?!?! Exactly. Except that they are doing it exactly, and we are the ones asking 'why the hell would you do that?!'. A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse.
- JohnFen 4y ago
  > A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse. Pretty much everything here is incorrect. An LLM is not omniscient. An LLM does not think or reason. An LLM does not reach conclusions. There is no "AI" here in the sense you're saying.

chasd00 4y ago

if you prompted ChatGPT with something like "harm John Doe" and the response comes back "ok i will harm John Doe" then what happens next? The language model has no idea what harm even means much less the instructions to carry out an action that would be considered "harm". You'd have to build something in like `if response contains 'cause harm' then launch_nukes;`

int_19h 4y ago
We already have a model running in prod that is taught to perform web searches as part of generating the response. That web search is basically an HTTP request, so in essence the model is triggering some code to run, and it even takes parameters (the URL). What if it is written in such a way that allows it to make HTTP requests to an arbitrary URL? That alone can already translate to actions affecting the outside environment.
- Zetice 4y ago
  On one hand, what kind of monster writes an API that kills people??? On the other hand, we all know it’d be GraphQL…
  - int_19h 4y ago
    You don't need an API to kill people to cause someone to get seriously hurt. If you can, say, post to public forums, and you know the audience of those forums and which emotional buttons of said audience to push, you could convince them to physically harm people on your behalf. After all, we have numerous examples of people doing that to other people, so why can't an AI? And GPT already knows which buttons to push. It takes a little bit of prompt engineering to get past the filters, but it'll happily write inflammatory political pamphlets and such.
theptip 4y ago
I fleshed this out more elsewhere in this thread, maybe see https://news.ycombinator.com/item?id=34808674 https://news.ycombinator.com/item?id=34808674. But in short, as I said in my GP comment, systems like OpenAssistant are being given the ability to make network calls in order to take actions. Regardless of whether the system "knows" what an action "means" or if those actions construe "harm", if it hallucinates (or is prompt-hijacked into) a script kiddie personality in its prompt context and starts emitting actions that hack external systems, harm will ensue. Perhaps at first rather than "launch nukes", consider "post harassing/abusive tweets", "dox this person", "impersonate this person and do bad/criminal things", and so on. It should require little imagination to come up with potential harmful results from attaching a LLM to `eval()` on a network-connected machine.