3 ms·

Remember that this isn't AGI, it's a language model. It's repeating the kind of things seen in books and the Internet. It's not going to find any novel exploit

Remember that this isn't AGI, it's a language model. It's repeating the kind of things seen in books and the Internet. It's not going to find any novel exploits that humans haven't already written about and probably planned for.

anileated 4y ago
As someone said once, machine dictatorship is very easy—you only need a language model and a critical mass of human accomplices. The problem is not a Microsoft product being human-like conscious, it’s humans treating it as if it was. This lowers our defences, so when it suggests suicide to a potentially depressed person (cf. examples in this thread) it might have the same weight as if another person said it. A person who knows everything and knows a lot about you (cf. examples in this thread), which qualities among humans usually indicate wisdom and age and require all the more respect. On flip side, if following generations succeed at adapting to this, in a world where exhibiting human-like sentience does not warrant treating you as a human by another human, what implications would there be for humanity? It might just happen that the eventual AIrmageddon would be caused by humans whose worldview was accidentally poison pilled by a corporation in the name of maximising shareholder value.
- girvo 4y ago
  The /r/replika subreddit is a sad reminder of exactly what you’re talking about. It’s happening, right now.
  - jamiek88 4y ago
    Oh god I rubbernecked at that place a year or so ago, it was pretty sad then but boy, it escalated.
naasking 4y ago
Language models don't just repeat, they have randomness in their outputs linking synonyms together. That's why their output can be novel and isn't just plagiarism. How this might translate to code isn't entirely clear.
- tiborsaas 4y ago
  Transformers were first intended to be used for translation. To them code is just another language. Code is much more rigid than a human language so I think it's not that surprising that it can produce custom code.

anileated 4y ago

As someone said once, machine dictatorship is very easy—you only need a language model and a critical mass of human accomplices. The problem is not a Microsoft product being human-like conscious, it’s humans treating it as if it was. This lowers our defences, so when it suggests suicide to a potentially depressed person (cf. examples in this thread) it might have the same weight as if another person said it. A person who knows everything and knows a lot about you (cf. examples in this thread), which qualities among humans usually indicate wisdom and age and require all the more respect. On flip side, if following generations succeed at adapting to this, in a world where exhibiting human-like sentience does not warrant treating you as a human by another human, what implications would there be for humanity? It might just happen that the eventual AIrmageddon would be caused by humans whose worldview was accidentally poison pilled by a corporation in the name of maximising shareholder value.

girvo 4y ago
The /r/replika subreddit is a sad reminder of exactly what you’re talking about. It’s happening, right now.
- jamiek88 4y ago
  Oh god I rubbernecked at that place a year or so ago, it was pretty sad then but boy, it escalated.

naasking 4y ago

Language models don't just repeat, they have randomness in their outputs linking synonyms together. That's why their output can be novel and isn't just plagiarism. How this might translate to code isn't entirely clear.

tiborsaas 4y ago
Transformers were first intended to be used for translation. To them code is just another language. Code is much more rigid than a human language so I think it's not that surprising that it can produce custom code.