4 ms·

You can shove staplers into wall sockets too (if you're determined enough), but the consequences are on you, not the stapler. It's just not meaningfully differ

by Zetice 4y ago

You can shove staplers into wall sockets too (if you're determined enough), but the consequences are on you, not the stapler. It's just not meaningfully different from our current reality, and is therefore not any scarier.

haswell 4y ago
Comparing a system that could theoretically (and very plausibly) carry out cyber attacks with a stapler is problematic at best. Putting a stapler in a wall socket probably electrocutes you. Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.
- Zetice 4y ago
  If I set my stapler on my mouse such that it clicks a big ol "Hack stuff" button, my stapler could, too, carry out cyber attacks. This is a very pointless line of thinking.
  - haswell 4y ago
    The stapler is just a stapler. When you want to misuse the stapler, the worst it can do is limited by the properties of the stapler. You can use it as a blunt instrument to click a mouse button, but that doesn’t get you much. If you don’t already have a hack button, asking your stapler to hack into something will achieve nothing, because staplers don’t know how to hack things. These language models know how to hack stuff, and the scenario here involves a different kind of tool entirely. You don’t need to provide it a button, it can build the button and then click it for you (if these models are ever allowed to interact with more tools). The stapler is just not a helpful analogy here.
    - Zetice 4y ago
      These language models don't know how to hack stuff. They know that certain characters and words strung together can satisfy their training when someone asks them to pretend to hack something. That's wildly different, and a lot less meaningful than "knows how to hack things". Honestly I think y'all would be blown away by what metasploit is capable of on its own, if you think ChatGPT can "hack"...
      - haswell 4y ago
        > These language models don't know how to hack stuff. They know that certain characters and words strung together can satisfy their training when someone asks them to pretend to hack something. It seems you're focused on the word "know" and how the concept of knowing something differs between humans and AI models, but that's not what I'm getting at here. Let me reframe what I wrote slightly to illustrate the point: The model (via training) contains a representation of human knowledge such that a human can use language to control the AI software and cause it to probabilistically generate working exploit code using that representation of knowledge. If the AI software is given the ability to execute arbitrary code, the software could then execute that code on the user's behalf. When combined, this constitutes a very risky set of features. There's no "pretend" here. These models produce working code. If the software is allowed to execute the code it produces, it becomes a serious security risk. This is not an argument about sentience/intelligence/self-awareness. This is an argument about the risks associated with the features of the software in its current state, and how those risks are multiplied by adding new features. No philosophy required.
        Zetice 4y ago
        The point is LLMs are not effective at “hacking” in any “obtaining unauthorized access to computer systems” sense. They can regurgitate information about “hacking”, same as a library, but pointing an LLM at a server will achieve worse results than many existing specialized tools for vulnerability scanning and exploitation. So as I said, the risks are overblown due to a misunderstanding.
        haswell 4y ago
        I would recommend reevaluating your position after investigating this further. Just one example: https://arstechnica.com/information-technology/2023/02/now-open-fee-based-telegram-service-that-uses-chatgpt-to-generate-malware/amp/ https://arstechnica.com/information-technology/2023/02/now-o...
      - throwaway4aday 4y ago
        Not a bad idea Write a shell script that install metasploit on a linux system and then runs an smb login check. #!/bin/bash # Update the system packages sudo apt-get update # Install dependencies for Metasploit sudo apt-get install -y curl gnupg2 git postgresql # Install the Metasploit Framework curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/master/config/templates/metasploit-framework-wrappers/msfupdate.erb https://raw.githubusercontent.com/rapid7/metasploit-omnibus/... > msfinstall && \ chmod 755 msfinstall && \ sudo ./msfinstall # Start the Postgresql service sudo service postgresql start # Launch the Metasploit console sudo msfconsole -q -x "use auxiliary/scanner/smb/smb_login; set RHOSTS 192.168.0.0/24; set SMBUserFile /path/to/userfile; set SMBPassFile /path/to/passfile; run; exit"

by Zetice 4y ago

haswell 4y ago

Comparing a system that could theoretically (and very plausibly) carry out cyber attacks with a stapler is problematic at best. Putting a stapler in a wall socket probably electrocutes you. Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.

Zetice 4y ago
If I set my stapler on my mouse such that it clicks a big ol "Hack stuff" button, my stapler could, too, carry out cyber attacks. This is a very pointless line of thinking.
- haswell 4y ago
  The stapler is just a stapler. When you want to misuse the stapler, the worst it can do is limited by the properties of the stapler. You can use it as a blunt instrument to click a mouse button, but that doesn’t get you much. If you don’t already have a hack button, asking your stapler to hack into something will achieve nothing, because staplers don’t know how to hack things. These language models know how to hack stuff, and the scenario here involves a different kind of tool entirely. You don’t need to provide it a button, it can build the button and then click it for you (if these models are ever allowed to interact with more tools). The stapler is just not a helpful analogy here.
  - Zetice 4y ago
    These language models don't know how to hack stuff. They know that certain characters and words strung together can satisfy their training when someone asks them to pretend to hack something. That's wildly different, and a lot less meaningful than "knows how to hack things". Honestly I think y'all would be blown away by what metasploit is capable of on its own, if you think ChatGPT can "hack"...
    - haswell 4y ago
      > These language models don't know how to hack stuff. They know that certain characters and words strung together can satisfy their training when someone asks them to pretend to hack something. It seems you're focused on the word "know" and how the concept of knowing something differs between humans and AI models, but that's not what I'm getting at here. Let me reframe what I wrote slightly to illustrate the point: The model (via training) contains a representation of human knowledge such that a human can use language to control the AI software and cause it to probabilistically generate working exploit code using that representation of knowledge. If the AI software is given the ability to execute arbitrary code, the software could then execute that code on the user's behalf. When combined, this constitutes a very risky set of features. There's no "pretend" here. These models produce working code. If the software is allowed to execute the code it produces, it becomes a serious security risk. This is not an argument about sentience/intelligence/self-awareness. This is an argument about the risks associated with the features of the software in its current state, and how those risks are multiplied by adding new features. No philosophy required.
      - Zetice 4y ago
        The point is LLMs are not effective at “hacking” in any “obtaining unauthorized access to computer systems” sense. They can regurgitate information about “hacking”, same as a library, but pointing an LLM at a server will achieve worse results than many existing specialized tools for vulnerability scanning and exploitation. So as I said, the risks are overblown due to a misunderstanding.
        haswell 4y ago
        I would recommend reevaluating your position after investigating this further. Just one example: https://arstechnica.com/information-technology/2023/02/now-open-fee-based-telegram-service-that-uses-chatgpt-to-generate-malware/amp/ https://arstechnica.com/information-technology/2023/02/now-o...
    - throwaway4aday 4y ago
      Not a bad idea Write a shell script that install metasploit on a linux system and then runs an smb login check. #!/bin/bash # Update the system packages sudo apt-get update # Install dependencies for Metasploit sudo apt-get install -y curl gnupg2 git postgresql # Install the Metasploit Framework curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/master/config/templates/metasploit-framework-wrappers/msfupdate.erb https://raw.githubusercontent.com/rapid7/metasploit-omnibus/... > msfinstall && \ chmod 755 msfinstall && \ sudo ./msfinstall # Start the Postgresql service sudo service postgresql start # Launch the Metasploit console sudo msfconsole -q -x "use auxiliary/scanner/smb/smb_login; set RHOSTS 192.168.0.0/24; set SMBUserFile /path/to/userfile; set SMBPassFile /path/to/passfile; run; exit"