4 ms·

I just don’t understand how a transformer approach to human language can be expected to result in reasoning machines that are a threat to humanity. Feeding mor

I just don’t understand how a transformer approach to human language can be expected to result in reasoning machines that are a threat to humanity. Feeding more information in has absolutely resulted in phenomenal progress in predicting the next token in a given document, but how then do we expect even a perfect token predictor machine to start exhibiting independent emotions, desires, secret plots, and long term strategies? This doesn’t mean we won’t create such machines, but I suspect LLMs aren’t a real risk.

mitthrowaway2 3y ago
Here's one possibility: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
- alex_young 3y ago
  That’s a lot of words to say something like “if an LLM asks you to execute some code it could be dangerous”. Seems somewhat obvious. Would you execute random code from a ‘friend’ who emailed you? Does the LLM have any nefarious intentions?
  - mitthrowaway2 3y ago
    No, that isn't quite what it is saying. The LLM is simply running itself recursively on a task that you've assigned, which is the basic premise of agent models like Auto-GPT. Turns out that's dangerous. But too bad, it's also very useful, so it's going to be done, safe or not.
    - alex_young 3y ago
      Running recursively on a task sounds a lot like executing code. Let’s say we instead evaluate infiniteMonkeyBot. infiniteMonkeyBot simply issues random commands to a Unix prompt. Hypothetically this system is horrendously bad - it could potentially launch atomic weapons if we happen to connect those to the same system. However, both infiniteMonkeyBot and our scary LLM are unlikely to be connected in this way and lack any desire or understanding necessary for nefarious behavior.
voitvoder 3y ago
[flagged]

mitthrowaway2 3y ago

Here's one possibility: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

alex_young 3y ago
That’s a lot of words to say something like “if an LLM asks you to execute some code it could be dangerous”. Seems somewhat obvious. Would you execute random code from a ‘friend’ who emailed you? Does the LLM have any nefarious intentions?
- mitthrowaway2 3y ago
  No, that isn't quite what it is saying. The LLM is simply running itself recursively on a task that you've assigned, which is the basic premise of agent models like Auto-GPT. Turns out that's dangerous. But too bad, it's also very useful, so it's going to be done, safe or not.
  - alex_young 3y ago
    Running recursively on a task sounds a lot like executing code. Let’s say we instead evaluate infiniteMonkeyBot. infiniteMonkeyBot simply issues random commands to a Unix prompt. Hypothetically this system is horrendously bad - it could potentially launch atomic weapons if we happen to connect those to the same system. However, both infiniteMonkeyBot and our scary LLM are unlikely to be connected in this way and lack any desire or understanding necessary for nefarious behavior.

voitvoder 3y ago

[flagged]