3 ms·

Go to lesswrong.com and do some searches about "goal," "reward," "agent," "alignment" etc. This has been addressed extensively. I don't like Yudkowsky's extreme

Go to lesswrong.com and do some searches about "goal," "reward," "agent," "alignment" etc. This has been addressed extensively. I don't like Yudkowsky's extreme framing about killing everyone instantly, nanotech, or very high probabilities of doom, so I'd avoid his stuff and read what others said. One example: suppose you can use the open-source LLAMA 6, released in 2027, to create a bot that earns billions by trading obscure currencies. It's not even deceptive - it straightforwardly tells you that scaling up your account will eventually cause economic instabilities in a small African country with some probability (Soros's trading forced the British government to withdraw from the European Exchange Rate Mechanism (ERM) and devalue its currency in 1992). Do you think nobody will do it? Would every single person care or even foresee that it could lead to widespread suffering (such as the inflation in Venezuela or Argentina)?

takinola 3y ago
This assumes it is possible to create such a bot. There is no indication that such a bot is possible. This is exactly my issue. All the disaster scenarios require some event that is not clear that an AGI could accomplish.
- zone411 3y ago
  Huh? Do you think it's impossible to create a bot that profits from trading various markets? It's been done many times, so obviously, it's possible. There are already bots that use LLMs to trade based on SEC filings, etc.
  - adastra22 3y ago
    You seem to think there is a lot more alpha here than actually exists.
  - takinola 3y ago
    Trading bots obviously exist. Can you build one that scales profits infinitely (as suggested above)? I would be very, very skeptical that is even possible. At some scale, trading edges stop working simply because market efficiencies kick in. Could you make a bunch of money in the market with an AGI? Possibly. Could you wreck the financial system because the AGI figures out some trading strategy? I have serious doubts.
    - deleted 3y ago
      [deleted]
    - croo 3y ago
      But that's the thing - AGI in theory will be able to constantly change the trading strategy. It will not figure out some trading strategy it will figure out all the strategies and how to apply it and what will be the consequences with how big of a probability. The only other thing capable of outsmart it will be another AGI and nothing else will come even remotely close.

takinola 3y ago

This assumes it is possible to create such a bot. There is no indication that such a bot is possible. This is exactly my issue. All the disaster scenarios require some event that is not clear that an AGI could accomplish.

zone411 3y ago
Huh? Do you think it's impossible to create a bot that profits from trading various markets? It's been done many times, so obviously, it's possible. There are already bots that use LLMs to trade based on SEC filings, etc.
- adastra22 3y ago
  You seem to think there is a lot more alpha here than actually exists.
- takinola 3y ago
  Trading bots obviously exist. Can you build one that scales profits infinitely (as suggested above)? I would be very, very skeptical that is even possible. At some scale, trading edges stop working simply because market efficiencies kick in. Could you make a bunch of money in the market with an AGI? Possibly. Could you wreck the financial system because the AGI figures out some trading strategy? I have serious doubts.
  - deleted 3y ago
    [deleted]
  - croo 3y ago
    But that's the thing - AGI in theory will be able to constantly change the trading strategy. It will not figure out some trading strategy it will figure out all the strategies and how to apply it and what will be the consequences with how big of a probability. The only other thing capable of outsmart it will be another AGI and nothing else will come even remotely close.