5 ms·

When I compare threat of chemists (as example) to AGI it does not make sense, I genuinely don't understand what possible threat AGI can poses, like what is thre

• iammjm 4y ago
  I think the “unknown unknowns” of a possibly godlike-intelligence, especially in regards to the alignment of its vs human values, is what worries people
• lumost 4y ago
  Have you seen a ChatGPT writing a stream of consciousness? The curious and unexpected challenge of modern AI is that we have succeeded in making useful AIs which inherit all of the biases/problems of their training material. ChatGPT can effectively write exploits if you prompt it correctly - we'll likely see the first mass AI driven 0-day security vulnerabilities in the next ~2 years. GPT Transformer model's are known to solve control/RL problems well. I wouldn't be surprised if we see ChatGPT derived models operating in physical space soon as well. A major limitation with flexible factory automation robots has been that it's too expensive to program them for the task - I don't think this will be the case 5 years from now.
  • Avamander 4y ago
    > we'll likely see the first mass AI driven 0-day security vulnerabilities in the next ~2 years. Maybe for outdated and less-common software, even then assuming it gets the input about what specifically to exploit. Which makes it not much of a threat any time soon.
• bena 4y ago
  What could you do if you were incredibly intelligent, had the corpus of the world's knowledge, and access to any and everything connected to a network? It could influence people through the creation of media. Shut off vital systems. And stuff I probably couldn't even conceive of.
  • bamboozled 4y ago
    I guess the thing is, why would it do malicious things? I think if an AI or AGI could understand the world and have motives to do bad, it would have an ego or an identity to discriminate between itself and the world. Then I guess it needs an intrinsic will to survive like a biological creature? If it did have this drive to survive, I suppose it would also need to experience fear and pain of some kind, or else it might blow itself up, or build a better version of itself which might destroy it, so it would in some ways it might exhibit similar inhibitions as all other conscious and or biological beings. On the other hand if we build something that's super smart, and doesn't really care about or understand survival, it might just turn itself off for not really understanding it's purpose. The most dangerous thing I feel is humans using these things as weapons.
    • 6gvONxR4sf7o 4y ago
      The issue is that it doesn’t have to be malicious to cause harm. That’s the point of the paper clip optimizer thought experiment. It’s not about motivation as much as just what it happens to have learned to do.
      • bamboozled 4y ago
        Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Who or what will pay for the energy to do that? Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?
        • 6gvONxR4sf7o 4y ago
          Why would it not? Superintelligent doesn’t imply human-like intelligence. Prolog is superintelligent, for example. It may be more useful to say a super powerful being, rather than super intelligent, just to sidestep the anthropomorphizing. It’s easy to imagine a super powerful “stupid” (read: misaligned) bot going awry.
• drexlspivey 4y ago
  As a thought experiment imagine if ChatGPT had unrestricted internet access (and no filter). You could ask things like “can you hack that website for me?” it could produce python code and then run it. The result of the previous command would inform the next command. Now take it a step further and say that GPT-5 needs no user prompt to initiate a run and could start taking actions by prompting itself. I could see things going south in this example scenario.
  • epistemer 4y ago
    That is hardly as dangerous as a chemist making bombs.
• thedorkknight 4y ago
  Doesn't have to be truly "godlike" to be a threat, just relatively godlike. Do humans pose a threat to the species around us? Are we "godlike"?
• epistemer 4y ago
  It strikes me many in our society live in a constant stream of perceived threats. If you are basically self propagandizing yourself that the world is a very dangerous place I don't think it is shocking to see major threats everywhere. I know some people that don't talk about anything other than perceived threats even though they literally live in the safest society in human history. AGI really fits this mindset like a glove because it can be perceived to be a threat to literally anything. It is the perfect idea to worry about. Chemists doing bad things is obviously not really something to worry about when doing a historic cost/benefit analysis so it won't even register.
• floxy 4y ago
  It seems like you need to worry about people wielding the AI before you have to worry about the AI itself. Just think how most people are at least somewhat immune to the advertisement / mass persuasion techniques of today. But what happens when there is an army of bots with the equivalent of 1/10/100/1000 human minds assigned to sway each person on an individual level of various political things? If there were 1000 people working to investigate you and manipulate the information you see, subtly rewriting your incoming and outgoing text messages/emails, while playing off your biases, desires, etc., wouldn't that be troubling? Or course that would be too expensive to do right now, but with an AI in the cloud in the future...