5 ms·

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype"

It's always really embarrassing to come to these comment sections and see a lot of smart people talk about how they're not being fooled by the "marketing hype" of existential AI risk. Literally the top of the page is saying that they have no conclusive evidence that ChatGPT could actually increase the risk of biological weapons. They are undertaking this effort because the question of how to stop any AI from ending humanity once it has the capabilities is completely unsolved. We don't even know if it's solvable, let alone what approach to take. Don't you actually believe it is not essential to practice on weaker AI before we get to that point? Would you walk through a combat zone without thinking about how to protect yourself until after you hear a gunshot? I expect many replies about ChatGPT being "too stupid" to end the world. Please hold those replies, as they completely miss the point. If you consider yourself an intelligent and technical person, and you think it's not worth thinking about existential risks posed by future AI, I would like to know when you think it will be time for AI researchers (not you personally) to start preparing for those risks.

solarpunk 3y ago
what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes? if that's the case, how much resources do you think should be dedicated to regulating it? more or less than currently identified existential risks? which entities should be paying for the regulatory controls? what's the proposal here? it's odd because only this one single company that is hedging it's entire existence on "oh boy what if this thing is dangerous some time in the near future" is doing silly stunts like this. why aren't they demanding nvidia start building DRM enabled thermite charges into A100s?
- digging 3y ago
  > what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes? It certainly could. More likely, if an LLM is used, it will be as a piece integrating various specialized agents. I, not an expert in AI interpretability or alignment research, can't say if what they're doing is worthwhile or not in addressing existential risk. But I also don't know if actual experts can say that either. > how much resources do you think should be dedicated to regulating it? Definitely not a lower amount than we currently are allocating. > what's the proposal here? That the smart people here stop looking for any excuse to deny and ridicule the existential threat posed by future AI. Every thread involving OpenAI (a company I personally dislike and don't trust) doesn't need to just turn into series of glib, myopic jokes.
  - solarpunk 3y ago
    i wouldn't worry too much about this. if people were being serious, rather than cynically weaponizing non-expert anxieties in pursuit of regulatory capture or marketing, the pragmatic solution to all future worries about AI alignment is simply utilizing the DRM resources already built into every computing device on the planet, to disable hardware in the event the program on it does bad things.
    - digging 3y ago
      > the pragmatic solution to all future worries about AI alignment is simply ... A sentence beginning with this is, I can pretty much guarantee, never going to end in truth. I will leave it as an exercise for the reader to determine why remotely bricking every computer on Earth (or even just a subset known to be infected, which might reside in a hostile nation) might not be pragmatic.
      - solarpunk 3y ago
        dude... the concern is /existential/ though, right?! to summarize, i'm not advocating for this, i'm just emphasizing there's a nifty little framework already in place.
        digging 3y ago
        Yes, but so is climate change, and we can't even get countries to agree to grow more slowly, let alone shut down everything. > i'm just emphasizing there's a nifty little framework already in place. More than one! Nuclear war, economic isolation, ground invasion. All kinds of nifty things we could do to stop dangerous AI. None of them are likely to happen when the risk is identified. To summarize, any easy solution to superhuman AI trying to kill all humans you can think of in a few seconds, someone has probably already thought about.
        solarpunk 3y ago
        what if the ai superintelligence is banking on us running datacenters non-stop in the hopes of continued return on investment. all while humans are deprived of electricity needed to operate air conditioning in a continuously heating world. and we all die off from what was initially anthtrogenic, but now ai-induced climate change? i've got a two birds; one stone solution.
- Vecr 3y ago
  > why aren't they demanding nvidia start building DRM enabled thermite charges into A100s? Tweet that at Yudkowsky, he'll probably endorse it.
bithive123 3y ago
Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent. This is always the pattern; the "greatest minds" invent a machine to kill, then another great technician invents a machine to kill that machine, and so on. Presuming that total security can be achieved mechanically, and in that pursuit bringing about only more insecurity. Humanity can barely manage the existential risks for which it is not responsible; entering into an AI arms race with itself seems completely unnecessary, but I'm certain it will happen for the reasons already mentioned.
- digging 3y ago
  > Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent. The alternative, not trying at all, sounds more intelligent to you? Or just easier? Many agree with you that defense is inherently harder than offense. It may even be effectively impossible to survive AGI, who knows? You don't, I can be pretty sure of that, because no human has ever publicly proven it one way or the other. The only wrong answer to this hard problem, though, is "give up and see what happens."
  - bithive123 3y ago
    Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat, and that the only rational decisions is to figure out how to kill it. You also seem to equate intelligence with technical capability. I question all of that.
    - digging 3y ago
      > Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat, That is not correct. > and that the only rational decisions is to figure out how to kill it. That is also not correct and not something I claimed.
- 3y ago
RandomLensman 3y ago
Hypothetical risks of hypothetical machines. We don't even know if it needs solving.
groby_b 3y ago
We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. (The same playground reasoning the EA community engages in, which is a large vector for the xrisk hype. Just multiply by a large enough number, and you will surely have the biggest number) To the credit of the authors, they don't engage in that. There is no hand wringing over the absolutely unlikely case of "but what if the AI awakens". But it's still an extremely weak study - it proves nothing (none of the results are statistically significant), and even if it had shown significant uplift, it's meaningless without a control. Of course people who have access to a knowledge store do slightly better than people who don't. I'm willing to bet that "access to a 10-book research library" produces roughly the same uplift. Without that (trivial) control, it's really bad study design. And the moment you take this study and its non-results and call it "Building an early warning system for LLM-aided biological threat creation", you've absolutely lost all credibility.
- foo3a9c4 3y ago
  Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing. > We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused. > xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth? > All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development. Premise 1. I value my own survival over just about anything else. Premise 2. If an existential catastrophe occurs, then I will die. Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe. Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.
michael_nielsen 3y ago
I'm not sure why they're so often so bad. I wonder if it's the Upton Sinclair effect; to paraphrase slightly: "It is difficult to get a person to understand something, when their hoped-for future wealth depends on not understanding it."
- jph00 3y ago
  There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it. The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT. Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.
  - foo3a9c4 3y ago
    > There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it. > The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT. > Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support. IMO your comment doesn't substantively address michael_nielsen's comment, but I might be wrong. The following is how I understand your exchange with michael_nielsen. The two of you are talking about three sets of people: Let A be AI notkilleveryoneism people. Let B be AI capabilities developers/supporters. Let C be people concerned with regulatory capture and centralization by AI firms. A and B are disjoint. A and C have some overlap. B and C have considerable overlap. michael_nielsen is suggesting that the people of B are refusing to take AI risk seriously because they are excited about profiting from AI capabilities and its funding. (eg, a senior research engineer at OpenAI who makes $350k/year might be inclined to ignore AIXR and the same with a VC who has a portfolio full of AI companies) And then you are pointing out that people of C are getting less money to investigate AI centralization than people of A are getting to investigate/propagandize AI notkilleveryoneism. So, your claim is probably true, but it doesn't rebut what michael_nielsen suggested. And I believe it's also critical to keep in mind that the actual funding is like this: capabilities development >>>>>>>>>> ai notkilleveryoneism > ai centralization investigation
nradov 3y ago
Nah, they're just making shit up and talking their book. Intelligent people reject the entire premise. Stop being so naive.
kjkjadksj 3y ago
The type of ai that can end humanity will not be a chatbot. Knowing statistically which word is liable to be next in a sentence is what these tools actually do when you take off the marketing glowup. Whats more likely is that these sorts of ais take a lot of jobs that produce churn content, a good thing if people spent that waste of a time doing something creative instead perhaps, but unfortunately our world does not alleviate us from busy work to provide us with food shelter and creative outlets in its wake. Quite the opposite. Maybe thats what needs our attention in a more automated world than anything from sci fi.

solarpunk 3y ago

what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes? if that's the case, how much resources do you think should be dedicated to regulating it? more or less than currently identified existential risks? which entities should be paying for the regulatory controls? what's the proposal here? it's odd because only this one single company that is hedging it's entire existence on "oh boy what if this thing is dangerous some time in the near future" is doing silly stunts like this. why aren't they demanding nvidia start building DRM enabled thermite charges into A100s?

digging 3y ago
> what's the premise here? this thing will become iteratively better until it could potentially be capable of bad outcomes? It certainly could. More likely, if an LLM is used, it will be as a piece integrating various specialized agents. I, not an expert in AI interpretability or alignment research, can't say if what they're doing is worthwhile or not in addressing existential risk. But I also don't know if actual experts can say that either. > how much resources do you think should be dedicated to regulating it? Definitely not a lower amount than we currently are allocating. > what's the proposal here? That the smart people here stop looking for any excuse to deny and ridicule the existential threat posed by future AI. Every thread involving OpenAI (a company I personally dislike and don't trust) doesn't need to just turn into series of glib, myopic jokes.
- solarpunk 3y ago
  i wouldn't worry too much about this. if people were being serious, rather than cynically weaponizing non-expert anxieties in pursuit of regulatory capture or marketing, the pragmatic solution to all future worries about AI alignment is simply utilizing the DRM resources already built into every computing device on the planet, to disable hardware in the event the program on it does bad things.
  - digging 3y ago
    > the pragmatic solution to all future worries about AI alignment is simply ... A sentence beginning with this is, I can pretty much guarantee, never going to end in truth. I will leave it as an exercise for the reader to determine why remotely bricking every computer on Earth (or even just a subset known to be infected, which might reside in a hostile nation) might not be pragmatic.
    - solarpunk 3y ago
      dude... the concern is /existential/ though, right?! to summarize, i'm not advocating for this, i'm just emphasizing there's a nifty little framework already in place.
      - digging 3y ago
        Yes, but so is climate change, and we can't even get countries to agree to grow more slowly, let alone shut down everything. > i'm just emphasizing there's a nifty little framework already in place. More than one! Nuclear war, economic isolation, ground invasion. All kinds of nifty things we could do to stop dangerous AI. None of them are likely to happen when the risk is identified. To summarize, any easy solution to superhuman AI trying to kill all humans you can think of in a few seconds, someone has probably already thought about.
        solarpunk 3y ago
        what if the ai superintelligence is banking on us running datacenters non-stop in the hopes of continued return on investment. all while humans are deprived of electricity needed to operate air conditioning in a continuously heating world. and we all die off from what was initially anthtrogenic, but now ai-induced climate change? i've got a two birds; one stone solution.
Vecr 3y ago
> why aren't they demanding nvidia start building DRM enabled thermite charges into A100s? Tweet that at Yudkowsky, he'll probably endorse it.

bithive123 3y ago

Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent. This is always the pattern; the "greatest minds" invent a machine to kill, then another great technician invents a machine to kill that machine, and so on. Presuming that total security can be achieved mechanically, and in that pursuit bringing about only more insecurity. Humanity can barely manage the existential risks for which it is not responsible; entering into an AI arms race with itself seems completely unnecessary, but I'm certain it will happen for the reasons already mentioned.

digging 3y ago
> Asking if humanity can invent a machine to protect itself from the existential threats created by the other machines it has invented to me does not sound that intelligent. The alternative, not trying at all, sounds more intelligent to you? Or just easier? Many agree with you that defense is inherently harder than offense. It may even be effectively impossible to survive AGI, who knows? You don't, I can be pretty sure of that, because no human has ever publicly proven it one way or the other. The only wrong answer to this hard problem, though, is "give up and see what happens."
- bithive123 3y ago
  Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat, and that the only rational decisions is to figure out how to kill it. You also seem to equate intelligence with technical capability. I question all of that.
  - digging 3y ago
    > Rather than enquire into the nature of the problem, you've started with the conclusion that AGI is an existential threat, That is not correct. > and that the only rational decisions is to figure out how to kill it. That is also not correct and not something I claimed.
3y ago

RandomLensman 3y ago

Hypothetical risks of hypothetical machines. We don't even know if it needs solving.

groby_b 3y ago

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. (The same playground reasoning the EA community engages in, which is a large vector for the xrisk hype. Just multiply by a large enough number, and you will surely have the biggest number) To the credit of the authors, they don't engage in that. There is no hand wringing over the absolutely unlikely case of "but what if the AI awakens". But it's still an extremely weak study - it proves nothing (none of the results are statistically significant), and even if it had shown significant uplift, it's meaningless without a control. Of course people who have access to a knowledge store do slightly better than people who don't. I'm willing to bet that "access to a 10-book research library" produces roughly the same uplift. Without that (trivial) control, it's really bad study design. And the moment you take this study and its non-results and call it "Building an early warning system for LLM-aided biological threat creation", you've absolutely lost all credibility.

foo3a9c4 3y ago
Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing. > We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused. > xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth? > All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development. Premise 1. I value my own survival over just about anything else. Premise 2. If an existential catastrophe occurs, then I will die. Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe. Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.

michael_nielsen 3y ago

I'm not sure why they're so often so bad. I wonder if it's the Upton Sinclair effect; to paraphrase slightly: "It is difficult to get a person to understand something, when their hoped-for future wealth depends on not understanding it."

jph00 3y ago
There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it. The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT. Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.
- foo3a9c4 3y ago
  > There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it. > The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT. > Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support. IMO your comment doesn't substantively address michael_nielsen's comment, but I might be wrong. The following is how I understand your exchange with michael_nielsen. The two of you are talking about three sets of people: Let A be AI notkilleveryoneism people. Let B be AI capabilities developers/supporters. Let C be people concerned with regulatory capture and centralization by AI firms. A and B are disjoint. A and C have some overlap. B and C have considerable overlap. michael_nielsen is suggesting that the people of B are refusing to take AI risk seriously because they are excited about profiting from AI capabilities and its funding. (eg, a senior research engineer at OpenAI who makes $350k/year might be inclined to ignore AIXR and the same with a VC who has a portfolio full of AI companies) And then you are pointing out that people of C are getting less money to investigate AI centralization than people of A are getting to investigate/propagandize AI notkilleveryoneism. So, your claim is probably true, but it doesn't rebut what michael_nielsen suggested. And I believe it's also critical to keep in mind that the actual funding is like this: capabilities development >>>>>>>>>> ai notkilleveryoneism > ai centralization investigation

nradov 3y ago

Nah, they're just making shit up and talking their book. Intelligent people reject the entire premise. Stop being so naive.

kjkjadksj 3y ago

The type of ai that can end humanity will not be a chatbot. Knowing statistically which word is liable to be next in a sentence is what these tools actually do when you take off the marketing glowup. Whats more likely is that these sorts of ais take a lot of jobs that produce churn content, a good thing if people spent that waste of a time doing something creative instead perhaps, but unfortunately our world does not alleviate us from busy work to provide us with food shelter and creative outlets in its wake. Quite the opposite. Maybe thats what needs our attention in a more automated world than anything from sci fi.