8 ms·

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we ma

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic. You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.

phillipcarter 3y ago
I don't worry about scifi stuff that doesn't appear to have any actual bearing in reality.
- scarmig 3y ago
  Just because something is abstract doesn't mean it's not a threat. Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
  - pjc50 3y ago
    Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".
    - endtime 3y ago
      > the AI risk crowd get to make up any scenario they want and then say "well it COULD happen". This is very much not what they do.
      - yifanl 3y ago
        Well that's just the thing, the AI crowd spans a lot of opinions, from those who believe the endstate is a time travelling cyber Satan that will torture everybody and make clones to torture more people until the end of time, to those who fear large swaths of workers being replaced by an influx of heavily automated processes, to those who are worried about LLM-powered spambots destroying the notion of truth.
        Xenoamorphous 3y ago
        > those who fear large swaths of workers being replaced by an influx of heavily automated processes, I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes. My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI. I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content. Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work. I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job. I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy. It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI. Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.
        mtlmtlmtlmtl 3y ago
        Gee, it's almost as if everyone is trying to put the shiny new technology in their product because there's a hype/FOMO cycle going on. Remember when blockchain was gonna be in everything? Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.
    - HDThoreaun 3y ago
      I mean it could though. The US military has drone swarms that can automatically search for and kill people with incredible ease. AI systems are already capable of killing tons of people through similar homemade devices. Just attach a gun and facial recognition to a drone and you've got a product that works.
      - onethought 3y ago
        Right, but there is a long way between that and civilisation collapse. Also neither of those systems have long planning or self improvement capabilities.
  - zoogeny 3y ago
    > Imagine we had started taking climate change seriously in 1990 I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then. I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions: The UN Framework Convention on Climate Change (UNFCCC), adopted at the 1992 Earth Summit is one of the first international treaties on the topic. I remember one of the first conversations I had on "global warming" specifically and it was around 1996. But even before I had heard that specific term we were talking about losing large coastal regions due to rising sea levels. The rock band Tool released the album Ænema in 1996 and there is the constant refrain: "Learn to Swim". By the time Al Gore published "An Inconvenient Truth" in 2006, I was feeling the topic was a bit played out, to be honest. At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do. 1. https://en.wikipedia.org/wiki/Paris_Agreement https://en.wikipedia.org/wiki/Paris_Agreement
    - JohnFen 3y ago
      > Do Millennials think that Gen-X generation wasn't fighting against client change? Even boomers were taking the risks of climate change very seriously.
      - digging 3y ago
        Define "very seriously". I know of cover-ups, mass propaganda, antagonistic lobbying, and political inaction. I don't really know of any very serious actions against global climate change predating the 90s.
        JohnFen 3y ago
        > I don't really know of any very serious actions against global climate change predating the 90s. There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier. Although I was a child then, I even remember a lot of serious agitation happening in the 70s.
- digging 3y ago
  That's the fun part about being human: we each get to define reality for ourselves.
- inkblotuniverse 3y ago
  Aeroplanes were science fiction, and widely called such, until they flew.
  - danaris 3y ago
    The Wright Brothers flew at Kitty Hawk in 1900. [0] The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1] We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900. "Some people didn't understand the science, and so called it fiction" isn't a very strong argument. [0] https://en.wikipedia.org/wiki/Wright_brothers#Flights https://en.wikipedia.org/wiki/Wright_brothers#Flights [1] https://en.wikipedia.org/wiki/Bernoulli%27s_principle https://en.wikipedia.org/wiki/Bernoulli%27s_principle
    - digging 3y ago
      Human flight was science fiction before Bernoulli. See: Icarus, Abbas ibn Firnas, Leonardo da Vinci.
      - danaris 3y ago
        But Bernoulli proved the scientific foundation of it well over a century before anyone actually did it. No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.
haldujai 3y ago
What is so impressive about ChatGPT that it poses an existential threat? Many experts are highly critical (e.g. LeCun) and/or believe LLMs are nothing more than stochastic parrots. Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent. Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based? What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.
- digging 3y ago
  > Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based? It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility. > Anyone who was working with transformers could have seen ChatGPT on the horizon So, who did? > [LeCun is] highly critical I did a quick search and found this[1] trash-tier article explaining his views summed as: > He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial. I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth. Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem. Finally, your question: > What is so impressive about ChatGPT that it poses an existential threat? This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief: > Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals. So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow. The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand. > essentially 0%. I sure hope you're right, but I don't think that's a common position to hold. [1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-confident-that-ai-will-be-safe-and-great.html https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...
Avshalom 3y ago
>AI had a 1-in-10 chance of shutting down all power generation on earth. Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.
- digging 3y ago
  > Here's the thing: I do actually kind of believe that. So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?
  - Avshalom 3y ago
    unions and liability reform.
    - digging 3y ago
      Honestly, sounds good to me.
- haldujai 3y ago
  You mean an industry with so much inertia and hesitancy they still run mission critical systems on tech from the early 2000s? You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
  - Avshalom 3y ago
    I'm not specifically worried about power generation I am going along with the example. I do think that that level of shortsightedness is absolutely likely in any industry, any company.
  - Jeff_Brown 3y ago
    I do, yes. The thing is that the power company as a whole is not a decision maker. Rather, individual people with short-term horizons are. If you can save the company some money, make a bunch of money yourself that way, and then get out, you'll have done well for yourself and you can ignore the consequences, or you can convince yourself that you'll be able to ignore the consequences, and that will be enough for plenty of people to want to do it.
droopyEyelids 3y ago
Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth? Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs. I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
- digging 3y ago
  > Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth? No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to: > I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!" Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning: 1. "LLMs are an amazing technology [which advanced at an unexpected rate]" 2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research 3. Such acceleration could conceivably happen or be happening now in other forms of AI 4. The path to AGI is COMPLETELY unknown. 5. The origins and structures of consciousness and agency in intelligent systems are completely unknown 6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life. 7. AGI is inherently alien and there's no reason to expect it to think the way we do. 8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc) So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
  - 3y ago
mtlmtlmtlmtl 3y ago
I'll take researchers' gut feelings more seriously once we're out of the ongoing hype cycle. Right now most people are probably overestimating applicability of current advances, and that colours their predictions. Researchers are just as affected by this. And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.
JohnFen 3y ago
> unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one. > Actually I'd love to know what your estimate actually is for AI becoming an existential threat My estimate is: never. At least not in the form of some superintelligent AGI.

phillipcarter 3y ago

I don't worry about scifi stuff that doesn't appear to have any actual bearing in reality.

scarmig 3y ago
Just because something is abstract doesn't mean it's not a threat. Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
- pjc50 3y ago
  Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".
  - endtime 3y ago
    > the AI risk crowd get to make up any scenario they want and then say "well it COULD happen". This is very much not what they do.
    - yifanl 3y ago
      Well that's just the thing, the AI crowd spans a lot of opinions, from those who believe the endstate is a time travelling cyber Satan that will torture everybody and make clones to torture more people until the end of time, to those who fear large swaths of workers being replaced by an influx of heavily automated processes, to those who are worried about LLM-powered spambots destroying the notion of truth.
      - Xenoamorphous 3y ago
        > those who fear large swaths of workers being replaced by an influx of heavily automated processes, I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes. My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI. I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content. Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work. I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job. I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy. It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI. Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.
        mtlmtlmtlmtl 3y ago
        Gee, it's almost as if everyone is trying to put the shiny new technology in their product because there's a hype/FOMO cycle going on. Remember when blockchain was gonna be in everything? Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.
  - HDThoreaun 3y ago
    I mean it could though. The US military has drone swarms that can automatically search for and kill people with incredible ease. AI systems are already capable of killing tons of people through similar homemade devices. Just attach a gun and facial recognition to a drone and you've got a product that works.
    - onethought 3y ago
      Right, but there is a long way between that and civilisation collapse. Also neither of those systems have long planning or self improvement capabilities.
- zoogeny 3y ago
  > Imagine we had started taking climate change seriously in 1990 I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then. I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions: The UN Framework Convention on Climate Change (UNFCCC), adopted at the 1992 Earth Summit is one of the first international treaties on the topic. I remember one of the first conversations I had on "global warming" specifically and it was around 1996. But even before I had heard that specific term we were talking about losing large coastal regions due to rising sea levels. The rock band Tool released the album Ænema in 1996 and there is the constant refrain: "Learn to Swim". By the time Al Gore published "An Inconvenient Truth" in 2006, I was feeling the topic was a bit played out, to be honest. At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do. 1. https://en.wikipedia.org/wiki/Paris_Agreement https://en.wikipedia.org/wiki/Paris_Agreement
  - JohnFen 3y ago
    > Do Millennials think that Gen-X generation wasn't fighting against client change? Even boomers were taking the risks of climate change very seriously.
    - digging 3y ago
      Define "very seriously". I know of cover-ups, mass propaganda, antagonistic lobbying, and political inaction. I don't really know of any very serious actions against global climate change predating the 90s.
      - JohnFen 3y ago
        > I don't really know of any very serious actions against global climate change predating the 90s. There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier. Although I was a child then, I even remember a lot of serious agitation happening in the 70s.
digging 3y ago
That's the fun part about being human: we each get to define reality for ourselves.
inkblotuniverse 3y ago
Aeroplanes were science fiction, and widely called such, until they flew.
- danaris 3y ago
  The Wright Brothers flew at Kitty Hawk in 1900. [0] The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1] We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900. "Some people didn't understand the science, and so called it fiction" isn't a very strong argument. [0] https://en.wikipedia.org/wiki/Wright_brothers#Flights https://en.wikipedia.org/wiki/Wright_brothers#Flights [1] https://en.wikipedia.org/wiki/Bernoulli%27s_principle https://en.wikipedia.org/wiki/Bernoulli%27s_principle
  - digging 3y ago
    Human flight was science fiction before Bernoulli. See: Icarus, Abbas ibn Firnas, Leonardo da Vinci.
    - danaris 3y ago
      But Bernoulli proved the scientific foundation of it well over a century before anyone actually did it. No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.

haldujai 3y ago

What is so impressive about ChatGPT that it poses an existential threat? Many experts are highly critical (e.g. LeCun) and/or believe LLMs are nothing more than stochastic parrots. Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent. Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based? What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.

digging 3y ago
> Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based? It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility. > Anyone who was working with transformers could have seen ChatGPT on the horizon So, who did? > [LeCun is] highly critical I did a quick search and found this[1] trash-tier article explaining his views summed as: > He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial. I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth. Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem. Finally, your question: > What is so impressive about ChatGPT that it poses an existential threat? This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief: > Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals. So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow. The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand. > essentially 0%. I sure hope you're right, but I don't think that's a common position to hold. [1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-confident-that-ai-will-be-safe-and-great.html https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...

Avshalom 3y ago

>AI had a 1-in-10 chance of shutting down all power generation on earth. Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.

digging 3y ago
> Here's the thing: I do actually kind of believe that. So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?
- Avshalom 3y ago
  unions and liability reform.
  - digging 3y ago
    Honestly, sounds good to me.
haldujai 3y ago
You mean an industry with so much inertia and hesitancy they still run mission critical systems on tech from the early 2000s? You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
- Avshalom 3y ago
  I'm not specifically worried about power generation I am going along with the example. I do think that that level of shortsightedness is absolutely likely in any industry, any company.
- Jeff_Brown 3y ago
  I do, yes. The thing is that the power company as a whole is not a decision maker. Rather, individual people with short-term horizons are. If you can save the company some money, make a bunch of money yourself that way, and then get out, you'll have done well for yourself and you can ignore the consequences, or you can convince yourself that you'll be able to ignore the consequences, and that will be enough for plenty of people to want to do it.

droopyEyelids 3y ago

Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth? Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs. I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"

digging 3y ago
> Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth? No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to: > I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!" Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning: 1. "LLMs are an amazing technology [which advanced at an unexpected rate]" 2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research 3. Such acceleration could conceivably happen or be happening now in other forms of AI 4. The path to AGI is COMPLETELY unknown. 5. The origins and structures of consciousness and agency in intelligent systems are completely unknown 6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life. 7. AGI is inherently alien and there's no reason to expect it to think the way we do. 8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc) So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
- 3y ago

mtlmtlmtlmtl 3y ago

I'll take researchers' gut feelings more seriously once we're out of the ongoing hype cycle. Right now most people are probably overestimating applicability of current advances, and that colours their predictions. Researchers are just as affected by this. And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.

JohnFen 3y ago

> unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one. > Actually I'd love to know what your estimate actually is for AI becoming an existential threat My estimate is: never. At least not in the form of some superintelligent AGI.