5 ms·

> We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai.

by narush 3y ago

> We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai. I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs! FWIW, I personally think that AIs will be able to invent more powerful/intelligent AIs. But we should be stating this as as conjecture, not a fact - mostly b/c my experience in saying things like this to people just causes them to be _extremely scared_. And for most people (myself included) extreme fear is a really bad place to try to make decisions from. > We need an equivalent of the Food and Drug Administration for new technology, and we need it yesterday. I wish there was more detail on this. Does Yuval Noah Harari really understand how the FDA works and is it really an appropriate model for the AI Safety Administration (AISA)? I mean - the FDA is not really batting 100% - and can we afford to miss swings when it comes to AI safety? Maybe we can. Maybe we can't. Anyone saying we can afford to miss swings is probably selling you something (or their livelihood depends on it). Anyone saying we can't afford to miss swings is probably terrified. I honestly don't know how to possibly feel other than ???!!??

Etrnl_President 3y ago
The only thing that can help is open-sourcing this technology; It is too dangerous in the hands of a monopoly or even a small oligopoly.
shrimpx 3y ago
> ai can make exponentially more powerful ai Is this true to any extent right now, or just blatantly incorrect assumptions on which to base clickbait and FUD?
- NumberWangMan 3y ago
  I think it's an assumption based on the fact that we don't seem to be anywhere near the maximum amount of computing capacity that we could be using to train AIs. It might be just throwing more compute at the problem, or it might be that we are nowhere close to the ideal algorithms / model architectures, and AI will aid the process of getting there. To turn it around a bit, what is your null hypothesis? If it's that AI will not be able to make more powerful AI, why do you believe that? Given that there are two cases, and smart people disagree about which one we're headed towards, and one could have very dire consequences while the other is no big deal, I think it's reasonable not to assume that the latter is true without exercising a lot of due diligence.
  - AnimalMuppet 3y ago
    "We can apply more compute power" is something very different from "exponentially self improving". Assuming that "exponentially improving" will show up if given enough compute power is very much not a null hypothesis.
    - NumberWangMan 3y ago
      Ok, so there's self improving, where an AI tries to make itself smarter, and then there's just exponential improvement, where an AI does one or more of the following, with or without a human in the loop: * acquires money through legitimate business and buys more computing power to train a smarter AI. * acquires money through illegal means and does the same. * designs a more efficient model architecture or training process that results in a smarter AI. I don't know how far away from this we are, it's very hard to predict. I do predict that at first we'll see mostly things where a human is directing the AI, but there's selective pressure toward not having a human involved, because someone who just sets up an AI and lets it loose is going to have a thing that reacts faster and can work 24/7. Irresponsible businesses that let their AIs run rampant are going to make more money, and be able to afford better AIs. It's the same story that has played out again and again over history, Moloch (1) pushing us to do the selfish but harmful thing by rewarding those that do it. Remember that exponential curves seem pretty slow at first, and it can be hard to tell when you are standing on one. It might be that we're still years away from exponential self-improvement, but I think it's worth taking seriously as a possibility. Even with just humans designing AIs, the amount of effort required has become exponentially less as we got more compute, designed better libraries and tools, people published research and copied it and built on it. We can't predict exactly what or when the next step will be, but I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic. [1] <https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>
      - AnimalMuppet 3y ago
        > I don't know how far away from this we are, it's very hard to predict. On this point, I am in total agreement with you. One way to tell might be, what happens if we let an AI try to improve itself? Does it get better, or worse? My money currently is on "worse". Well, how much worse? How far away are we from it being able to improve itself? I think we are an enormous distance away, almost an immeasurable distance, but I could be wrong. > I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic. It's always logistic in the end. (But if we're on the first part of the curve, where it's exponential, that "in the end" bit doesn't matter.)
- AnimalMuppet 3y ago
  Right now? Not true to any extent, no. But it's not exactly "blatantly incorrect assumptions". It's more wishful thinking. (The "self-improving, exponentially more powerful" AI is currently the only hope for an AGI - except that there isn't currently any realistic hope of a self-improving, exponentially more powerful AI.)
  - pixl97 3y ago
    Wishful? More like nightmare thinking, hence the desire for some people wanting to ensure that if it can happen, that it doesn't.
- pixl97 3y ago
  The issue is whenever it becomes true it is likely way too late to regulate it. For example let's imagine a hypothetical world that didn't have guns yet be we think they were possible. Well if you're waiting till one group made guns first you're going to be at their mercy going after their gun armed soldiers with swords.
AnimalMuppet 3y ago
> > We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai. > I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs! It seems to me that the current approach to AIs can improve along only two axes: better training data, or better algorithms. I don't think this kind of AI can generate better training data for the next generation of itself, just by the nature of how it operates. It (currently) doesn't know truth from falsehood; how is it going to know what training data is better or worse than other data? And I don't think it can improve its algorithm either. It may be able to implement an algorithm that someone else specifies, but I'm not sure that it can write a working algorithm, let alone an improved working algorithm. So I don't think that self-improving AI is currently on the table.
0xbadc0de5 3y ago
When someone says "we can regulate x, but we must act quickly," it's important to remember that the "we" they are referring to are often so poorly informed on the subject that the average house cat could outperform them if tested on the material and they are spoon-fed talking points by corporate lobbyists who pay their campaign budgets. Harari is a scare monger making bank off promoting the pop-culture "Terminator is bad" caricature of AI.
- MollyRealized 3y ago
  I was fairly upset with CBS' "60 Minutes" as I received a urgent message from one of my parents about the dangers of AI. It had such gems as: Google CEO: "It's just predicting the next word." News Story Narrator: "But it doesn't feel like that--" Yes, because how it "feels" is certainly objective material for a news story. And my favorite: CBS News Correspondent: "You don't fully understand how it works. And yet, you've turned it loose on society?" I actually dug into Giphy for a Frankenstein GIF for that one ... sigh Really getting sick of Fear-of-the-Other ...
  - thealchemistdev 3y ago
    Yup. Feelings don't care about your facts.
- JieJie 3y ago
  Still, I don't think that having a regulation that AI must reveal themselves when asked would be overstepping. It would draw a red line between aligned and unaligned AI. It may not even be enforceable, but it would be a revealing characteristic that gives useful information, even when it lies. Especially when it lies.
etiam 3y ago
>> We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai. I dropped in to comment on that bit too. You've described it well already, but I'd still like to add: .. hypothetically, in the future. To be fair, in principle then it seems perfectly feasible that a sufficiently much more advanced nuke might invent an even more advanced nuke too, hypothetically, in the future. Not really to detract from the rough direction of the argument in the post though, because despite the hype, imprecision and questionably declared speculation of the sort you pointed out, I think he's basically right to be concerned about the hazards of throwing even current and imminent "AI" into Humanity's communications and cultural evolution.

by narush 3y ago

Etrnl_President 3y ago

The only thing that can help is open-sourcing this technology; It is too dangerous in the hands of a monopoly or even a small oligopoly.

shrimpx 3y ago

> ai can make exponentially more powerful ai Is this true to any extent right now, or just blatantly incorrect assumptions on which to base clickbait and FUD?

NumberWangMan 3y ago
I think it's an assumption based on the fact that we don't seem to be anywhere near the maximum amount of computing capacity that we could be using to train AIs. It might be just throwing more compute at the problem, or it might be that we are nowhere close to the ideal algorithms / model architectures, and AI will aid the process of getting there. To turn it around a bit, what is your null hypothesis? If it's that AI will not be able to make more powerful AI, why do you believe that? Given that there are two cases, and smart people disagree about which one we're headed towards, and one could have very dire consequences while the other is no big deal, I think it's reasonable not to assume that the latter is true without exercising a lot of due diligence.
- AnimalMuppet 3y ago
  "We can apply more compute power" is something very different from "exponentially self improving". Assuming that "exponentially improving" will show up if given enough compute power is very much not a null hypothesis.
  - NumberWangMan 3y ago
    Ok, so there's self improving, where an AI tries to make itself smarter, and then there's just exponential improvement, where an AI does one or more of the following, with or without a human in the loop: * acquires money through legitimate business and buys more computing power to train a smarter AI. * acquires money through illegal means and does the same. * designs a more efficient model architecture or training process that results in a smarter AI. I don't know how far away from this we are, it's very hard to predict. I do predict that at first we'll see mostly things where a human is directing the AI, but there's selective pressure toward not having a human involved, because someone who just sets up an AI and lets it loose is going to have a thing that reacts faster and can work 24/7. Irresponsible businesses that let their AIs run rampant are going to make more money, and be able to afford better AIs. It's the same story that has played out again and again over history, Moloch (1) pushing us to do the selfish but harmful thing by rewarding those that do it. Remember that exponential curves seem pretty slow at first, and it can be hard to tell when you are standing on one. It might be that we're still years away from exponential self-improvement, but I think it's worth taking seriously as a possibility. Even with just humans designing AIs, the amount of effort required has become exponentially less as we got more compute, designed better libraries and tools, people published research and copied it and built on it. We can't predict exactly what or when the next step will be, but I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic. [1] <https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>
    - AnimalMuppet 3y ago
      > I don't know how far away from this we are, it's very hard to predict. On this point, I am in total agreement with you. One way to tell might be, what happens if we let an AI try to improve itself? Does it get better, or worse? My money currently is on "worse". Well, how much worse? How far away are we from it being able to improve itself? I think we are an enormous distance away, almost an immeasurable distance, but I could be wrong. > I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic. It's always logistic in the end. (But if we're on the first part of the curve, where it's exponential, that "in the end" bit doesn't matter.)
AnimalMuppet 3y ago
Right now? Not true to any extent, no. But it's not exactly "blatantly incorrect assumptions". It's more wishful thinking. (The "self-improving, exponentially more powerful" AI is currently the only hope for an AGI - except that there isn't currently any realistic hope of a self-improving, exponentially more powerful AI.)
- pixl97 3y ago
  Wishful? More like nightmare thinking, hence the desire for some people wanting to ensure that if it can happen, that it doesn't.
pixl97 3y ago
The issue is whenever it becomes true it is likely way too late to regulate it. For example let's imagine a hypothetical world that didn't have guns yet be we think they were possible. Well if you're waiting till one group made guns first you're going to be at their mercy going after their gun armed soldiers with swords.

AnimalMuppet 3y ago

> > We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai. > I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs! It seems to me that the current approach to AIs can improve along only two axes: better training data, or better algorithms. I don't think this kind of AI can generate better training data for the next generation of itself, just by the nature of how it operates. It (currently) doesn't know truth from falsehood; how is it going to know what training data is better or worse than other data? And I don't think it can improve its algorithm either. It may be able to implement an algorithm that someone else specifies, but I'm not sure that it can write a working algorithm, let alone an improved working algorithm. So I don't think that self-improving AI is currently on the table.

0xbadc0de5 3y ago

When someone says "we can regulate x, but we must act quickly," it's important to remember that the "we" they are referring to are often so poorly informed on the subject that the average house cat could outperform them if tested on the material and they are spoon-fed talking points by corporate lobbyists who pay their campaign budgets. Harari is a scare monger making bank off promoting the pop-culture "Terminator is bad" caricature of AI.

MollyRealized 3y ago
I was fairly upset with CBS' "60 Minutes" as I received a urgent message from one of my parents about the dangers of AI. It had such gems as: Google CEO: "It's just predicting the next word." News Story Narrator: "But it doesn't feel like that--" Yes, because how it "feels" is certainly objective material for a news story. And my favorite: CBS News Correspondent: "You don't fully understand how it works. And yet, you've turned it loose on society?" I actually dug into Giphy for a Frankenstein GIF for that one ... sigh Really getting sick of Fear-of-the-Other ...
- thealchemistdev 3y ago
  Yup. Feelings don't care about your facts.
JieJie 3y ago
Still, I don't think that having a regulation that AI must reveal themselves when asked would be overstepping. It would draw a red line between aligned and unaligned AI. It may not even be enforceable, but it would be a revealing characteristic that gives useful information, even when it lies. Especially when it lies.

etiam 3y ago

>> We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai. I dropped in to comment on that bit too. You've described it well already, but I'd still like to add: .. hypothetically, in the future. To be fair, in principle then it seems perfectly feasible that a sufficiently much more advanced nuke might invent an even more advanced nuke too, hypothetically, in the future. Not really to detract from the rough direction of the argument in the post though, because despite the hype, imprecision and questionably declared speculation of the sort you pointed out, I think he's basically right to be concerned about the hazards of throwing even current and imminent "AI" into Humanity's communications and cultural evolution.