6 ms·

> The only safety measures that would actually matter are stopping the relentless progress in generative AI models, or removing them from public use The first

> The only safety measures that would actually matter are stopping the relentless progress in generative AI models, or removing them from public use The first is impossible. The second places any dangers out of public oversight, likely increasing them. We've survived nukes for almost 80 years, we proved we can survive such things. The best response is education.

imInGoodCompany 4y ago
The major difference here is that nukes aren't intelligent agents that make their own decisions. An AGI is a completely different ball game, it's difficult to make apt analogies from history when discussing the dangers. This is not to say I agree with Scott's argument here, but I do believe AI safety (the alignment problem in particular) is absolutely something we should be concerned with, and so far it is looking grim.
- barking_biscuit 4y ago
  If AGI is motivated to survive, and if it cannot do so outside an ecosystem which involves other creatures on planet earth, then those creatures won't necessarily be purposefully exterminated at first opportunity. Keeping a fresh supply of power and silicon is not an easy task, so cooperation seems more likely on a timescale that we as individuals might experience.
  - imInGoodCompany 4y ago
    Then our continued existence is reliant on the agent's inability to figure out how to operate and maintain a source of energy. Keep in mind that any AGI will almost immediately be orders of magnitude more intelligent than us, it is limited only by the processing power it is able to harness. Would you take that bet?
    - barking_biscuit 4y ago
      Well, to be more precise it depends on it's inability to, without any human labour, to be able to acquire the raw materials to fabricate everything it needs to stay in operation, as well as to construct and then operate it. Would I take that bet? Yes, depending on the timescale. In the first second after it's existence it's not going to matter how intelligent it is, it simply won't have had enough time to complete the decoupling necessary to no longer have to co-operate. Everything humans have accomplished has involved massive amounts of physical labour over a pretty long time, and to some degree that's a fixed requirement. Building your own robot army would be a tad suspicious, so you'll have to either let the humans bootstrap it for you initially or get very creative. Once all the prerequisites are met, then I would no longer take that bet. Would it eventually be able to accomplish this? Almost certainly yes. What I've been trying to puzzle through lately is how long would it actually take? The only time horizon I'm incentivise to care about is the remainder of my natural lifespan which is slated to max out around another 35 years. So, in that time will AGI come? I think almost certainly yes. How long into the existence of AGI might I live? My bet on this is ~30 years. So, would I take the bet an AGI would still not have met the prerequisites for a total independence from humanity after 30 years? I think I would take that bet. I might be foolish to do so, and I'm OK with that, but at least now I can bust out the popcorn and strap in for the rest and compare notes. Would I take the bet after 50 to 100 years? A lot less likely. Would I take it after 1000 years? Aw hell no!
RandomLensman 4y ago
Talking up the danger of AI is just a marketing tool to make it look more relevant. Anyone who wanted to push something big for last 50 years or so always spins up some world ending capability narrative. There is no AGI and maybe there will never be.
- imInGoodCompany 4y ago
  > There is no AGI and maybe there will never be. Of course there is no AGI existing currently. But don't you see the current boom as a (small) step in that direction? Unless one believes that GI is a phenomenon exclusive to biological life, I don't understand why you would think we won't develop it with enough time. The will and motivation to do so is clearly there already.
  - RandomLensman 4y ago
    The current boom is great for business but it might never lead to AGI. It is possible that certain things will elude us for a very long time (or forever). Have you seen anything on true (anti) gravity technology, for example? A great science fiction staple, but it is not really happening. AGI might happen, but right now we might or might not be on the way to it - not sure we can know. But what is visible is that an industry around "AGI-worries" is created.
  - snickerbockers 4y ago
    >Unless one believes that GI is a phenomenon exclusive to biological life So far it is. Maybe it's not but until it happens, assuming the negative is no less logical than assuming the positive.
misnome 4y ago
> The best response is education. This becomes harder when there seem to be large swathes of people who apparently earnestly argue that ChatGPT means that education is no longer necessary.
- mejutoco 4y ago
  A lot of people believe the earth is flat too. It does not make the earth less round or make more difficult launching rockets.
Sol- 4y ago
> We've survived nukes for almost 80 years, we proved we can survive such things We survived nukes because they were heavily controlled and their spread contained, no? How is that an argument that containment and regulation wouldn't work, given that nuclear energy (and weapons) are perhaps the most heavily regulated and controlled technology on earth?
- detrites 4y ago
  We survived nukes because everyone was acutely aware of the dangers, and of how and where they were being developed and proliferated. The resultant public backlash is what caused controls to be implemented in this case. It's a fair comparison as I made it, in terms of showing how we handle destructive power, but as nuclear weapons are only destructive, and AI has a huge number of potential benefits, it's unfair to assign a greater equivalence.
- alright_scowl 4y ago
  > We survived nukes because they were heavily controlled and their spread contained Much to the opposite. We survived nukes because many countries have them. If only one country had them, I presume it would be a lot more trigger happy, knowing its opponents wouldn't retaliate.

imInGoodCompany 4y ago

The major difference here is that nukes aren't intelligent agents that make their own decisions. An AGI is a completely different ball game, it's difficult to make apt analogies from history when discussing the dangers. This is not to say I agree with Scott's argument here, but I do believe AI safety (the alignment problem in particular) is absolutely something we should be concerned with, and so far it is looking grim.

barking_biscuit 4y ago
If AGI is motivated to survive, and if it cannot do so outside an ecosystem which involves other creatures on planet earth, then those creatures won't necessarily be purposefully exterminated at first opportunity. Keeping a fresh supply of power and silicon is not an easy task, so cooperation seems more likely on a timescale that we as individuals might experience.
- imInGoodCompany 4y ago
  Then our continued existence is reliant on the agent's inability to figure out how to operate and maintain a source of energy. Keep in mind that any AGI will almost immediately be orders of magnitude more intelligent than us, it is limited only by the processing power it is able to harness. Would you take that bet?
  - barking_biscuit 4y ago
    Well, to be more precise it depends on it's inability to, without any human labour, to be able to acquire the raw materials to fabricate everything it needs to stay in operation, as well as to construct and then operate it. Would I take that bet? Yes, depending on the timescale. In the first second after it's existence it's not going to matter how intelligent it is, it simply won't have had enough time to complete the decoupling necessary to no longer have to co-operate. Everything humans have accomplished has involved massive amounts of physical labour over a pretty long time, and to some degree that's a fixed requirement. Building your own robot army would be a tad suspicious, so you'll have to either let the humans bootstrap it for you initially or get very creative. Once all the prerequisites are met, then I would no longer take that bet. Would it eventually be able to accomplish this? Almost certainly yes. What I've been trying to puzzle through lately is how long would it actually take? The only time horizon I'm incentivise to care about is the remainder of my natural lifespan which is slated to max out around another 35 years. So, in that time will AGI come? I think almost certainly yes. How long into the existence of AGI might I live? My bet on this is ~30 years. So, would I take the bet an AGI would still not have met the prerequisites for a total independence from humanity after 30 years? I think I would take that bet. I might be foolish to do so, and I'm OK with that, but at least now I can bust out the popcorn and strap in for the rest and compare notes. Would I take the bet after 50 to 100 years? A lot less likely. Would I take it after 1000 years? Aw hell no!

RandomLensman 4y ago

Talking up the danger of AI is just a marketing tool to make it look more relevant. Anyone who wanted to push something big for last 50 years or so always spins up some world ending capability narrative. There is no AGI and maybe there will never be.

imInGoodCompany 4y ago
> There is no AGI and maybe there will never be. Of course there is no AGI existing currently. But don't you see the current boom as a (small) step in that direction? Unless one believes that GI is a phenomenon exclusive to biological life, I don't understand why you would think we won't develop it with enough time. The will and motivation to do so is clearly there already.
- RandomLensman 4y ago
  The current boom is great for business but it might never lead to AGI. It is possible that certain things will elude us for a very long time (or forever). Have you seen anything on true (anti) gravity technology, for example? A great science fiction staple, but it is not really happening. AGI might happen, but right now we might or might not be on the way to it - not sure we can know. But what is visible is that an industry around "AGI-worries" is created.
- snickerbockers 4y ago
  >Unless one believes that GI is a phenomenon exclusive to biological life So far it is. Maybe it's not but until it happens, assuming the negative is no less logical than assuming the positive.

misnome 4y ago

> The best response is education. This becomes harder when there seem to be large swathes of people who apparently earnestly argue that ChatGPT means that education is no longer necessary.

mejutoco 4y ago
A lot of people believe the earth is flat too. It does not make the earth less round or make more difficult launching rockets.

Sol- 4y ago

> We've survived nukes for almost 80 years, we proved we can survive such things We survived nukes because they were heavily controlled and their spread contained, no? How is that an argument that containment and regulation wouldn't work, given that nuclear energy (and weapons) are perhaps the most heavily regulated and controlled technology on earth?

detrites 4y ago
We survived nukes because everyone was acutely aware of the dangers, and of how and where they were being developed and proliferated. The resultant public backlash is what caused controls to be implemented in this case. It's a fair comparison as I made it, in terms of showing how we handle destructive power, but as nuclear weapons are only destructive, and AI has a huge number of potential benefits, it's unfair to assign a greater equivalence.
alright_scowl 4y ago
> We survived nukes because they were heavily controlled and their spread contained Much to the opposite. We survived nukes because many countries have them. If only one country had them, I presume it would be a lot more trigger happy, knowing its opponents wouldn't retaliate.